The team reviewed preliminary benchmarking results showing performance improvements in various clients after the Osaka fork, though some data is still pending due to memory and caching issues. Discussion focused on EIP 8038 which targets state access operations and aims to address worst-case scenarios in contract storage and account access. The conversation ended with plans to gather more feedback from affected entities and further analyze the impact of dynamic cost changes on application performance, while also addressing concerns about complexity and potential impacts on smart wallets and Layer 2s.
Click to expand detailed summary
Maria presented preliminary results from recent runs after the Osaka fork, noting that numbers are still being run on Prague without the Transaction Limit EIP. She highlighted price decreases in precompiles and optimizations in Besu for MoD and div operations, as well as fixes to MOD tests that better estimate full operation costs. Maria explained that excluding scaffolding opcodes from estimations provides more accurate values for target opcodes, but data from Erigon and Reth clients is still missing due to OOM implementations and caching issues in precompiles like ECRECOV and EC pairing. Final numbers will require performance measurement on Goerli-optimized clients.
Maria presented benchmarking results showing that K-check operations were 20% cheaper than previously, though she noted these were preliminary numbers and final results would include ball-optimized operations. The tests were run on Geth, Besu, and Nethermind clients, excluding Erigon and Reth due to memory issues, with Maria offering to share the test code and tools used for reproduction. Dragan raised questions about memory issues, which Maria suggested discussing asynchronously with Rafael from EthPandaOps and Camille from Nethermind who are managing the benchmark tools.
The team discussed benchmarking results and performance issues related to precompiles and caching. Maria noted that caching was likely affecting the test results, and the team agreed to double-check the test definitions and running methods. Ĺukasz and Jochem shared their knowledge of previous benchmarks and migration efforts, while Dragan proposed removing changes to precompile pick-up due to implementation nuances. The group agreed to focus on worst-case scenarios for gas increases and to disable caching in tests where possible. They planned to continue discussions on benchmarking and state operations in the next update.
The meeting focused on discussing EIP 8038, which targets changes to state access operations and aims to address worst-case scenarios in contract storage and account access. Jochem presented the challenges posed by caching in benchmarks, emphasizing the need to target existing and non-existing accounts to accurately assess performance. Maria and Daniel discussed the potential increase in gas costs due to state operations, with Maria noting that the exact impact is still uncertain and will depend on client optimizations. Spencer provided an update on the devnet release, highlighting progress on integrating EIPs and addressing framework complexities. The group also discussed the need for specific tests to benchmark the reservoir mechanism, with Maria and Jochem agreeing that existing tests should suffice. Butta shared feedback from validator operators on EIPs, noting concerns about complexity and potential impacts on smart wallets and Layer 2s. The conversation ended with plans to gather more input from affected entities and further analyze the impact of dynamic cost per byte on application performance.
Next Steps:
Maria: Share the link to the analysis code and the actual test used to estimate K-check opcode performance for team review
Luis (Besu team): Complete PR for adding new implementation based on optimizations for different operations to equalize with mods
Maria/Rafael/Camille: Follow up with Erigon and Reth teams regarding out-of-memory issues when running benchmarks
Team (all): Review preliminary 7904 numbers and provide async feedback on anything that stands out
Maria/Team: Double check EC pairing and ECRECOV precompile tests due to caching issues with repeated inputs
Maria/Team: Fix precompile tests to use different inputs to prevent caching (especially EC pairing and ECRECOV)
Team (all): Review compute tests to ensure they capture worst-case scenarios and arenât affected by client-specific caching optimizations
Maria/Team: Conduct final benchmarking run with ball-optimized implementations for 7904
Jochem: Share benchmarks comparing cache hits vs misses for state operations
Ĺukasz: Have Camille join discussion about benchmark methodology and caching prevention strategies
Spencer: Port static YAML tests to Python to properly handle 8037 block gas limit dimension
Spencer: Take a look at whether additional tests are needed for reservoir mechanism worst-case scenarios over the next week
Maria/Team: Think about new worst cases now that state creation gas is not included in the 60 million transaction limit
Maria: Discuss with Butta about smart wallet concerns regarding whether contract deployment is affected by 2780
Maria/Team: Follow up with entities that expressed high concern about EIPs (especially 8037 and 8038) to clarify impact
Butta: Reach out to Arbitrum and Optimism for their opinions on gas repricing
Butta: Try to get wallet stakeholder opinions during EFCC
Carl: Provide in-depth data on affected entities by end of month
Maria/Butta: Reach out to specifically affected entities once Carlâs data is available
Maria/Team: Understand actual day-to-day impact of dynamic cost per byte complexity with quantization trick through follow-ups with concerned entities