The team discussed various technical issues and updates related to the BP01 activation, including bug fixes and version releases, as well as concerns about mainnet and devnet performance. They reviewed progress on benchmark testing, gas pricing, and block-level access list tests, addressing client-related challenges and planning further development and testing phases. The team also discussed upcoming releases, testing priorities, and performance concerns, deciding to cancel certain meetings and continue with planned testing and development activities.
Click to expand detailed summary
Mario welcomed participants to ACDT 61 on November 10th and noted that the agenda was similar to previous weeks. He inquired about significant events from the previous week’s BP01 activation, to which Barnabas responded that there were no significant events to discuss. Mario then suggested moving on to the next topic, which was the status of the BP01 activation.
The team discussed issues with BPO2 and a bug in Nethermind, which Tim was tasked to investigate. Marcin explained that they had released version 0.1 with bug fixes but it introduced worse bugs, so they reverted to version 0.2. They plan to release a new version after the mainnet fork with known optimizations and fixes, prioritizing stability over rare bug fixes. The team also mentioned a broken Snap sync due to regression in downloading block receipts and a consensus bug.
The team discussed several issues related to the mainnet release and devnets. Parithosh mentioned that MergeN changes affecting block-level access lists were released before the fork. Mario suggested reaching out to Reth to confirm if they plan another release. Enrico noted that nodes are detecting some curious RPC calls on mainnet, which Mario asked to investigate further. Barnabas reported a halt in Nimbus SuperNodes and mentioned ongoing maintenance on devnet3 to address disk space issues. The team agreed to follow up with the Nimbus team about the halt issue and to consider shutting down devnet3 once mainnet is ready.
The team discussed network blob management, with Barnabas explaining that 20 blobs per block over a 10-day period should result in approximately 640GB, though nodes were running out of disk space. Jen from Reth reported that they had released v1.9.1 on Friday to address an EVM regression bug, which was necessary for the Fusaka release. The team noted that while Prism had a mainnet strategy release, there was no update on a stable release this week.
The team discussed the status of the Prism release, with Preston working on preparing a release after weekend soak testing. They also reviewed progress on benchmark testing for the 60 million gas mainnet, where Marcin reported that initial testing with Nethermind and Geth using EIP7904 was successful, and the next step involves generating new payloads to match the updated consensus rules.
The team discussed benchmark testing and gas pricing, with Ameziane explaining that EIP-7904 might show a decrease in mgas/s due to increased transactions per gas limit, suggesting the need to include TPS metrics in the dashboard. Maria confirmed that while there were two gas repricing breakouts previously, no further breakouts are scheduled immediately due to DevConnect, though recordings of the overview sessions are available. Mario inquired about the next gas repricing call, and Francesco suggested holding another session on Wednesday to move beyond the overview stage for some EIPs.
The team discussed the status of block-level access list tests and their planned launch in DevNet. Spencer explained that recent changes in the test releases, including the addition of EVM fuzzing scenarios, have caused widespread failures across clients. The team is working on fixing these issues, with Felipe expected to address the problems either tomorrow or Wednesday. They also discussed the need for clients to better print debugging information related to block-level access list hashes.
The team discussed block-level access lists, with Mario noting that clients agreed to include both expected and generated bad blocks in the access list. They decided not to launch DevNets until Hive test failures are reduced to a manageable level. Justin provided an update on ePBS, mentioning that an off-protocol value field will be added to the Execution payload and previous RANDAO will be included in the bid. The next ePBS breakout call is scheduled for December 5th, with DevNet Zero likely to happen in mid-January.
The team discussed whether to cancel upcoming calls, with general agreement to cancel the November 24th meeting due to travel schedules and team members being offline. They also addressed concerns about bugs in the upcoming mainnet release, with Parithosh noting that while some issues were regression-related, most staking entities have redundant setups and backup systems in place to handle potential failures.
The team discussed the stability of the test network, with Barnabas noting it was smoother than previous merges and forks. Mario asked about concerns from other clients, and Ben mentioned they had released a fix for an issue 12 hours after the initial release. Parithosh inquired about testing priorities, and Saulius suggested conducting nonfinality testing on a larger network before the mainnet hard fork. The team agreed to plan this testing for after DevConnect.
The team discussed performance and reliability concerns related to blob handling in the post-peer DAS system. Parithosh confirmed that Sam is investigating potential issues, though it’s challenging to identify bugs due to the testnet infrastructure reuse. The group addressed questions about private blobs and reorg-heavy situations, with Barnabas noting no issues were observed with private blobs but reorg communications might cause problems for specific ER clients. Kasey raised concerns about the strain on nodes during reorgs, particularly when exercising blockQ code paths, and suggested testing these scenarios in testnets. Parithosh also mentioned a recently fixed bug in the MEV boost relay configuration that addressed a race condition.
The team discussed nonfinality testing on devnetri, agreeing to wait for the resolution of the Nimbus bug before proceeding. They decided to cancel the calls scheduled for the 17th and 24th, with ACDT resuming on December 1st. Casey reported that the Prism release, expected by Preston, would not be completed that day.