Meeting Summary:
The meeting focused on DevNet 6 status updates and planning for DevNet 7. Barnabas reported that most busy nodes had recovered from issues, though some nodes including Teco Ergon and Lodestar Gas remained problematic due to database resync requirements. Karim explained that the issues stemmed from Prism sending multiple new payloads without FCU, which he had attempted to fix but may need to revert as Prism plans to revert their modification. Dustin from the Nimbus team confirmed they are working on fixing issues related to Fortress implementation. The team discussed plans for DevNet 7, targeting a Wednesday launch, with Barnabas noting he would be off for two and a half weeks after that launch. The group agreed this should be the final DevNet with a spec freeze, and discussed the need for Alpha 12 release on the CL side and new spec tests on the EL side for the agreed changes.
Click to expand detailed summary
The meeting began with participants waiting for Barnabas to join, who arrived a few minutes late. There were technical issues with Sam’s audio that required multiple participants to assist with muting his microphone. Once Barnabas joined, the meeting was ready to start, though the actual discussion content is not captured in this transcript segment.
Barnabas reported issues with multiple Bezu nodes, including Teco Ergon and Lodestar Gas, which were recovered after database resyncs. Karim explained that the issues stemmed from a fix made to address Prism sending multiple new payloads without FCU, which caused latency in Bezu. Karim indicated that Prism plans to revert their modification, potentially requiring a reversal of the fix he implemented, and he will propose a PR to revert the changes for more stability.
The team discussed an issue with FCU behavior in node syncing, where Karim explained it appears to be a state inconsistency when matching Prysm behavior rather than a spec-related problem. potuz expressed surprise that this issue affected Besu since the behavior had been running for months on mainnet, and noted that changes were only made for Goerli but not for Fuji. The team agreed to investigate further, particularly checking mainnet for similar issues and verifying the behavior with different batch sizes. Barnabas also provided an update on DevNet 6, confirming that Lodestar HX1 was running the expected Death Star behavior with orphaned blocks, and noted that some clients including Nimbus were still experiencing issues with Fortress implementation.
The team discussed issues with block building in the devnet, where Nimbus nodes were primarily self-building blocks instead of using external builders. Dustin confirmed they know the cause and have a broad understanding of the fix, though he couldn’t provide a specific timeline as he won’t be writing the PR. Barnabas noted that all clients are now onboarded, and the team confirmed plans to launch DevNet 7 on Wednesday pending spec updates and client updates.
The team discussed launching DevNet 7, with Barnabas suggesting Wednesday as the target date due to personal time off, though Thursday or the following week were also considered. The group agreed to implement a spec freeze for this DevNet, allowing only minor parameter changes but no major design modifications. Justin Traglia and Mario confirmed they planned to release updates on the CL and EL sides respectively, with Mario noting two open EIP changes that needed discussion in the breakout session. The team also arranged for breakout room sessions to discuss remaining specification changes, with Pooja coordinating recording for these sessions.
Next Steps:
- Karim: Propose a PR to revert the recent fix related to Prism’s payload behavior and test for stability.
- potuz: Investigate if the issue with Prism’s behavior affects mainnet and adjust Fulu behavior if necessary.
- Dustin (Nimbus team): Work on the fix for the choice issues and investigate the external builder block usage.
- spencer: Prioritize and fix the issue with the spare test diverging from EIP-2780 for DevNet 7.
- Mario: Finalize the 2 open EIP changes in the EL breakout and implement the changes for the release.
- Barnabas: Target to launch DevNet 7 on Wednesday, with a fallback to Thursday or next week if not possible.
- Pooja: Coordinate with Mark Wolovin to record the breakout sessions, specifically the EL1 session.
- Barnabas: Create the breakout groups for the meeting.
Recording Access: