Agenda
- devnet5 simulations and scaling
Meeting Time: Wednesday, June 24, 2026 at 14:00 UTC (60 minutes)
Meeting Time: Wednesday, June 24, 2026 at 14:00 UTC (60 minutes)
This meeting focused on team updates and current challenges with DevNet 5 stability. Gajinder reported that the Zim team successfully resolved scaling issues and ran 32 nodes across 4 subnets to finality, with plans to increase node counts and test longer duration networks. Shariq mentioned implementing optimizations and running interop tests with ETH Lambda in one subnet, planning to add more clients. Mihir from Lantern reported average type 2 proof generation taking 2.8 seconds with some block building delays over 10 seconds causing finality gaps. Pablo from ETH Lambda noted CPU resource limitations with 16 nodes causing finality lags, while the team discussed performance issues with recursive aggregation and proof verification times. The group identified potential solutions including increasing block proposal intervals, using lighter XMSS variants, or separating proposer signatures from attestation proofs, with Thomas suggesting continuing the debugging process and potentially increasing slot times in the spec to address performance concerns.
Gajinder reported progress on scaling issues in the Zoom client during Shadow SIM tests, successfully running 32 nodes across four subnets to finality. The team plans to increase node count and test longer slot durations, constrained by server memory limitations. Gajinder also discussed recent technical updates including multi-threading the quick library and merging Zigli P2P fixes, with plans to begin multi-client testing by week’s end. The team decided to simplify the block building process by removing redundant PRs and assertions related to state’s latest justified functionality.
The team discussed progress on DevNet 5 implementation, with Shariq reporting successful interoperability testing with ETH Lambda in one subnet and plans to expand to 24 validator devnets. Mihir presented performance results from testing on an Intel Xeon server, noting that while proof generation takes about 2.8 seconds with one distinct aggregate attestation, block building sometimes exceeds 10 seconds causing gaps in finality. The team plans to increase the number of subnets and add more clients in future testing, and will implement support for PR1166 in LeanSpec.
Pablo reported changes to use the current HESS certifier instead of the global certifier max, along with updates to attestation R-score prioritization and improvements to consensus debuggability. Bankat provided updates on Jean’s performance in multi-client runs with no spec or SSE layout regressions, and mentioned ongoing work to integrate with the shadow network simulator for more reproducible testing.
Boma reported running validators with different subnets last week, noting issues with block building, type 2 verification, and finalization performance. Thomas provided updates on spec cleanup and cryptography work, including exploration of binary fields as a potential replacement for Poseidon, with Emil confirming focus on binary fields research. Thomas requested the team create a document listing current DevNet 5 issues and problems in the DevNet 5 channel.
Gajinder and Thomas discussed stabilizing DevNet 5 and addressing issues with block building and merge proofs. They agreed to continue debugging, merge remaining PRs, and increase the block proposal time interval to allow for testing on smaller machines. Thomas suggested listing problems in the DevNet5 channel and running a month-long network test with hundreds or thousands of validators. Kamil mentioned that ethlambda is shadow compatible and developed a distributed shadow solution for large-scale simulations.
The team discussed implementing sleep slips for simulation instead of true aggregation, with Gajinder explaining that these slips would be configured to log simulated computation times rather than real times. Pablo reported CPU resource limitations on their servers, noting finality lags with 16 nodes running on a 32-core machine, which Thomas attributed to proof aggregation issues. The discussion included potential solutions such as switching to a lighter XMSS variant, increasing machine requirements, or adjusting slot times in the specification, though Thomas noted that significant performance gains would be difficult to achieve.
The team discussed optimization strategies for block proposal times, with Pablo suggesting separating proposer signatures from attestation signatures to potentially achieve a two-fold improvement. Gajinder and Thomas agreed that increasing the slot time might be a better approach than the proposed signature separation, as it would align with the final format planned for the ELSRA hard fork. Thomas proposed continuing debugging and simulating DevNet 5 while increasing the slot time temporarily, with plans to re-evaluate performance improvements in a few months. The team agreed to compile a list of problematic issues for DevNet 5 in the discussion channel, including client interoperability and validator performance.
K$WCTY25)K$WCTY25)K$WCTY25)YouTube recording available: https://youtu.be/aLL96jApx6A