All Core Devs - Testing (ACDT) #46 | July 28 2025

Protocol Calls & happenings
,
5

ACD Testing Call #46 - Quick notes.

Moderator: Parithosh Jayanthi

Next Steps

  • Devnet 3
    • Network is live.
    • Launch BPO-4 tomorrow to reduce blob count.
    • Continue investigation into Nimbus peering and orphan block issues (Pari, Bharath).
    • Barnabas to share async notes.
  • Eth Config
    • Danno to update PR removing JSON hash from eth_config.
  • Gas Limit Testing
    • Alexey to analyze secp256r1 gas pricing.
    • Teams to report peer-related anomalies; Marius to review Geth behavior with PR logs.
    • Investigate Nethermind peer discovery issue (Csaba, Raul).
  • BloatNet
    • Geth team to finish tracing/memory fix.
    • Client teams to review Cperezz’s test plans and align with EELS integration.
  • Benchmarking
    • Mario to finalize EEST benchmark tests and resolve blockers.
  • Sunnyside Labs
    • Continue large block size propagation tests.

Fusaka Devnet 3 Updates

Parithosh shared:

  • Devnet 3 launched late last week.
  • Network is finalizing with 3 BPOs already completed.
  • Live tracking: Dora Explorer
  • Blob Configuration**
    • Currently running at 18 blobs per block.
    • BPO-4 will reduce blob count (currently at 18 blobs/block).
    • BPO-4 expected to go live tomorrow.
  • Added rate limiters:
    • 30% nodes are gigabit “super nodes”
    • Rest are 100 Mbps down / 50 Mbps up
    • Based on EIP-7870
  • Validator Key distribution is uneven; staking top-ups added for addditional test coverage.
    • Network includes every client pair.
    • Top-ups used to normalize validator stake across the board.
  • Clients experiencing:
    • Nimbus EL/CL unable to connect to bootnodes
    • Higher-than-expected orphan blocks (no pattern, possibly MEV-related)
  • MEV updates (Bharath):
    • 672 validators have successfully registered for MEV.
    • Block delivery confirmed from the latest slot.
    • Investigation ongoing into node connectivity issues:
    • Initial findings point to a problem with Lodestar.
      • Block publishing for Nimbus is currently failing
      • other than these two issues, regular blocks are being delivered consistently.
      • MEV Relay Dashboard

Other discussions

  • Manu: Do all nodes have the same rate limiting, or does it depend on the node type?
  • Pari: It depends. Full nodes are capped at 100 Mbps, while super nodes have more. All specs are per EIP-7870. If those limits prove insufficient, we’ll flag and evaluate adjustments.
  • Pari: Nimbus EL + CL clients are facing peering issues, unable to connect to bootnodes. Bharath will investigate this asynchronously.
  • Pari: There’s a higher-than-expected rate of orphan blocks. No consistent client is at fault — random distribution, no clear pattern. The only major change from previous devnets is the presence of MEV.
    There was an issue where aclient block tried to build on an old parent. Root cause is still unclear. Block link

Next steps:

  • Bhartah will continue to investigate Nimbus and other issues.
  • Barnabus will share his notes async.

Eth Config Discussion

  • Mario
    • Noted that eth_config is now available.
    • Testnet and mainnet configurations need to be reviewed.
    • EEST (Execution Engine Spec Tests) should understand the config; queries can be placed during the testing phase.
  • Pari: Asked if RLP serialization is in progress.
  • Marius
    • Raised concerns about hashing JSON-RPC responses.
      • If all clients return the same JSON-RPC result, it can become brittle.
      • Suggested we find a better way to hash the data.
    • Prefers RLP, but open to SSZ if other clients prefer it.
  • Danno
    • Questioned use of RLP: “If we shouldn’t use RLP, why go that route?”
    • Later suggested dropping the hash entirely.
  • Tim Asked: “Why are we not supposed to use RLP?”
  • Roman: Clarified that the Execution Layer (EL) currently only supports RLP, not SSZ.
    • Said dropping the hash might complicate things for testing teams.
  • Mario Vega: Suggested testing teams can perform the hashing themselves if needed.
    • Referenced EIP-specified JSON standardization: RFC 8785.
    • Personally liked the idea of including the hash.
  • Danno: Mentioned another change
  • Pari: Confirmed it seems like a similar change; no objections from the group.
  • Danno will include the change in the updated PR.

Decision:

  • Drop the hash from the config.
  • Danno will update the PR accordingly.

Next Phase Testing - Timeline Discussion

Pari mentioned

  • Shadow fork and large testnet are in planning.
  • Reference doc: Fusaka Devnet 3 Notes
  • Alex stokes: Urged to begin the next phase of testing as soon as possible.
  • Pari: Emphasized that client teams should focus on orphan block issues and other topics discussed earlier in the call.

Gas Limit Testing Updates

  • Pari updated
    • Mainnet is currently running at 45 million gas with no major issues noted.
    • Investigation into Consensus Layer (CL) validator behavior is underway, and analysis will be shared soon.
  • Alexey
    • Plan to check the real gas cost of secp256r1 this week.
    • Expressed doubt about time availability to update the pricing if anything significant is found.
    • Asked if other clients observe similar behavior; requested that findings be reported.

Peering Issues

  • Marius van der Wijden
  • Csaba
    • Highlighted a peer discovery issue involving the Nethermind client.
    • Problem appeared about 5 days ago.
    • This testing has only been live on mainnet for less than a month, so needs further understanding.
  • Raul
    • Suggested sharing peer IDs to investigate potential common patterns.
  • Pari
    • Both Nethermind and Geth issues appear to be unrelated to the 45M gas limit setting itself.

BloatNet Testing Discussion

  • Cperezz
  • Marius van der Wijden
    • Reported a serious issue on BloatNet with Geth:
      • When Geth detects a state change, it tries to dump all states.
      • This behavior consumes a lot of memory, causing nodes to crash.
      • Geth had 55 node crashes, and the database got corrupted.
    • Root cause: nodes ran out of memory during tracing.
    • Fixes implemented:
      • Working on tracing with less memory usage.
    • Shared infrastructure resource usage:

      “We allocated ~10TB because of tracing and <1TB of RAM due to other chain operations.”

  • Pari
    • Confirmed resyncing the node from the corrupted database has been started.
    • Will investigate tracing issues further.
  • Mario
    • Commented on the limitations of state growth testing:
      • Tests produced cannot contain very large state due to technical constraints.
      • Most current tests are computationally intensive rather than focused on state growth.
    • Highlighted the need for state-specific benchmarking.
  • Cperezz (follow-up)
    • Emphasized that the new tests are more state-related.
    • Ideally, such tests should be executed using BloatNet state.
  • Marius van der Wijden

“It would be ideal to write the tests in a way so they can be replayed on arbitrary networks.”

  • Pari
    • Shared that the Nethermind team has started EELS integration.
    • Suggests continued asynchronous coordination on integration efforts.

EEST Integration, Benchmarking, and Testing Reports

  • Mario
    • Announced a new release: EEST Benchmark v0.0.3, which includes all benchmark tests.
    • Noted that Nethermind tools can now use genesis files.
    • Stated that genesis will be the starting point for future benchmarks.
    • Shared that this is still a work in progress.
    • Blocker: All benchmarks still need to be functional and verified.

CLZ Benchmark Results

  • Louis
    • Presented findings via this slide deck.
    • Shared methodology and test notes.
    • Tests based on EIP-7825 and focus on worst-case block sizes.
    • Two gas limit scenarios tested: 72M and 100M.
    • Call participants were generally supportive of the CLZ benchmark methodology.
  • Pari
    • On reducing gas pricing, he recommended sticking to 5.
    • Mentioned the intent to reduce pricing in the upcoming Glamsterdam fork.
  • Mario
    • Noted the benchmarking methodology:

      “Use EELS first, then test across client implementations in future forks.”

Sunnyside Labs Testing Update

  • Minhyuk (Sunnyside Labs)
    • Provided a progress update:
      • Running two testnets with regular transactions to monitor how larger block sizes affect network data propagation.
      • Began test execution.
      • Teku is currently the only client supporting backfill, though some issues remain.
      • Team is working on resolving backfill issues and aiding with broader test coverage.
    • Report will be shared on Discord once ready.
    • Confirmed that the Perfect Column Devnet appears to be functioning correctly at first glance.

Any corrections tothe notes, please add here