Meeting Summary:
The team held their 33rd PQ Interop meeting where various team members provided updates on client developments, DevNet activities, and metrics analysis. Key discussions centered around network performance metrics, stability issues, and plans for improving validator operations including potential mesh topology changes and aggregator rotation. The team also reviewed progress on automated testing tools and interop testing, while noting increased contributions from new collaborators and discussing upcoming events and the rescheduling of the next meeting.
Click to expand detailed summary
The team held PQ Interop meeting number 33, with Unnawut coordinating due to Will’s travel. Anshal provided updates on Zoom, including merging DevNet 3 metrics VR, updating payload and gossip signatures, resolving checkpoint sync issues, and implementing networking updates for robust gossip propagation. Shariq reported on DevNet 3 metrics updates and sync issue fixes, with work ongoing on key separation PR and potential interop on DevNet 4 by week’s end. Ruslan mentioned no updates for Kulin but noted progress on automated interop testing tools.
The team provided updates on various client developments and DevNet activities. Mercy reported internal fixes including patches to synthetic anchor blocks and attestation data checks. Shaaibu announced completion of DevNet 3 and participation in the multi-client interop run, while also beginning work on DevNet 4 specifications. Pablo’s team worked on metrics improvements including station committee tracking and table size metrics, and continued development on DevNet 4 PRs. Mihir shared updates from Lantern, including bug fixes based on DevNet feedback, improvements to syncing, state pruning support, and DevNet 4 PR support. Katya reported that DevNet was running stably with some clients experiencing stalls, and noted that new clients needed Grafana dashboard access.
Unnawut presented metrics analysis from their DevNet using an observatory tool that compiles Prometheus metrics into Jupyter notebooks. Key findings included acceptable PQ signature times (16ms average, 32ms at 99th percentile) but concerns about low signature aggregation rates (1-5 signatures per slot for 13 validators), frequent reorgs (up to 12), and finalization delays of up to 82 slots. The team discussed plans to investigate missing metrics, increase validator count for DevNet 3, and potentially rotate aggregators in future versions, with Parthasarathy confirming that rotation is already ready in the lean-quick start implementation.
The team discussed network metrics and client stability issues, with Katya proposing the need for syncing metrics to track which clients are currently syncing. Thomas provided updates on research efforts, including improvements to Clonki 3 and LeanVM, as well as ongoing work on DevNet 4 features and a new ZK-friendly encoding. The team noted an increase in contributions from new collaborators on good first issues, which has helped split the work and improve code testing. A question was raised about potential impacts on LeanSpec due to a supply chain attack on Python packages, but the team was unsure of any specific effects.
The team discussed plans for a long-running DevNet with multiple subnets, where Parthasarathy proposed introducing a new mesh topology for validators and requested client implementation of this feature. Thomas explained he was avoiding major changes to the spec due to complexity, particularly around the new LeanVM update and recursive aggregation, but would address testing coverage and simplify complex nested logic in the future. Anshal suggested removing the greedy aggregation part as an optimization and proposed adding recursive aggregation functionality. The team also discussed an interop test repository shared by Ruslan, which implements automated network testing scripts, though some clients like Lighthouse don’t support devnet-3 yet. The next PQ Interop call was rescheduled to April 8th due to HCC, and team members were informed about upcoming events in Cairns including Beast Mode and Fort Mode.
Next Steps:
- Katya: Share logs and restart stalled clients after the call
- Katya: Provide Grafana dashboard access to new clients who DM her
- Anshal: Create a PR on top of Thomas’s cryptography PR today
- Parthasarathy: Raise a PR on Lean Spec for mesh topology implementation later today or tomorrow
- All clients: Review and implement the mesh topology PR for multi-subnet testing before DevNet 4
- Mihir (Lantern): Have new image available with checkpoint sync fixes by tomorrow
- Mihir (Lantern): Look at the two or three additional PRs for DevNet 4 this week
- Thomas: Merge DevNet 4 PRs including Poseidon 1 switch and new ZK-friendly encoding along with LeanVM update
- Pablo and Lambda team: Communicate date for remote side event (Twitter Space) for HCC via Telegram
- Ruslan: Work with clients (Lambda and Lighthouse) to resolve issues running interop tests - requests contact for discussion on arguments/Docker images
Recording Access:
NO CALL ON WEDNESDAY APRIL 1 — EthCC. NEXT CALL: WEDNESDAY APRIL 8 @ 14:00 UTC.
Complete | 
In progress | 
Not started | 
Unknown