Pectra Retrospective

The Nethermind team have discussed our opinions on increasing the rate at which we ship forks and thought of some ways to improve the process.

Fork Scoping

We should aim to have a provisional plan for the next 5 hard forks, mapping Vitalik’s roadmap into concrete updates. There would still be flexibility to change fork scoping, especially more distant ones, but having this provisional timeline would greatly help in deciding where to allocate resources.

For imminent forks (within a year) we should avoid adding new EIPs to the scope unless they are already implemented and tested; and the current scope is live on a devnet.

Targeting a cadence of around six months per hard fork could help to prevent forks becoming too large. Large feature forks could be alternated with smaller forks focused on UX and quality of life improvements, giving more time to test complex new features; it is also crucial to consider cross-fork dependencies and plan accordingly.

Parallelising Implementation

Having a clear plan can help to parallelise work on future forks. Once a fork is shipped, we could already have the next in devnet phase rather than starting to discuss the scope at that point.

Specs and Testing

EIP champions should have greater responsibility in ensuring their specifications are rigorous in order to avoid differences in interpretation of the spec leading to delays. They should also coordinate with the testing team to create spec tests. Formalising and testing earlier in the process would save time overall.

Adopting EIP versioning (EIP-7577) would help to coordinate implementing spec changes.

Furthermore, more resources could be allocated to testing, and client teams could be more involved.

ACD Efficiency

Client teams should do more asynchronously before ACD to make the calls more productive.

Specifically, there could be a place where client teams can review EIPs, leave opinions, and feedback. We already have Ethmagicians, but we could create a dashboard which is more structured than a forum thread. This would display which EIPs are provisionally scheduled for each fork and could be annotated by client teams with their opinions and implementation status.

Additionally, there should be greater expectations for ACD agenda items to be reviewed before the call where possible (teams could assign members to review different items), to avoid situations where no one has had a chance to review.

4 Likes

This is a rewording of my comments during ACD last week. The problem isn’t the fork size, it’s that forks are bloated with EIPS that seem “small enough” that including them feels like it will not cause any delay in the fork. These claims are often taken at face value.
Core developers get very busy implementing EIPs that are completely unrelated to the big picture of the roadmap, but it feels like something is being done because yet another EIPS is included.

One can add as many committees and smaller decision groups as one wants, the problem will still be the same if there is no commitment to the overall roadmap, and if frivolous EIPs keep getting scheduled.

I agree with the Nethermind team, that there should be a much higher expectation on the quality of the testing and the specification of each EIP. This is exactly what we have tried to achieve with the verkle trees/stateless effort, and yet this has played against it: people realized how complex it was and preferred (putatively) smaller EIPs that felt like we something would ship fast.

There needs to be a way to commit to long term items on the roadmap as a way to signal that work doesn’t get wasted. In other words, we need commitment from core developers and especially the research team, that it is safe to work on extremely challenging and difficult forks. Without these guarantees:

  1. it’s impossible to plan and work in parallel - only the next fork is thought of while everything else is a “maybe”
  2. the amount of people willing to take the risk of working on extremely ambitious forks will decrease, as the costs to do so are high and the rewards non-existent.

This is not a question of decentralization: large, centralized companies run into the same kind of issues as they grow successful.

There’s no easy way to solve these issues, but the few steps below would be a good start:

  • EIP champions should justify on ACD how an EIP advances the roadmap (or Increase the security of the network, obviously). That means EIP that are currently scheduled for Prague or Osaka.
  • Don’t change the road map because twitter panics. Any change to the roadmap should be a month-long process involving many stakeholders, probably during an interop or a major conference.
  • Improve communication between research and core devs. The latest example is inclusion lists (and quantum resistance a close second). While these topics do need to be researched, they have been pushed onto the core development agenda way too early, caused a lot of confusion and had a negative impact on the timelines as teams worked on building them before realizing the idea wasn’t ready for its prime.
  • Commit to forks long-term. This doesn’t mean that any change should be slated for a specific fork ahead of time. This means there needs to be tracks, and if the output of any of these tracks is ready (i.e. devnets running and execution specs written) then they should be shipped in the next fork. We can have 4 forks a year with this (or 0, which means teams are working on adding value to the protocol, and that’s fine).
4 Likes

Two arguments being mixed together here. If an EIP isn’t ready, we should be comfortable moving it to the release after, in order to hit a regular delivery cadence. Confidence in the timing of the next release takes pressure off this decision.

100%. One of the best ways to do this is to plan n+2 ahead. We can move a lot faster on fork n+1 with this in mind also to prevent a lot of rebasing/merge conflict with more fork scope certainty.

However dropping clients (and whole sets of validators who run them) is a far more serious consideration and definitely not something that should become normal.

Definitely wouldn’t be realistic, we need to do multi-layer hard forks together and not leave anyone behind ideally. However, we can also look at where we can direct some CL/EL-only upgrades where entire features aren’t blocked due to X feature that’s not ready for the n+1 hard fork.

The Lodestar team posted a retro on our blog with some of these suggestions/ideas here: Lodestar’s Pectra Retrospective and Future Fork Ideas

If you require some of that within this thread for the AI generator or persistence purposes @timbeiko , let me know and I’ll summarize a version to post in this thread.

3 Likes

The Teku team has shared a document reflecting on some of the points raised by Tim and also some observations from the team. The full retrospective document can be found here. Please read below a summary of our discussion:

  • It is impressive to see how much we have evolved in our testing capabilities, largely due to the support from EthPandaOps and the tooling they have developed. We cannot stop praising them for all their work and how much more productive we are because of their help.
  • It is really good to see many teams working together and collaborating towards making the protocol better. Interop events and Eth R&D workshops have been super valuable.
  • Scope creep has impacted our productivity and workflow and made us take longer to deliver value. We should try to have a better understanding of the features going into a fork earlier in the cycle, and converging into a stable scope sooner.
  • Ideally, every fork should have a clear goal (associated with the Ethereum roadmap). This can help core-devs to understand where to focus and also help us during the process of scoping and prioritizing features.
  • We need to rethink the way we make decisions about EIP inclusion in a fork. Ideally, we want more mature EIPs being scheduled, after teams have had time to understand their value, impact and implementation details.
  • The separation between ACDE and ACDC seems artificial and creates more harm than good. We believe that we should move towards having a unified call, to talk about scope, priorities and governance related to the protocol. Separate “technical” calls can be held between Execution teams and Consensus teams to discuss anything relevant just to one of the layers.

Thanks!

8 Likes

Reth team has gathered opinions on Pectra scoping and development internally, here is the aggregated summary our thoughts:

EIP Consideration

Currently, the EIP is considered ready when it has been summarised, the spec has been defined and sufficient number of community members have discussed it. Then the champion of the EIP brings it on a wider ACD discussion forum and it can be considered for inclusion. The crucial step that we think is missing from this process are test vectors. This would make implementation easier for every client as well as put the authors into the mindset of encoding and possibly discovering the edge cases that their change presents.

Scoping

Pectra scoping has effectively started after Dencun has landed on mainnet and lasted essentially for almost 9-10 months (until the last EIP inclusion). We are strongly opinionated that ACD collectively should:

  • normalize considering EIPs for inclusion for the fork after next (later referred to as fork n + 1)
  • freeze “major” EIPs for the fork n + 1 , by the time fork n lands on mainnet
  • consider committing to a fixed fork schedule (e.g. 2 forks a year)

Testing

We should define a clear set of requirements for clients to be included in devnet on launch. These must include passing all necessary engine Hive test and could include having run an interop devnet locally with 2 or more other clients.

Devnet Parallelization

At this moment, we think that devnet parallelization (simultaneously running devnets for fork n, n + 1, n + ...) is a complex change to the current ACD process and requires more discussions. While we are not against the change, we do think that it introduces a set of challenges/questions including but not limited to:

  • fork activation: should we run fork n + 1 devnet with changes from fork n or should it be an isolated devnet?
    • the former approach puts additional strain on developers to troubleshoot potential bugs from fork n on fork n + 1 devnet
    • the latter approach requires downstream rebase of the changes which is time consuming and might introduce additional bugs
  • what to do with interdependent EIPs across forks?

Summary

We think that first and foremost we should define and change the fork scoping process and are open-minded to future changes with regard to devnet testing.

7 Likes

Thank you to everyone who has shared their perspective on this thread so far!

Putting it all together, there are three overarching themes that emerge:

  1. Increasing the frequency of network upgrades (and limiting scope to achieve this)
  2. Agreeing on clear technical standards for EIPs at various stages of the process, from their initial introduction to AllCoreDevs to their inclusion in an upgrade and eventual deployment
  3. Higher-order process reforms for ACD

I think we can and should make progress on all three tracks in parallel!

Starting with (1), our guiding question should be “what would it take to ship Fusaka ~6 months after Pectra”? My personal answer here is that the scope for Fusaka should be limited to the already SFI’d EIPs: PeerDAS and EOF.

While this may not be the “perfect fork”, it undoubtedly advances Ethereum’s roadmap by scaling L2 blob capacity. Additionally, PeerDAS and EOF would both meet any technical bar we’d want to impose on EIPs for consideration, as they already run on multi-client devnets.

This brings me to (2). Assuming we agree to freeze the Fusaka scope ~now, then the first step before debating the Glamsterdam’s scope should be to agree on what the technical requirements for proposed EIPs should be. Hopefully, we can agree on that relatively rapidly, leaving several months for us to discuss various proposals (and for their champions to get them up to our standards).

In parallel to this, we should have a deeper discussion about (3). I’m excited about the different proposals for more radical rethinkings of AllCoreDevs but want to make sure we don’t rush to implement a whole new system without fully thinking thought through its second order effects. Realistically, it will take many months to get people onboard with a new way of working and I think we should accept that rather than naively expect to overhaul ACD overnight.

Putting all of this together, my tentative proposal would be to:

  • Try and ship smaller, faster forks, starting with Fusaka
  • Agree on a set of minor reforms (like those mentioned above) for the Glamsterdam planning cycle in the next few weeks
  • Kickstart an effort around a broader ACD reform, with the expectation that more substantial process changes would be proposed for the next fork planning cycle, hopefully in ~6 months.
6 Likes

The Prysm team has internally discussed Pectra Retro. Here is a summary of our thoughts :point_down:

Key Learnings and Observations

  1. Underestimating complexity of EIP

EIPs can often appear simple at first glance, but their true complexity surfaces more during implementation (Example: **EIP-7549)**. This experience has taught us the importance of evaluating EIPs and client changes with greater care before making decisions on what to include. This also relates to formalizing specs earlier and having ready-testing infra would help avoid delays caused by misevaluation in spec interpretation.

  1. EIP prioritization did not address community need

Some EIPs accepted in Pectra, in hindsight, did not address the most urgent need today—scaling. Scaling-focused EIPs like PeerDAS should have taken priority over current Pectra EIPs. Every fork may want to have a north-star theme. Reflecting on Pectra, it’s clear how crucial it is to establish a strong and focused theme for each upgrade. Clear thematic goals will help prioritize features and reduce scope creep. Each fork should have a stable scope early in the cycle.

  1. More streamlined decision making process

Certain decisions during Pectra took longer than necessary. Relying solely on ACD calls slowed progress due to their weekly cadence. We should move towards more frequent, open discussions via breakout rooms and Discord async to keep all stakeholders informed. ACD calls should focus on final decision-making rather than being the primary discussion forum. Meanwhile, having a unified call structure for ACDE and ACDC would improve efficiency.

  1. Testing limitations

The primary bottleneck to shipping isn’t building—it’s testing. Multiple client implementation combos and limited testing entropies make finding critical consensus bugs late in the stage. EthPandaOps is simply legendary—there’s no other way to put it—but the reality is they can’t support the testing needs of 11 + more client teams. Kurtosis has proven to be a really useful tool for devnet testing. Greater investment in testing infrastructure within client teams is crucial. Early devnets and parallel streams for layer-specific upgrades could help reduce bottlenecks, allowing testing to happen earlier and more frequently.

  1. Sustaining client development while shipping new features

Balancing research ambitions with practical implementation limits is critical. Client teams not only develop new features but must also maintain code health. Having a known and more visible schedule - planning forks well in advance and locking down the N+1 fork scope when N ships can help reduce the burden on client teams.

Path Forward and Process Improvements

Incorporate More Community Feedback

Client teams and community members should have stronger avenues for sharing input, with greater transparency in how decisions are made. Anyone that wants to participate in a certain topic can be informed but there exists a deadline, it’s on you if you miss the deadline and didn’t provide your input. Creating a structured dashboard for EIP reviews and feedback could improve the process compared to scattered forum threads. Based on this feedback, as a collective we can better weigh EIP priorities.

Shift ACD Focus

Use ACD calls for final decisions while encouraging more efficient offline discussions and asynchronous reviews before calls. This will help avoid long presentations and in-depth technical debates during meetings.

Improve Testing Infrastructure

Prioritize expanding testing capacity and recruiting more qualified testers on individual client teams level to reduce bottlenecks. Define clear requirements for EIP inclusion, including passing necessary Hive tests and running interop tests with other clients.

Smaller and More Frequent Forks

Target a cadence of one hard fork every six months, alternating large feature forks with smaller UX and quality-of-life improvements. This schedule would reduce social pressure on each fork and provide more time for testing complex features.

Parallel Devnet Strategy

Start devnets for future forks earlier in the process. While testing one fork, begin preparing a rolling devnet for the next fork to explore and spec target EIPs early. This approach would minimize delays caused by late scope discussions.

Emphasize EIP Champion Responsibility

EIP champions should take on greater responsibility for ensuring the readiness of their specifications with sufficient spec tests, coordinating with testing teams, and justifying how their proposals advance the roadmap or improve network security.

3 Likes

Lighthouse Team Feedback

  1. Early and Aggressive Scope Freezes
    We support a stronger commitment to early scope freezes. Scoping out fork N+1 while N is being shipped, and freezing N+1 when N is shipped sounds like a good framework. Although there should be rare exceptions for critically important features.
  2. Governance and Call Structure
    We think separating “scoping/roadmap” discussions from deep technical dives could streamline the process. This would work will with the parallelization of fork N+1 scoping and fork N development. The technical calls should probably still maintain the CL/EL division, but the scope/based calls probably should not. In this model, or even in the existing model, more forceful moderation on topic direction would help.
  3. Steering committee
    We’re cautious about adding bureaucracy and believe many of the benefits could be reaped by separating scoping from technicals calls, and harsher moderation.
  4. Frequent Forks
    Although each fork introduces process overhead, we hope we can make this more efficient. Parallelizing fork N+1 scope with fork N development reduces per-fork overhead. More frequent forks will also relieve pressure on getting EIPs in for fork N if N+1 doesn’t lag as far behind.
  5. Finding a “North Star”
    Without a unifying milestone like the Merge or Withdrawals, it’s not always clear what to prioritize. We see the next big push as PeerDAS. The “beam chain” is a bit further down the road. In between there is uncertainty about which EIPs or improvements should take precedence. A bottleneck here, is the huge amount of work it is to stay abreast of all research work, and with enough depth to have informed opinions. We believe more structured polling or active outreach to teams could help break this gridlock.
  6. Slow-Client Delays
    There is some threshold after which we must move forward without a client. This would have a major impact on users of this client and migration path for them should be made available. We will also need to weight the landscape of client diversity after this client is dropped. However when we’ve reached thresholds of “client diverse enough” with a low-enough impact on users, we should move forward.
3 Likes

Feedback from the Besu team:

  • Some EIPs were not specced sufficiently and delayed the fork because several rounds of spec changes where necessary along the way
  • Scoping Pectra took way too long. Freezing the SFI list should happen very early in the planning process
  • EIP-7742 was dropped too quickly. It might make sense to revisit it in the future to allow CL only hard forks
  • PandaOps and testing teams are top notch
  • Gating devnets seems like a good idea, but wasn’t strictly enforced. Seems to still make sense to do it
  • Hive tests > EEST tests. Hive uncovered more bugs in our case, because EEST does not execute100% of the code path. Hive tests should be used for gating devnets.
3 Likes

In my opinion, this hard fork was messy and full of things that do not benefit Ethereum neither in the short term or in the long term. The only improvements that are tangible is native AA and the increase of the blob count. The rest of the EIPs were unnecessary and most were only refactorings which might as well never happened.

It is clear to me there is a problem in the decision-making part of the protocol rather than in the “development”. I think the development is fine but the decision making is too fragmented.

Fragmented decision making

The biggest issue isn’t development—it’s the growing inefficiency of decision-making caused by too many people trying to make governance calls at once.

  1. Too many visions for Ethereum

    • Decision-making is slow and chaotic when 100+ people are involved. Discussions often get derailed, wasting valuable time.
    • Large groups struggle with agility—a smaller, focused decision-making body would be faster and more effective.
  2. As Ethereum grows the inneficiency will get worse

    • As Ethereum’s ecosystem expands with more developers, client teams, and stakeholders, the chaos in ACD calls will scale with it.
    • Without a structured governance process, decision-making will continue slowing down, leading to unnecessary delays in upgrades and forks.

A Better Path Forward

Like @asn said, Ethereum needs a *smaller, structured decision-making body handle governance, while keeping ACD focused only on technical discussions.

Without this change, ACD will become increasingly ineffective, and major decisions will be made informally outside the calls, turning ACD into a symbolic process rather than a functional one. If Ethereum is to keep evolving efficiently, governance must scale as effectively as development does.

1 Like

Links to Pectra retrospective responses

Consensus Layer teams

Grandine

Lighthouse

Lodestar

Nimbus

Prysm

Teku

Execution Layer teams

Besu

Erigon

Geth (individual responses)

Nethermind

Reth

Community

1 Like

This was Grandine’s first hard fork where we fully participated in the process, so it’s difficult for us to compare it with previous forks. However, we have a few key insights to share:

Lack of a Clear Flagship Feature

Pectra ended up being a hard fork with many small improvements but no strong central message, or direction. We believe hard forks should be driven by a single flagship feature, similar to how the upcoming PeerDAS fork is structured. Additional smaller changes should only be included in exceptional cases - ideally, only if they don’t delay the shipment of the flagship feature. This approach would help control scope and accelerate fork releases.

Stronger Requirements for Non-Flagship EIPs

Any non-flagship EIPs should only be included if the proposer has implemented them in a real client and they are significantly integrated into a few ecosystem projects. This ensures that we fully understand the impact and potential friction introduced for users. It would also help prevent seemingly simple “single-line change” EIPs that, in reality, require extensive refactoring in clients and major adjustments in dependent community projects.

Faster Fork Releases Through Parallel Development

Forks need to be shipped faster, and the only viable way to achieve this is through parallel development. PeerDAS is a great example - Pectra hasn’t even been deployed on the testnet yet, and we’ve already had multiple PeerDAS devnets. However, for this model to work, teams need to be large enough to split internally so that dedicated sub-teams can focus on individual forks without key contributors being stretched across multiple efforts.

1 Like