Forming a Ring: ETH v64 Wire Protocol Ring

[matthalp]

Hi everyone! As the designer of the LES protocol I am really glad to see that some of you support adding my message format extensions to ETH64

@zsfelfoldi Thank you for coming up with these ideas! I enjoyed reading through them and thinking about how they would be used. Nice work!

I would also strongly suggest using my handshake message format which is a general key-value list allowing peers to communicate extra parameters and capabilities in an easily extendable way.

This is a good suggestion! This would be a general purpose way to accommodate the " Making Pruned State Explicit" proposal. I do think that there should be some mandatory fields, especially to address the networkId/chainId issue raised by @ajsutton. However, these mandatory fields can be moved to be mandatory keys.

The most complex addition of LES is the flow control mechanism which I believe has already been more or less proven to be useful but I am currently polishing it to truly show its potential. I would totally support making it available in ETH64 but on the other hand I do realize now that the way it is currently used in LES is probably also too strict and brittle.

I agree that “Consider Incorporating Client-side Flow Control” is something to have in the long-term. While it may not be there initially, the goal of the “Encapsulate Metadata as a Header Rather than Inlining Before Payload” is to make it easy to add metadata fields like this in the future.

Recently I had a very useful discussion with FrankSzendzielarz and he suggested adding a message similar to http 503 (temporarily unavailable) or 429 (too many requests) and sending that in case of a buffer underrun instead of instantly disconnecting.

This is an interesting idea! I wonder how this would work end-to-end. Would the message contain some information to help the almost-spammy peer know when it can begin sending messages again? It seems like there needs to be some explicit agreement set for what will help the almost-spammy peer’s behavior improve.

Having a common general protocol format would even allow us easily merging LES and ETH64 which was also suggested in this thread and which I would also support if done properly.

I was somewhat worried that " Folding LES into ETH" would be controversial. I’m glad to hear this is open for consideration.

[mbaxter]

One idea that could simplify peer management is to add messages for requesting chain state information. Peers initially report their chain’s best hash and total difficulty via the Status message, but keeping up-to-date stats on peers is otherwise somewhat indirect.

Some possible request / response messages:

• GetChainState()
• ChainState(blockHash, blockNumber, totalDifficulty)

And it may also be useful to verify total difficulty claims by querying other peers with:

• GetTotalDifficulty(blockHash)
• TotalDifficulty(blockHash, totalDifficulty)

[zsfelfoldi]

@matthalp here is the proposal I promised:

Issue: Modular Protocol Framework proposal

[matthalp]

Great! I will review and comment there.

[matthalp]

@mbaxer I think it makes a lot of sense to have explicit chain state request/responses instead of relying on receiving the NewBlock/NewBlockHashes methods from peers. It’s also worth noting that chain state updates changes from a pub-sub type model to a polling model. There are some slight trade-offs here but a client typically communicates with at most 25 peers at a time, so there probably isn’t the scale to make any of the trade-offs notable.

The total difficulty messages are also interesting, but I have to think that through more. This morning I was wondering why headers don’t just include total difficulty instead of just the difficulty at that block, which may also solve some of the issues here (and it’s sealed in the header).

[matthalp]

Just adding this here to not forget: It is probably worth removing the bloom for each receipt in the Receipts message since that information can be reconstructed from the block. Downloading receipts in fast sync is already pretty bandwidth-intensive and receipts consist of the outstanding majority of data downloaded as compared to the actual account states.

[veox]

@matthalp

It’s also worth noting that chain state updates changes from a pub-sub type model to a polling model. There are some slight trade-offs here but a client typically communicates with at most 25 peers at a time, so there probably isn’t the scale to make any of the trade-offs notable.

I run a LES lightserv geth node that typically handles ~500 peers. It used to do ~1500 at the height of the KryptoKitty boom.

Changing from PUB-SUB to polling will have a negative impact on a minority of nodes in the network; a minority I wouldn’t flinch calling vulnerable.

[matthalp]

@veox Thank you for pointing this out. I was specifically talking about ETH peers but had not discussed LES peers.

In LES there is an Annouce message that does broadcast the head hash, number, and TD (and some other metadata), which solves some of the issues @mbaxter is alluding to. Perhaps it is worth adopting this message in ETH if we do consider merging ETH and LES. Pub-Sub would remain under this paradigm, which I think does make sense.

[jpitts]

Sorry to break in off-topic, but is there a document which describes the major advances in the Wire Protocol, or events which led to certain important upgrades?

I created a versions doc for the Protocol itself, will soon create one for the EVM. It would be great to be able to do this for the Wire Protocol.

ethereum-specification/docs/blob/master/releases.md

[matthalp]

@ferranbt Can you link to where the message sizes are specified in Parity and Geth? My understanding is that the maximum message size should be 16 MB (uncompressed).

I think we should try to get this standardized if there is ambiguity here or at least have a discussion about why they are different.

[ferranbt]

There is a limit in the number of objects (headers, receipts…) or bytes sent per request (soft limit) in Geth and Parity. However, those limits are different in both implementations: max number of objects (Geth, Parity) and soft limit (Geth, Parity)

[FrankSzendzielarz]

Related to this is a new proposal which would facilitate this I think: A cross-protocol, cross-implementation standard for server capacity management and flow control

[matthalp]

@ferranbt From an implementation perspective it seems like you could be conservative and allocate the maximum 16 GB that is agreed upon between the clients. Less conservative would be to take the upper bound between the two.

I’m not sure I can see doing better than that. The number of domain objects or the soft limit are really implementors’ choice. If you have strong feelings I’d reach out to the client developers either through issues on their repo or through one of the messaging systems they use.

[matthalp]

I talked offline with @zsfelfoldi and it seems like folding in LES to ETH is going to too ambitious for ETH v64.

Generally speaking about generality with peer-to-peer networking: some of the proposals for generalizing concepts smells to me like something that should be modifications to the RLPx wire protocol itself and not sub-protocols. The specific cases here are the (1) request IDs, (2) client-side flow control, and the general key-value list. I think request IDs can be addressed this round, but client-side flow control and generalized key-value list may need to wait. Personally, I am not a fan of ambiguity in specifications as they can be abused. I like adding things with specific use cases as needed. Implementors willing to put in the work to make the changes demonstrates that it is worthwhile.

[matthalp]

Thank you everyone for your contributions! There are a lot of great ideas here.

I want to take a moment to establish the scope of ETH v64:

1. Protocol warts: remove glaring holes, ambiguity, technical in current ETH v64 protocol (possibly at the RLPx wire level)
2. Synchronization: make it easier to write synchronization code
3. State: make client state more apparent amongst peers (e.g. archive vs. full) and even reduce data transferred (receipt bloom filters I’m looking at you)

Over the next few days I’m going to draft “minimum viable EIPs” for the ideas that seem like can be candidates for ETH v64. Each of these EIPs will have proper attribution to those who proposed it. I will ping either author to drive it hence forward unless they are not interested. That is not to say the ideas I do not draft into EIPs are not worthwhile, I just do not see them within the immediate scope of the three goals for ETH v64 mentioned above.

Let mebe explicit that amongst my proposals on here I only consider myself an author of Making pruned state explicit as the others are previously established ideas from prior works.

The next step is then getting client implementor buy in. This seems to be the right place to get consensus because (1) they are the implementors and (2) this is so low-level that other stakeholders higher in the stack will not even notice.

[FrankSzendzielarz]

Yeah I am not sure what the obstacle would be. IMO flow control and capacity management should not be part of the light protocol at all, but a cross cutting concern affecting all protocols and servers that require quota and rate limiting and eviction policies.

I don’t see any particular great need though to merge LES with ETH.

[matthalp]

I don’t see any particular great need though to merge LES with ETH.

As mentioned above I think over time we will see the lines blur between an ETH and LES client. At the ETH level we already have full- and fast-synced nodes and hiding their differences does cause strain on being able to sync. My rationale was that if clients are already having to think about the full-/fast- distinction then light clients only extend the spectrum.

Wearing my go-ethereum hat, I just feel like there could be some code consolidation to make things more maintainable. But perhaps that is a nit.

[FrankSzendzielarz]

Check out the linked proposal. I agree there are commonalities, but I think the same thing applies to Whisper, Swarm etc., and capacity management is a problem for the whole server. Why have rate and quota limits applied to specific clients alone, when the same logic can be set as a server limit too? So rather than focusing on merging LES into ETH to take advantage of rate limiting, I propose taking rate limiting out of LES and making it a general sub-protocol.

[matthalp]

I’m with you on rate limiting. I was moreso speaking about the actual messages supported by LES and ETH since they are both handling blockchain and account state constructs (plus the additional cryptographic constructs in LES). Whisper and Swarm have fundamentally different messages.

[yperbasis]

I’ve written a paper about a new sync protocol and algorithm, Red Queen’s. See also my presentation at Ethereum Core Devs Eth1x/Istanbul Planning Meeting in Berlin. It’s essentially an attempt to formalise Leaf Sync.

We can discuss it here or in a dedicated thread,

Red Queen's: New Sync Proposal

As part of Ethereum 1x efforts, we want to tackle the issue of the growing blockchain state. Large state size leads to long sync times. On top of that, geth’s fast sync is not that fast and takes a few hours. Parity’s warp sync is faster, but is not supported in geth. We propose a new sync protocol and an algorithm, which we believe can perform snapshot sync in less than an hour. The idea is somewhat similar to Warp Sync and especially to Leaf Sync (under development in geth). Moreover, the new …

Happy to join forces with anybody working on new sync protocols.