If I understand what you are suggesting, it is that we should not use RLP for the envelope and instead just have 1 byte for the version, followed by remaining bytes for the payload? Presumably, we would reserve 255 for future expansion (e.g., extension value)?
If so, I’m not against it and I think @AlexeyAkhunov has argued in the past that we shouldn’t be using RLP as much as we do. I would like to get feedback from client developers before making such a change though, as there is value in everything in a protocol using the same serialization format as it generally makes client development easier than when you have custom protocols for different parts of the system.
In this case, it may be worth it to avoid having to have an extra 1-4 bytes on every transaction.
I think that forcing clients to include multiple serialization schemes to be minimally consensus-compliant is not the direction we should go. Even if the savings is non-negligible. Eventually we should move away from RLP, but in the interim, I believe we should accept what we have.
I am recognizing a fork in the road and discussing why a particular path may help with our scalability problems. What you have done is awesome, and I have already built an EIP from your great idea.
With your proposal unchanged a developer is free to define an transaction such as; rlp(‘b’,blob ) - now because this blob is more than 55 charicters, RLP will force us to burn to burn three bytes just to describe how long the blob is: (e.g. xb9, x04, x00) - with the addition of the list and version param that is a 5 byte overhead - not ideal and negates much of the savings.
I love that rlp(‘d’) == ‘d’, and another reason to like this feature is it creates the possibility of polyglot encoding. Because all transactions start with an array element, we know that no other transaction is going to start with 0x00-0x7f - this is reserved by RLP, and now in the future could be reserved a byte-boundary encoding. If a wayward client accidentally decoded it with RLP - it would still produce an object! (The wrong object mind you, but an object none the less).
It is almost as if the protocol left the door open for a single-byte version param. With a single byte as the type envelope, the basic transaction is a cool 138 bytes.
To @matt’s point - I agree, we shouldn’t burden developers with features that aren’t important. In this case byte-boundary encoding of 9 param will take maybe an afternoon to impalement - it is just cutting a string in 8 places - no big deal. We might be able to pack an extra byte or two with bit-packing, which is one step further and you’ll see this asn.1 and protobuf for encoding of enums and booleans - but that is unnecessary effort for this data structure.
You aren’t the only person who is dissatisfied with RLP encoding for everything. However, until such time as we have a new standard encoding mechanism that we can start applying everywhere I’m very hesitant to switch to something besides RLP here, even if it isn’t as efficient as other options. You may want to consider championing a push to change the encoding mechanism used throughout Ethereum as I believe there is weak support across the board. If you do, you may want to start by talking to ETH2 developers to see what they are planning on using, in case they already have a plan to switch away from RLP.
A question was proposed by @AFDudley about how 2718 will deal with pending transaction pools, since new transaction types may introduce complexities into the transaction transactions, transaction ordering, etc.
My initial thought on the matter is that 2718 itself doesn’t introduce any problems with pending transactions and transaction ordering since by itself it only has a single transaction type that is just like the legacy transaction type. New transaction types however may have a problem with needing multiple pending transaction pools and figuring out how to do transaction ordering, and so discussion on the matter should probably go into those EIPs (e.g., 2711, 1559, etc.) rather than in 2718 (where we don’t yet know what problems we’ll need to solve).
There was a follow-up suggestion that we perhaps add this to the security section just as a hint to future typed transaction EIP authors that they need to think about that. I’m not the fence about this idea currently, as it still feels out of scope for this EIP, but I can appreciate the line of thinking.
What do other people think about the matter? Should this EIP try to discuss how pending transactions and transaction ordering among multiple transaction types works, or should that problem be left up to future EIPs that introduce new transaction types?
I’m not sure if this was mentioned before, but a few months ago I was working on a similar proposal and hoped we could move on from using RLP to either CBOR or SSZ.
If we want to do that, perhaps we should not wrap payload into RLP, instead we could choose to use a single byte as transactionType. Could also consider this field as “version”. This allows for 256 different types, which seems okay. This would also mean that signing with transactionType = 0 would be easier, than with the proposed extra RLP wrapping.
@axic I like this. Removal of anything that is strictly based on RLP if it can be done in a sane manner has my support. I would advocate for LEB128 incoding for the transaction type which allows us 128 transaction types before we end up needing 2 bytes. All of the remaining bytes can be the opaque encoded transaction.
Right, LEB128 would be a good way, however if we say the first byte is a transactionType (or version) limited to 0x00 … 0x7f, then we do not need to introduce LEB128 as a complexity. If the need arises in the future, we can still introduce it (because any value below 0x80 has the same encoding in LEB128).
My main concern with moving away from RLP is that it introduces another dependency for clients to maintain and moves us in the wrong direction in terms of client simplicity.
there seems to be a lot of motivation/interest to get rid of RLP at various levels, which is likely to happen with binarification, code merkleization, etc.
certain kinds of integrations with Eth2 will introduce SSZ
Because of the above I think we could see a rather large change in the coming years, essentially getting rid of RLP.
Dapps only interface via RPC (which is not using RLP) and the transaction format (which we are about to change). Dapps do not use RLP for other purposes, IIRC all the ERCs use ABI encoding or an alternate format (with the exception of ERC712).
@matt so far all of the places where I’ve pushed us to drop RLP don’t introduce any new dependencies, but rather a simple binary encoding scheme and I believe that this case is no different.
@rook@axic@pipermerriam and several people in Discord have all now expressed a preference for not-RLP. While several alternatives were proposed, I have decided to initially update the EIP with a simple “first byte is transaction type, the rest is payload” encoding scheme so we have something to argue against besides something no one wants (RLP). I have mentioned in the specification that transactions type 0x00 through 0x80 are the only valid transaction types, so that we keep the door open for extending the type byte in the future if we need to.
I have also added a note that the transaction root and receipt root in the block will change, and I specified how to generate them. Part of this change includes switching the key from rlp(index) to leb128(index) as part of a push to move away from RLP. I’m not married to this, and if people think that change isn’t appropriate or will cause undue hardship I have no problem switching it back (though, we still need to change the other side of the mapping).
We could make it so enveloped legacy transactions are not RLP encoded, but you still would need an RLP encoder to sign them or validate their signature, so the benefits of using some other encoding system for putting them on the wire or including them in a block are pretty weak. I’m curious if people think there is value in having transactions encoded with something else on the wire and in a block, even though you need RLP for signing/validation?
I also did go with SSZ encoding for the receipts. I think this may be the first introduction of SSZ into ETH1x, so may be contentious and worth discussing further. I believe ETH2 is using SSZ everywhere, and that likely means that ETH1x is eventually going to need it. Also, SSZ is nicely compatible with the TransactionType as first byte encoding system.
I am glad the EIP is using a first byte as a version string, this cleanly resolves the ambiguity of messages in O(1) by using more than just block height and will make a better EIP over all. I guess this leaves the door open for version 0x01 a compact transaction type that is both easier to process and smaller than leb128. leb128 is an unnecessary burden when ECC public keys and signatures are essentially fixed width, you don’t need to burn bits on variable encoding if you know the size. Allowing for bigger keys can be done with one of the 127 version params we have.
At the moment, transaction type 0 is rlp so that we don’t have to decode/reencode to validate signatures or sign, since type 0 is intentionally backward compatible (signature wise) with legacy transactions. Future transaction types would be reasonable to use something like SSZ, which is positional (requires a schema to decode).
Hello, I asked on geth PR here about transaction concatenation and got a response that it is already decided that it will be concatenated. I have more questions if it is okay.
By iterating over bytes that represent a list of transactions we need to read tx type then get tx payload (or legacy rlp) and process it. Payload needs to contain its size or have static size so that we can deduce when the next item in the list is. This is how we need to do parsing, is this correct?
Question? why didnt we just reuse rlp as wrapper over TypedTransaction? List will be like: rlpList[ rlp(TypeTx | DataTx), rlpList[Legacy] ...]. Partially breaking RLP is by my opinion not best solution, if in the future there is need to replace RLP this will be a different task.
There is a desire to move away from RLP for transactions over time, and in the future we don’t want to require everyone have to do RLP decoding/encoding for transactions that are otherwise not RLP encoded. By having the first byte represent the transaction type and the remaining bytes be the payload, we are not tightly coupling with any encoding/decoding scheme as you can simply do something like transaction[0] and transaction.slice(1) in every language trivially without any special libraries or needing to understand a particular encoding scheme.
Also, the most likely encoding format for transactions in the future is SSZ, which is compatible with byte || data as that is equivalent to ssz(byte || data).
I believe you are correct that when decoding an RLP list of transactions you’ll need to do something like:
I do recognize the issue here in that currently devp2p is RLP encoded so things are a bit simpler for that particular transport if the transaction is also RLP encoded. However, transactions are made available in many other places (e.g., JSON-RPC, hardware wallet protocols, etc.) and requiring everyone everywhere to have the ability to encode/decode RLP just to extract the type byte isn’t a great story (if you consider that future transactions likely won’t all be RLP encoded).
I agree that with generic transaction Type | Data format we will get flexibility in how we want to represent data. And it seems like a good step forward.
But what I am questioning now is why are we breaking RLP with concatenation when there is no need. As you can see concatenation is in question not format, bcs we could have same result with using ordinary RLP list and treating new tx as chunk of bytes.
Okay lets dive deeper on However part to find the reason for why that decision is made. You say that transactions are available in other places that JSON-RPC and hardware wallet protocol use.
For Hardware wallet, hashing for signature will still be same and done only on Type | Data part (excluding signature). And does Hardware wallet have a need to have raw list of transactions?
For JSON-RPC, I am unsure if there is RPC that delivers only raw list of transaction. Ordinary list of transactions are in JSON format. Raw list if found in RCP is mostly contained inside raw block.
I think there may be a miscommunication, so bear with me if everything I say here is obvious/redundant… I want to make sure we have the same understanding.
Over devp2p, the transaction batch will be rlp([transaction0, transaction1, ..., transactionN]). Each item in this list will be either a byte array or legacy transaction (RLP list). If it is a byte array, then you’ll get the length prefix for the whole transaction already due to the wrapping RLP, and if it is a legacy transaction you’ll also get a length prefix, though you’ll likely decode the whole thing when decoding the outer list (current behavior).
The only time you won’t have a length for the individual transactions in the encoded payload is if you are using a list encoding format that doesn’t provide boundaries or lengths. I believe most list encodings have some mechanism for splitting list items, which includes RLP and JSON (the two most common in ecosystem today).