The present Union implementation is inherently ‘stable’ actually.
No, the existing Union is not stable. As you mention Teku, let’s do a consensus example. If you want to build an application that tracks whether a particular validator is slashed in a trust-minimized way using merkle proof of the response, you will have to keep updating your application several times a year to teach it about all the various hard forks, including all those that add completely unrelated features. This is, because the Merkleization of BeaconState is not stable across consensus-fork versions – whenever the total number of fields reaches a new power of two, generalized indices change, and therefore, the Merkle proof changes. Same when a field is removed from one version to the next. With StableContainer, the proof remains forward-compatible even if fields get added or deprecated over time.
Both fields are either present or absent.
Yes, in the business logic. But that doesn’t need to leak into the serialization or the merkleization. Adding more wrapping than strictly required in SSZ leads to overhead when using variable-length fields, with additional offset tables getting serialized.
Teku has dedicated Java types for every hardfork version of every structure.
Still possible. One can build dedicated types for all the possible subsets on top of StableContainer.
For example, back to the transactions, one could go with two types:
class BasicTransactionPayload(StableSubset(TransactionPayload)):
nonce: uint64
max_fee_per_gas: uint256
gas: uint64
to: Optional[ExecutionAddress]
value: uint256
input_: ByteList[MAX_CALLDATA_SIZE]
access_list: Optional[List[AccessTuple, MAX_ACCESS_LIST_SIZE]]
max_priority_fee_per_gas: Optional[uint256]
class BlobTransactionPayload(StableSubset(TransactionPayload)):
nonce: uint64
max_fee_per_gas: uint256
gas: uint64
to: ExecutionAddress # Required in blob transaction
value: uint256
input_: ByteList[MAX_CALLDATA_SIZE]
access_list: Optional[List[AccessTuple, MAX_ACCESS_LIST_SIZE]]
max_priority_fee_per_gas: Optional[uint256]
max_fee_per_blob_gas: uint256
blob_versioned_hashes: List[VersionedHash, MAX_BLOB_COMMITMENTS_PER_BLOCK]
Both would serialize and merkleize as the TransactionPayload StableContainer. But inside business logic you can map them to concrete types for the type safety, reducing runtime checks and so on. Note that the memory layout could also be shared, with both BasicTransactionPayload and BlobTransactionPayload being a subset of TransactionPayload. One could implement that with zero copying by having an inner private payload and then define accessors into it. That also helps in consensus, because fork transitions are no longer expensive: simply set all the deprecated fields to None, then cast to the new fork and initialize the new fields.
I think it would be useful to have something like StableSubset specced out and in Python as well — What EIP-7495 describes is solely a forward-compatible serialization and merkleization scheme, it shouldn’t restrict how one prefers to write code.
Note that ProtoBufs follows a similar approach in the encoding of OneOf. The usage appears like a union, but serialization is same as a series of optional, with the parser enforcing that only one of the options is set. This means that a client that is only interested in certain aspects can still parse messages from newer servers, without having to continuously update (forward-compatibility). It also means that a server can still parse messages from old clients, without having to convert them (backward-compatibility).