One thing to note is that for the purpose of merkleization, this is equivalent to List[T, 1], and to Union[None, T]. It is just the serialization that is more compact than those workarounds, and the obvious ability to write more concise code, instead of switching on list lengths or union selectors.
SSZ Union are not “already supported” across the board; so far, they are not used in any final Ethereum specification. Certain libraries such as nim-ssz-serialization currently only fully implement the limited support needed for handling the Optional[T] workaround. There are also no official tests for SSZ unions.
If “unions were introduced for exactly this purpose”, they would most-likely optimally encode for this purpose. However, they do not, and they are clearly a more powerful construct than just Optional[T]. The same design space as Union[None, T] is actually already offered by List[T, 1], which also encodes more compactly for fixed-length types, and actually is “already supported” and well tested.
Regarding “abuses the length byte”, I’m not sure what you are referring to, as SSZ does not explicitly encode a “length byte” anywhere. For example, in a List[T, N] of fixed-length type, N is implicitly derived from len(List) / sizeof(List). For variable-length types, N is derived from (&(*bytes) - &bytes) / sizeof(uint32) if len(List) > 0 or assumed to be 0 if len(List) == 0. The proposed Optional[T] SSZ type follows these same conventions.
The proposed Optional[T] works not just for fixed-length types, but also for most variable-length types such as Bitlist[N], Union[type_0, type_1, ...], Container with variable-length members, and Vector with variable-length members. As proposed, the only types that cannot be nested on the very next layer are List[T, N] and Optional[T], which both already have a natural way to denote absence (len = 0 and None). The concept of Illegal types already exists in SSZ. If needed, the format can be accordingly extended in the future.
Indeed, since the SSZ unions haven’t been used in any production spec yet, the Optional type described here can also be framed as a special case optimization for the the Union type. This was my original proposal (please see point 2 in the linked issue):
Point being, serialization and merkleization of SSZ Unions are still being discussed and not properly tested, as they are a new SSZ type not currently used in any finalized spec.
Note: the precise definition of Union is still a topic of discussion. Union is currently not yet used in consensus, and so there remains freedom to decide exactly how Union types are to be hash-tree-rooted and serialized. There are different approaches being proposed that attempt to maximize efficiency and simplicity.
For EIP-4844, using something simpler like the Optional[Address] proposed here, or even just plain List[Address, 1] (actually the exact same serialization and merkleization for fixed-length types), reduces the complexity of requiring an entire union framework to just represent an optional address. Yes, it also shaves off that single byte, in either List or Optional case, as a side effect.
Decision on this was postponed until we know whether SSZ Unions will actually be used in their full scope. If they are, their design regarding serialization and merkleization will have to be finalized, and exhaustive tests for them added. On the other hand, if we decide that SSZ Unions are not needed at this time, EIP-4844 could move to List[T, 1] or to Optional[T].
Agree that a consensus-specs PR is also warranted. Furthermore, if tomorrow’s SSZ breakout call reveals that we actually end up with a situation where full Union support is required, and not just the Optional subset, updates to the Union spec should be discussed as well to gain the same optimizations from @zah.
Namely that the union’s None case serializes without the selector (no downside), and potentially also that the [None, T] case serializes without the selector (no List / nullable Union at the very next layer). Optional and Union[None, T] ideally have the same properties in a world containing union, so Optional becomes just syntactic sugar to make it explicit that there is no intention to expand it to Union[None, T, U] later on, and to allow language features such as .isSome checks, similarly to how we have sugar for ByteVector etc.
In the other case, if we determine that full Union support is not needed at this time (for SSZ transactions discussion), I would advocate to only depend on the actually used Optional subset.
BTW, if there is any default Address constant that could be used to represent the None case, I would prefer that one over either Optional or Union. But I suspect that Optional support is actually really becoming necessary.