Summary
EIP-8297 replaces the hexary MPT with a partitioned binary tree (PBT). Keys get a zone prefix, nodes become binary, and paths get longer: on mainnet a written storage leaf has 45 to 48 branch records above it, an account leaf 32. The storage layer sees more records per update than today, and a different split between records that change every block and records that never do.
We replayed two separate weeks of Ethereum mainnet into a PBT and priced eight ways of laying its state out on disk, all under the same DRAM budget. Two are built and produce the same state root as a reference implementation. Six are models fed by the same replay, among them the layouts today’s clients use: a leveled LSM of trie nodes keyed by path (geth on Pebble, Nethermind and Besu on RocksDB) and a copy-on-write B+tree of pages (reth on MDBX).
The layouts differ in the unit they write, a record or a 4 KB page, and in how they write it: in place, copy-on-write, appended to a log, or merged through sorted runs. IO cost per block at 16 GB of DRAM for the storage engine, NVMe price profile, two windows (24.0M / 23.7M), lower is better:
| unit written | in place | copy-on-write | appended to a log | sorted runs (LSM) |
|---|---|---|---|---|
| pages | 17,438 / 17,160 (NOMT-style page store) | 64,378 / 63,648 (B+tree, as reth on MDBX) | 12,834 / 12,624 (page log) | 54,624 / 54,942 (page LSM) |
| records | not built | not built | 6,866–10,248 / 5,046–9,408 (log); 5,221–8,603 / 3,397–7,760 with lazier compaction | 5,854–6,908 / 5,252–6,319 (LSM keyed by path, as geth, Nethermind and Besu); 5,602 / 5,358 keyed by page |
Three findings hold in both windows.
- From 16 GB up, writing records beats writing pages. The cheapest page-writing layout, the page log, costs 1.3 to 2.4 times the record-writing ones; the in-place page store 1.7 to 3.2 times; the copy-on-write B+tree 6 to 12 times. A block changes about 18,000 records spread over about 2,400 merkle pages, and every page written moves 4 KB to change a handful of records.
- Among record layouts, the LSM and the log are within 1.6x of each other, and neither clears the 1.3x bar we use to name a winner. The path-keyed LSM that geth, Nethermind and Besu run today is already in the cheapest group. Keying it by page instead of by path halves its read misses (537 against 972 per block at 16 GB), but about 70% of its cost is compaction writes, so the total moves by at most a quarter.
- Below 16 GB the record layouts do not fit, and pages win. Their on-disk index needs 7.5 GB (log) or 9.3 GB (LSM) of filters and buffers in memory. At 4 and 8 GB the page log is the cheapest layout in both windows, 1.3x over the in-place page store on NVMe and 3.3x over the record log on a SATA-class SSD. At 2 GB the in-place page store is the cheapest by 2.9x.
For client teams this reads as follows. A team on a path-keyed LSM keeps its engine; a page-aligned key is a modest gain, and the next lever is compaction amplification. reth’s copy-on-write B+tree of pages is the costliest layout at every budget, so moving PBT state out of it is the largest single change on the table. A NOMT-style in-place page store is the only layout that fits under 4 GB, and appended to a log it is the cheapest one at 4 to 8 GB.
EIP-7870 recommends 32 GB of RAM for a full node. The consensus client and the OS share that memory, so we take about half, 16 GB, as the storage engine’s share. The ratios above are worst cases: each store’s cost is a range, and the ratio is taken at the end of each range least favorable to the winner.
Layouts
Two layouts are built; six are modeled from the same replay. The modeled figures come from the replay’s record and page streams and from measured cache behaviour, not from running Pebble, RocksDB or MDBX as the store. One engine figure is measured: a RocksDB was fed the replay’s record stream, once as the LSM of records and once as the log’s index, and the tables use each model’s write amplification scaled by what it wrote (Method).
Layouts that write pages:
- Page store, in place (built). Follows the page design of NOMT. Branch records are packed into 4 KB pages of six tree levels each, rewritten in place. Values live in a separate B-tree whose interior nodes stay in memory. Under 1 GB of fixed memory at mainnet state size.
- Page log (modeled). The same pages appended to a log instead of rewritten in place, found through a 16-byte-per-page index in DRAM (2.2 GB at mainnet size), compacted when half the log is garbage. No random writes; two syncs per block.
- Page LSM (modeled). The same pages as 4 KB values of a leveled LSM keyed by page id.
- Copy-on-write B+tree (modeled). Records in key order in 4 KB pages, 33 per leaf; a commit copies every changed leaf and interior page to a fresh location and syncs twice. This is the layout a reth node keeps its trie in under MDBX.
Layouts that write records:
- Log (built). Every changed record is appended to a log; old copies become garbage and are reclaimed by compaction. The index that maps each record to its position would take 72 GB in DRAM at mainnet size (5.0 billion records), so it is kept on disk as a leveled merge tree with a write buffer and a per-entry filter in memory (7.5 GB fixed). Reported at two compaction settings: compact when half the log is garbage (up to 2x the live data on disk) or when two thirds is (up to 3x).
- LSM of records, keyed by path (modeled). Every record is one entry of a leveled LSM keyed by its tree position, the layout geth (Pebble), Nethermind and Besu (RocksDB) keep their trie in. The engine constants are those of the log’s index: a 256 MB write buffer, levels growing by 10x, 1.85 bytes of filter per entry in memory (9.3 GB fixed). Two cache variants: a cache of records (geth’s clean cache) and a cache of 4 KB SST blocks.
- LSM of records, keyed by page (modeled). The same LSM with its key laid out as (page level, six-level-aligned path prefix, position within the page), so that one 4 KB data block holds the records of one page. Writes as the path-keyed LSM; reads as the page store.
Method
- Data: state diffs and execution reads from the public Xatu dataset (ethpandaops, CC BY 4.0). Each block’s net changes are classified as creations, rewrites or deletions. Reads of keys a block does not change are replayed as execution reads.
- Untouched state: the replay does not load the full state. Beside every written path it places stand-in leaves wherever the full state would have keys, so that paths have mainnet depth. Their positions follow from state-size statistics and EIP-8297’s hash placement. Tests check that the reduced tree emits the same records as the full tree.
- Cache: every layout gets the same total DRAM. Fixed parts (index, filters, resident nodes, buffers) are paid first; the rest is an LRU cache. Miss counts come from a stack-distance profile of the replay.
- Cost: random reads, sequential bytes, random page writes and fsyncs, priced with two device profiles. On the NVMe profile a random 4 KB write costs 4 sequential ones and a random read 3; on the SATA-class profile 16 and 10.
- Write amplification comes from the replay’s own update stream through one engine model for every merge tree, checked against RocksDB 11.8 fed the same stream for one of the four subtrees at that subtree’s scale (five levels, the untouched state interleaved as cold entries). The model gives 15.5 per update for the log’s index, where half the updates are compaction relocations that the buffer never absorbs; RocksDB wrote 0.83 of that, 0.88 at the lazier compaction setting where relocations are fewer, so the tables use 12.9 and 9.8. It gives 5.5 per record update for the LSM of records, which has no relocations; RocksDB wrote 1.01 of that. The page LSM’s 8.6 per page update is the model alone. The check counts entries written by flushes and compactions; in bytes RocksDB writes less, 64 B per entry against the model’s 81 B, since prefix compression and Snappy are not modeled.
- Verdict: six families (page store, page log, page LSM, B+tree, log, LSM), each represented by the variant with the best worst case. A family is named cheapest only if it beats every other family by 1.3x or more at the ends of both ranges least favorable to it.
- Scale: the key space is split into four address-hash subtrees, replayed separately and combined. On a 10,000-block window the combined result matches a single full replay to within 1%.
Results
Sample: blocks 24,000,000 to 24,049,999 and 23,700,000 to 23,749,999, about 6.9 days each; costs measured over the last 1,000 blocks of each after 49,000 blocks of warm-up. Cells read 24.0M / 23.7M, NVMe profile, reads priced per IO. A dash means the layout’s fixed memory does not fit the budget.
| DRAM | page store | page log | log, GC at 1/2 | log, GC at 2/3 | LSM path-keyed, record cache | LSM path-keyed, block cache | LSM page-keyed | page LSM | B+tree |
|---|---|---|---|---|---|---|---|---|---|
| 4 GB | 18,208 / 18,016 | 13,964 / 13,825 | - | - | - | - | - | 55,304 / 55,724 | 67,864 / 66,932 |
| 8 GB | 17,772 / 17,567 | 13,208 / 13,072 | 20,433 / 19,428 | 18,788 / 17,779 | - | - | - | 54,946 / 55,334 | 65,794 / 64,946 |
| 16 GB | 17,438 / 17,160 | 12,834 / 12,624 | 6,866–10,248 / 5,046–9,408 | 5,221–8,603 / 3,397–7,760 | 5,854–6,908 / 5,252–6,319 | 7,359 / 6,829 | 5,602 / 5,358 | 54,624 / 54,942 | 64,378 / 63,648 |
| 32 GB | 17,092 / 16,796 | 12,462 / 12,225 | 5,108–10,189 / 5,046–9,408 | 3,463–8,544 / 3,397–7,760 | 4,246–6,786 / 4,116–6,298 | 6,073 / 5,492 | 5,067 / 4,733 | 54,286 / 54,590 | 63,294 / 62,617 |
Cheapest family by the 1.3x rule: page log at 4 GB (1.30x / 1.30x over the page store) and at 8 GB (1.35x / 1.34x); undecided between log and LSM at 16 GB (LSM ahead by up to 1.54x / 1.48x, log by up to 1.07x / 1.58x) and at 32 GB (2.01x / 1.89x against 1.46x / 1.39x). At 128 GB the log can keep its index in DRAM, 1,427–3,966 / 1,390–3,572 per block, a budget no full node has.
On a SATA-class SSD at 16 GB: page store 68,460 / 67,470; page log 15,420 / 14,792; log 12,486–23,760 / 6,399–20,942, or 10,841–22,115 / 4,750–19,293 compacted at two thirds; LSM keyed by path 10,241–13,753 / 8,446–12,004 with a record cache and 15,260 / 13,703 with a block cache; LSM keyed by page 9,400 / 8,802; page LSM 57,028 / 56,902; B+tree 255,588 / 253,033. A random 4 KB write costs 16 sequential ones here, so the in-place page store and the B+tree fall further behind, and the page log, which has no random writes, comes within 1.1x of the block-cache LSM.
What each layout does per block at 16 GB (24.0M window, worst-case end of the range):
| layout | random reads | random 4 KB writes | sequential MB written + read | fsyncs |
|---|---|---|---|---|
| page store | 343 | 4,035 | 0.7 + 0 | 5 |
| page log | 358 | 0 | 26.7 + 26.7 | 2 |
| log, GC at 1/2 | 1,068 of 4 KB + 857 of 512 B | 0 | 10.2 + 10.1 | 1 |
| log, GC at 2/3 | same | 0 | 6.4 + 6.4 | 1 |
| LSM path-keyed, record cache | 972 | 0 | 9.7 + 8.2 | 1 |
| LSM path-keyed, block cache | 1,123 | 0 | 9.7 + 8.2 | 1 |
| LSM page-keyed | 537 | 0 | 9.7 + 8.2 | 1 |
| page LSM | 338 | 0 | 127.9 + 114.5 | 1 |
| B+tree | 941 | 15,379 | 0 | 2 |
Observations:
- The page store and the B+tree are bound by random writes, about 4,000 and 15,000 per block. Their reads are the fewest of all layouts, because one page holds six levels of the tree and a cold path costs one fifth of the log’s reads.
- The log pays two reads per miss, one for the index and one for the record, and its range is the widest because reads of records untouched since the replay began (about 850 per block) may or may not be cached on a real node. That, not its writes, keeps it from clearing the LSM. Compacting at two thirds garbage instead of half cuts its writes to 6.4 MB per block, below the LSM’s 9.7 MB.
- Garbage collection is a lever for both logs. At 16 GB on NVMe (24.0M / 23.7M), the record log costs 6,866–10,248 / 5,046–9,408 per block when compacted at half garbage, 5,221–8,603 / 3,397–7,760 at two thirds and 4,298–7,680 / 2,466–6,829 at four fifths; the page log 12,834 / 12,624, then 9,904 / 9,702, then 8,439 / 8,241. Disk use grows to 2x, 3x and 5x the live data.
- The LSM’s write amplification of 5.5 decides the second finding. The model measures it from this replay under a 256 MB buffer, which absorbs three quarters of record updates before they reach a level, and RocksDB fed the same records wrote 1.01 times that. Level by level the engine writes more at the top, where its memtable keeps every version of an entry until it flushes, and less at the bottom, where it rewrites 7 to 8 entries per entry moved against the model’s 11; the two cancel. At an amplification of 15 the path-keyed LSM would cost 12,172–13,226 / 11,494–12,562 per block at 16 GB and fall behind the log; at 30 it would cost 22,094–23,147 / 21,255–22,322 and fall behind the page log. For the log’s index RocksDB wrote 0.83 of the model’s 15.5 (0.88 at two thirds garbage), so the log’s figures are 3 to 12% below the model’s own; the excess is in the deeper levels, where the model rewrites more of a level per entry moved than the engine does.
- Execution reads barely move any result. Most keys a block reads are already cached.
Timed IO
To check the device profiles behind the model, we take each layout’s per-block IO counts at a given DRAM size and run them against a 400 GB file with the page cache bypassed (O_DIRECT on Linux, F_NOCACHE on macOS). This times the IO a layout would issue, not a full engine. Commit IO is known up front, so it is issued 32 at a time; execution reads that miss the cache are issued one after another, as a client without prefetching would (two dependent reads each for the log, one for the others).
Page store and log on three devices, milliseconds per block, sample 24.0M, worst-case end of the range:
| DRAM | i4i.xlarge (page / log) | i7ie.large (page / log) | MacBook SSD (page / log) |
|---|---|---|---|
| 8 GB | 72 / 55 | 90 / 99 | 286 / 61 |
| 16 GB | 69 / 30 | 89 / 36 | 188 / 52 |
| 32 GB | 66 / 30 | 88 / 36 | 160 / 46 |
At 16 GB and above the log is 2.2 to 3.6 times faster than the page store on all three devices. At 8 GB the order depends on the device. The page store’s time is mostly its 4,000 random 4 KB writes per block, so it follows the device’s random-write rate.
All nine layouts on the MacBook SSD, milliseconds per block, sample 24.0M, worst-case end of the range:
| DRAM | page store | page log | log, GC at 1/2 | log, GC at 2/3 | LSM path-keyed, record cache | LSM path-keyed, block cache | LSM page-keyed | page LSM | B+tree |
|---|---|---|---|---|---|---|---|---|---|
| 4 GB | 114 | 41 | - | - | - | - | - | 83 | 354 |
| 8 GB | 103 | 35 | 50 | 48 | - | - | - | 73 | 322 |
| 16 GB | 96 | 31 | 26 | 22 | 18 | 29 | 25 | 71 | 333 |
| 32 GB | 134 | 28 | 26 | 22 | 21 | 22 | 25 | 68 | 318 |
Each cell is the median of 100 blocks with the commit IO issued 32 at a time, plus the block’s execution-read misses one at a time at this device’s serial read latency (116 microseconds; two reads per miss for the log). At 16 GB the record layouts and the page log all land between 18 and 31 ms per block and cannot be separated on this device; the in-place page store takes 3 to 5 times longer and the B+tree 10 to 18 times. The device is a laptop SSD under APFS, and the same page-store mix took 188 ms per block in the earlier run against 96 here, so the order within one run is usable, the figures across runs are not.
Limits
- Two one-week windows, both from late 2025.
- The 16 GB budget is an assumption about how a full node splits its 32 GB.
- Six of the eight layouts are models. The LSM and the log’s index share one engine model: one disk read per lookup that misses the cache, filters fully resident (9.3 GB for the LSM of records), no cache invalidation by compaction, levels below the second extrapolated. The B+tree uses a static page map, fill 0.8, fanout 60, two syncs per commit. The page log and the page LSM reuse the page store’s cache profile, which counts a page written whole as cached without a read; a log or LSM block cache would not. The page-keyed LSM does the same.
- The write amplifications were checked on RocksDB for one subtree of one window and for writes only: entries written by flushes and compactions, not bytes (RocksDB writes 64 B per entry against the model’s 81 B, since prefix compression and Snappy are not modeled), and not the read IO per miss, which stays the model’s one block. The page LSM’s 8.6 is the model alone.
- Code reads are not in the dataset and are not replayed.
- The tables come from a counting model. The timed runs replay its IO counts on three devices, two of them small cloud instances with per-instance IOPS limits and one a laptop SSD under APFS. A typical node’s NVMe under Linux is not yet measured.
- The four-subtree split gives each subtree a quarter of the cache instead of one shared cache.
Next
- Read IO on a real engine: the model charges one block read per lookup that misses the cache, with filters resident. The RocksDB run that checked the write amplification can measure reads the same way.
- A key-encoding experiment in that engine: path-keyed against page-keyed records.
- The 4 to 8 GB regime: the page log against record layouts with smaller filters.
- Tiered compaction for the log’s index, which rewrites each entry about once per level instead of about ten times.
- Timed IO on a commodity NVMe drive under Linux.
The replay tools and per-window results are at GitHub - cocoyoon/eip8297-storage-research: Mainnet replay of EIP-8297 PBT on-disk layouts · GitHub (MIT). The input data is the public Xatu dataset; the block ranges above are enough to fetch it. One window is reproduced with the repository’s tools as follows; its README has the full commands, including the RocksDB check.
tools/mainnet/fetch.sh DATA 24000000 50 reads
uv run tools/mainnet/export_diffs.py --data DATA --out DIFFS --reads
for q in 0 1 2 3; do
pbt-char bench --mainnet-diffs DIFFS --mainnet-slot-counts tools/mainnet/data/slot_counts_2025-12-13.csv \
--resident-levels 2 --blocks 1000 --warmup-blocks 49000 \
--mainnet-sample-bits 2 --mainnet-sample-part $q --out results --label replay_p$q
done
uv run tools/mainnet/combine_parts.py --out results/replay_combined.md results/replay_p{0,1,2,3}.md
Adding --mainnet-record-stream STREAM to one part writes the record stream that pbt-char lsm-validate (built with --features rocksdb) feeds to RocksDB.
Feedback is welcome. Two questions for client teams: on a 32 GB full node, is about 16 GB for the execution client’s state storage a fair working figure, and do your LSM deployments keep their filters fully resident at mainnet state size?