discussion for: reduce transient pricing for common cases by charles-cooper · Pull Request #8158 · ethereum/EIPs · GitHub
2 Likes
I ran some crude benchmarks using revm. The tools and methodology are below, but the tl;dr is:
plain dup2+mul 10k times ~20ns / operation
mload of the same address 10k times ~12ns / operation
tstore of the same address 10k times ~25ns / operation
tstore of 10k different addresses comes ~40ns / operation
tstore + tload of 10k different addresses ~70ns / operation
sha3 costs about 500ns / operation (sha3 of a 32 byte buffer)
script to generate evm scripts: generate benchmarks for transient storage · GitHub
revm script: feat: add evm script by charles-cooper · Pull Request #1039 · bluealloy/revm · GitHub
i want to point out that the summary numbers are estimates, since there is some jitter from stack operations which throw off the numbers a bit (however, i do think they are more or less in the correct ranges). i added two control scripts which do some stack fiddling to mimic what the later operations do, so their timings can be subtracted from the relevant scripts.
results (from revm/bins/revm-test, running for file in *.evm; do cargo run --release --bin evm $file; done):
Finished release [optimized + debuginfo] target(s) in 0.15s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_control2.evm`
Run bytecode (3.0s) ... 108_523.678 ns/iter (0.992 R²)
Finished release [optimized + debuginfo] target(s) in 0.12s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_control.evm`
Run bytecode (3.0s) ... 222_961.624 ns/iter (1.000 R²)
Finished release [optimized + debuginfo] target(s) in 0.12s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_easy_mload.evm`
Run bytecode (3.0s) ... 126_342.356 ns/iter (1.000 R²)
Finished release [optimized + debuginfo] target(s) in 0.12s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_easy_tstore.evm`
Run bytecode (3.0s) ... 370_559.161 ns/iter (1.000 R²)
Finished release [optimized + debuginfo] target(s) in 0.13s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_sha3.evm`
Run bytecode (3.1s) ... 4_947_213.996 ns/iter (1.000 R²)
Finished release [optimized + debuginfo] target(s) in 0.12s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_tload_pure.evm`
Run bytecode (3.0s) ... 215_596.323 ns/iter (0.997 R²)
Finished release [optimized + debuginfo] target(s) in 0.12s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_tstore.evm`
Run bytecode (3.0s) ... 619_875.709 ns/iter (0.999 R²)
Finished release [optimized + debuginfo] target(s) in 0.12s
Running `/home/charles/src-references/revm/target/release/evm /home/charles/src-references/EIPs/benchmark_tstore_tload.evm`
Run bytecode (3.1s) ... 936_628.376 ns/iter (0.998 R²)
1 Like
I created some tstore/tload state tests here: tloadstore_opcode_statetests.json · GitHub
Results:
Tload: 46ms
Tstore: 205ms
For comparison, other tests I created on the same machine:
Push0: 86ms
Mstore8: 107ms
Mstore: 125ms
Sstore: 208ms
So it looks to me that tstore is priced correctly, tload is priced a bit high, but still in line with other opcodes
1 Like
are state tests a “good” or rigorous way to benchmark opcodes? how should we interpret / compare these results with @charles-cooper 's observations?
@Rjected i found there are a couple slightly tricky things to get right with the benchmarks:
- need to pop the result of TLOAD, otherwise the transaction can revert early (stack >1024 items) and time gets biased downwards
- tload from empty locations when the transient storage map is small or empty biases time downwards since the logic is
if key not in map: return 0and the existence check is very fast - repeatedly tloading from the same location when the transient storage map is small biases time downwards because lookups from small maps are in general faster than lookups from large maps
what i did to address these issues in the benchmarks was to issue tload of an address after tstoring it.
so given those caveats, marius’s benchmarks look about right to me. the sstore benchmark seems too fast though - maybe it’s not physically writing to disk?
i am not sure the coefficient in the EIP needs to be 3, either. with a coefficient of 1 we still get DoS protection (a single map maxes out under the current gas limit at 7738 slots ) but we get more breathing room for the “small maps” - 92 items before becoming more expensive than the current TSTORE rather than 30.
i updated the coefficient here: Update EIP-7609: reduce SLOPE coefficient in eip-7609 by charles-cooper · Pull Request #8272 · ethereum/EIPs · GitHub