EIP-7201: Namespaced Storage Layout

@frangio Understood about the NatSpec annotation and that’s great.

I already explained why this is not strictly true so I’ll refer to my previous messages. I’ll just re-emphasize this point:

This is what I am trying to say, it is strictly true. Your earlier message said this:

It is definitely possible to find a key k such that the slot of a mapping value m[k] is a string s of 32 printable ASCII characters, and if this is an array then m[k][0] will land at keccak256(s). The probability is 1e-14, making it difficult but feasible to mine one and if we can design namespaces such that the probability is reduced to the same as finding a hash collision (60 orders of magnitude smaller),

What your description fails to mention here is that if it was possible to mine such a thing it would generate 32 random printable characters. I am proposing to keccak256 a meaningful string. That is different than exactly 32 random characters. To try to mine a specific meaningful string is the same as trying to find a hash collision.

Here is an example of a meaningful namespace string "openzeppelin.storage.ERC20"

For us it was important to make namespace locations as disjoint as possible from the standard Solidity layout because the code we write has to be maximally generic. For others the considerations may be different.

Hashing a meaningful string with keccak256 is already disjoint from standard Solidity layout. This is no problem about having accidental collisions with Solidity using this simple method. Can you show or explain any problems that could happen with this?

Namespace/diamond storage is not new. It has been in use for years by many projects. If a new standard is going to change how it works I think it should have a good reason for doing so.

This is not what this standard aims to do, as has now been made explicit in the specification. The suggested formula in this ERC is a recommendation and other projects are free not to use it. I’d encourage you to standardize a formula identifier that people can use with @custom:storage-location for the formula that is just keccak256.

Yes, I agree. My previous statements were about collision resistance, and not about preimage resistance.

The suggested formula in this ERC is a recommendation and other projects are free not to use it.

Of course people can do whatever they want to. I don’t think this EIP should recommend an implementation of namespace/diamond storage that is different than how it has been being done for the past three years when there is nothing wrong with how it is has been being done.

If the way it has been being done for three years, which is by hashing a meaningful string, works and there is nothing wrong with that, then why recommend something else?

Yes, I agree. My previous statements were about collision resistance, and not about preimage resistance.

It is unclear to me what statements you are referring to. Please explain what is the problem with using keccak256 to hash a meaningful string to generate locations for structs. To say that it could collide with Solidity storage layout is misleading because any hash or storage location could do that, but it is not practical because hash collisions are impractical and not a concern.

The keccak256 hash of an arbitrary string is not guaranteed to be distinct from a storage location used by normal Solidity. There is an incredibly simple example which is the string "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00", whose hash is exactly the position of m[0] in a contract whose first variable is mapping (uint => uint) m. We are building tools that have to be robust and work with arbitrary Unicode strings. You talk about “meaningful strings”, but there is no formal definition of meaningful strings.

You’re not pointing out any particular problem with the formula in this EIP, other than being different to what others have used so far. I’ve explained why the EIP is compatible with the existing convention, and in fact how it can benefit it by standardizing a generic @custom:storage-location annotation. I’ve explained why the more complex formula is valuable for us given the constraints we’ve set for ourselves, the main constraint being that it has to be robust given arbitrary strings. As far as I see it this conversation has been exhausted. If you still don’t understand the explanations I’ve provided please reach out to me elsewhere so we don’t bother everyone else in this thread.

You talk about “meaningful strings”, but there is no formal definition of meaningful strings.

There doesn’t need to be a formal definition for “meaningful string” for how it has been used in the past 3 years and now, or how I see it being used in OpenZeppelin’s current upgradeable contracts. However some formalness could be added to it. Currently the EIP says “no whitespace characters”. It could also say no unprintable characters such as the null \x00, because there is no reason for meaningful strings to contain those. This can be enforced by tooling.

We are building tools that have to be robust and work with arbitrary Unicode strings.

Using keccak256 to hash meaningful strings is a robust way to create namespaces for structs. However you are right that strings with nonprintable characters or random strings are not “meaningful” strings.

You say that you need to work with arbitrary Unicode strings that are not meaningful. The current OpenZeppelin’s Upgradeable contracts are using meaningful strings. In what way are you currently using random or unmeaningful strings for storage of structs? Or what specific use cases do plan for this to be used for?

I argue that arbitrary or random strings are not namespaces because they aren’t names. Namespaces in software have transitionally been meaningful, not random or unreadable strings. I argue that a standard designed for unreadable or random strings should not be called a namespace standard.

Maybe there should be two standards here. One standard for unreadable or random strings used to store structs that clearly describes why it is useful and some of its use cases. And another standard that is a namespace standard that uses meaningful strings which is backward compatible with how namespaces/diamond storage has been used in the past three years and will not confuse people with hash manipulation that isn’t needed.

I disagree with the hash manipulation, specifically - 1) & ~0xff for a namespace storage standard to handle an edge case that nobody is using and no specific use cases have been described or given in the standard. I think that a new standard could be created in the future if this was needed that was very clear the specific use cases it was being created for, this way people are not confused with the idea that all namespaces need the hash manipulation.

1 Like

A namespace in a contract should be implemented as a struct type.

Could you please provide the rationale behind this specification?
Is it considered risky to implement this namespace as a type other than a pure struct type, such as a primitive type like address, or complex types like Foo[] or mapping(uint256 => Foo)?

There’s nothing inherently risky about that, but tools that consume this annotation might expect it to be a struct and may have unspecified behavior if you do something different.

1 Like

Hey!
It will be great if you’ll add how to add variables on upgraded contracts using the same storage space.
I’m pretty sure my syntax is incorrect but I’m sure you’ll get the idea and fix the code.

pragma solidity ^0.8.20;

contract ExampleV1 {
    /// @custom:storage-location erc7201:example.main
    struct MainStorage {
        uint256 x;
        uint256 y;
    }

    // keccak256(abi.encode(uint256(keccak256("example.main")) - 1)) & ~bytes32(uint256(0xff));
    bytes32 private constant MAIN_STORAGE_LOCATION =
        0x183a6125c38840424c4a85fa12bab2ab606c4b6d0e7cc73c0c06ba5300eab500;

    function _getMainStorage() private pure returns (MainStorage storage $) {
        assembly {
            $.slot := MAIN_STORAGE_LOCATION
        }
    }

    function _getXTimesY() internal view returns (uint256) {
        MainStorage storage $ = _getMainStorage();
        return $.x * $.y;
    }
}

contract ExampleV2 is ExampleV1 {
    /// @custom:storage-location erc7201:example.main
    struct MainStorage {
        uint256 z; // I want to add z to MainStorage
    }
    // The rest of the implementation, including how to use the pointer MAIN_STORAGE_LOCATION correctly

    function _getXTimesYTimesZ() internal view returns (uint256) {
        MainStorage storage $ = _getMainStorage();
        return $.x * $.y * $.z;
    }
}

Also, you can add a short description about whether those variables (x, y, z) are public or private and elaborate on their scope.

Huge appreciation, thanks!

No this is not correct and will not compile. Solidity does not allow extending inherited structs. You have two options:

  1. Implement ExampleV2 by copy-pasting ExampleV1 and modifying the struct.
  2. Implement ExampleV2 by inheriting from ExampleV1, but defining a new separate struct for the new variables.

Option 2 would look like this:

contract ExampleV2 is ExampleV1 {
    /// @custom:storage-location erc7201:example.v2
    struct V2Storage {
        uint256 z;
    }

    ...
}

Note that you would need to make _getMainStorage an internal function in order to use both MainStorage and V2Storage in the V2 contract.

1 Like

Thanks to you I was able to create the example code :slight_smile:

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.20;

contract ExampleV1 {
    /// @custom:storage-location erc7201:example.main
    struct MainStorage {
        uint256 x;
        uint256 y;
    }

    // keccak256(abi.encode(uint256(keccak256("example.main")) - 1)) & ~bytes32(uint256(0xff));
    bytes32 private constant MAIN_STORAGE_LOCATION =
        0x183a6125c38840424c4a85fa12bab2ab606c4b6d0e7cc73c0c06ba5300eab500;

    function _getMainStorage() internal pure returns (MainStorage storage $) {
        assembly {
            $.slot := MAIN_STORAGE_LOCATION
        }
    }

    function _getXTimesY() internal view returns (uint256) {
        MainStorage storage $ = _getMainStorage();
        return $.x * $.y;
    }
}

contract ExampleV2 is ExampleV1 {
    /// @custom:storage-location erc7201:example.secondary
    struct SecondaryStorage {
        uint256 z;
    }
    
    // keccak256(abi.encode(uint256(keccak256("example.secondary")) - 1)) & ~bytes32(uint256(0xff));
    bytes32 private constant SECONDARY_STORAGE_LOCATION =
        0x1924258aedabd40e2838e078c8996d6a5df44652ac0678deb4d54ca7ab098b00;

    function _getSecondaryStorage() internal pure returns (SecondaryStorage storage $) {
        assembly {
            $.slot := SECONDARY_STORAGE_LOCATION
        }
    }

    function _getXTimesYTimesZ() internal view returns (uint256) {
        MainStorage      storage $1 = _getMainStorage();
        SecondaryStorage storage $2 = _getSecondaryStorage();
        return $1.x * $1.y * $2.z;
    }
}

Gm, sers.

I’ve read the ERC and seen that it’s in the last call.

With the way Solc currently works, storageLayout will be empty when ERC7201 structs are used. This output is often used by tools for parsing Solidity files.

This can be easily fixed by adding that ERC7201_MOCK MUST be defined, as follows:

ContractStorage private ERC7201_MOCK;

Where ContractStorage is the ERC7201 struct.

This will be the only ‘regular’ variable, and will never be used. It won’t have effect on the contract, since namespaces are used.

With the mock in place, the complete storage layout can be parsed from Solc’s output by adding storage[i].slot + namespace.

This was the main thing I wanted to suggest.

What follows are a few others ideas on how the template for contracts with storage contracts can be standardized:

contract Profile {
    /// @custom:storage-location erc7201:zeroekkusu.magicians.Profile
    struct ProfileStorage {
        string _username;
        string name;
    }

    ProfileStorage private ERC7201_MOCK;

    // keccak256(abi.encode(uint256(keccak256("zeroekkusu.magicians.Profile")) - 1)) & ~bytes32(uint256(0xff))
    bytes32 private constant ERC7201_NAMESPACE = 0x5b0d49f4bb702a14af81a298d6304c57802c4493960a722fb612a039d9e77100;

    function $Profile() internal pure returns (ProfileStorage storage $) {
        assembly {
            $.slot := ERC7201_NAMESPACE
        }
    }

    function name() external view returns (string memory) {
        return $Profile().name;
    }
}

contract ExtendedProfile is Profile {
    /// @custom:storage-location erc7201:zeroekkusu.magicians.ExtendedProfile
    struct ExtendedProfileStorage {
        string bio;
    }

    ExtendedProfileStorage private ERC7201_MOCK;

    // keccak256(abi.encode(uint256(keccak256("zeroekkusu.magicians.ExtendedProfile")) - 1)) & ~bytes32(uint256(0xff))
    bytes32 private constant ERC7201_NAMESPACE = 0xb544e00821034a47c6890dc236e787d1935ceb592b881d8a41d96b1b7a311e00;

    function $ExtendedProfile() internal pure returns (ExtendedProfileStorage storage $) {
        assembly {
            $.slot := ERC7201_NAMESPACE
        }
    }

    function bio() external view returns (string memory) {
        return $ExtendedProfile().bio;
    }
}

Context: Usernames are usually unique, while the (display) names and bios can be altered freely.

The above approach introduces two things:

  • Standardizing a function for reading from and writing to storage
  • Distinguishing between ‘private’ and ‘internal’ variables in the ERC7201 struct

First, let’s address the elephant in the room.

The ‘private’ variables are prepended with _ and can (but should not) be directly accessed from the inheriting contract - much like in Python.

The approach is not completely fool-proof, but should be safe enough, because the developer must specify the contract the variable is from verbatim ($Contract()) and then break the rule by writing to the ‘private’ variable (._variable =). This assumes the basic familiarity with the ERC.

Contracts may choose not to make their $Contract function internal - e.g. OpenZeppelin - in which case they will need to add setters and getters for ‘internal’ variables.

The function name is standardized as $Contract, where Contract is the same label used in the contract name and ERC7201 struct. It is concise, persistent, and $ implies ‘storage’.

There are no other requirements.

ERC7201_NAMESPACE is just an arbitrary name, and any can be reused indefinitely, as long as it’s private.

Libraries such as Forge-Std can provide scripts for quickly generating an ERC7201 template. No base contract required, since Solidity doesn’t support generics anyway.

Let me know what you think!

Edit: Something’s been brought to my attention that I think should be clarified here. OpenZeppelin already makes a distinction between their regular and upgradable contracts. All the upgradeable ones use ERC-7201. Adding the mock requirement to the ERC would automatically rule out mixing layouts (using both ‘regular’ variables and namespaces). This means all upgradeable contracts would need to use ERC-7201. If this is undesirable, another solution for storageLayout can be explored, such as relying on toolchains (Foundry, Hardhat, etc) to generate the output.

1 Like

Thank you for sharing!

Your observation about storageLayout is true, in fact we ran into this when implementing the layout compatibility checks for OpenZeppelin Upgrades. Permanently adding a variable like ERC7201_MOCK was not an option for OpenZeppelin Contracts and I would not recommend it for other users either, because having done that the comparison of layouts between contract upgrades may result in the same pains we suffered with the gap-based approach. Namely, the _MOCK variables may become reordered, clash, etc. What we ended up doing in OZ Upgrades is to insert a preprocessing stage that automatically adds a similar storage variable to every contract with a namespace, this new source code is then recompiled for the sole purpose of obtaining the storageLayout output. This preprocessing is quite tricky! Ideally this would be solved by solc natively in the future by providing a flag where all types are included in storageLayout even if unused in a state variable. I highly recommend people use OpenZeppelin Upgrades to check layout compatibility before upgrading.

Your other points are valuable too.

The $Contract() naming convention is quite nice actually. For OZ Contracts we did _getContractStorage().

With ERC7201_NAMESPACE you may encounter issues with inheritance: if you have multiple variables like it in your inheritance hierarchy, Solidity will complain even if they are private, so you need to give them different names. For OZ Contracts we used ContractStorageLocation (arguably the capitalization of this variable is against Solidity conventions, but it is simpler to preserve the capitalization of the contract name).

Regarding private and internal variables I think the underscore prefixes are reasonable.

Scripts to generate templates/scaffolding sound useful!

Finally, while these are great guidelines and suggestions, I don’t think they need to be encoded in the ERC.

1 Like

Thank you for sharing.

I don’t see a need to standardize this. Not enough people are reading this NatSpac (are any implementations provided in the ERC?). So a much better approach is to just convince the Solidity team to implement this, and wait for people to actually use it.

Plus the main feature I see for contracts that are using defined storage slots is to make contracts with upgradable functionality which is the worse use case of blockchain anyway.


Overall, I recommend this approach be abandoned as a standard because the document does not specify something that is actually standardized in the real world. And separately because it is not something that promote good use cases.

Not enough people are reading this NatSpac

Is there a requirement of a minimum number of people reading this…What do you mean, Natspec definition or ERC?

So a much better approach is to just convince the Solidity team to implement this, and wait for people to actually use it.

How is enshrining a feature that may not get used into the Solidity language better or safer?

Plus the main feature I see for contracts that are using defined storage slots is to make contracts with upgradable functionality which is the worse use case of blockchain anyway.

Overall, I recommend this approach be abandoned as a standard because the document does not specify something that is actually standardized in the real world. And separately because it is not something that promote good use cases.

image

What are good cases? Is AAVE or Lens a good case? They use upgradeable contracts. How many immutable ERC721 scams and rugpulls have been made? Are those good use cases?

I can see the argument of smart contracts not supposed to be immutable in the first case, but upgradeability is a tradeoff that helps devs (and yes, it has tricky security/governance/trust implications and it must be treated as a possible attack vector).

In the context of this ERC however, I think that discussion has no place. IMO the cat is out of the bag, there are several approved ERCs regarding proxies and upgradeability. In the particular case of storage layout organization for implementation contracts, we have 2 options AFAIK:

  • EIP-2535 indirectly deals with storage layout organization contracts, but implementing the full EIP means I have to add to my contracts a level of complexity, gas overhead, proxy type and cognitive load coming from a domain specific terminology unrelated to blockchain, ethereum, etc

  • _gap arrays, which are not defined as an ERC, but suggested by OZ, and in practice error prone, hard to maintain and scale within a complex code base and across multiple versions.

ERC-7201 defines a safer, easier way of using (for example) UUPS proxies and managing storage layout across implementations than _gap arrays, with a formula that has possible gas savings in the future. Plus the name describes what it does, people reviewing the code will not need to have previous knowledge about jewels and other features the contracts may not be using.

How is this EIP impeding devs to keep using diamond storage @mudgen? EIP-2535 does not define “namespaced storage” anywhere, it says:

The particular layout of storage is not defined in this EIP, but may be defined by later proposals. Examples of storage layout patterns that work with diamonds are Diamond Storage and AppStorage.

ERC-7201 allows for several formulas for obtaining the storage location from the namespace, one of them could be the way some people have been doing it.

1 Like

If this EIP is to support multiple ERC definitions potentially, shouldn’t this EIP then be a ‘Living’ standard so that it may be updated to contain the appropriate ERC’s/namespaces that adhere to its defined NatSpec conventions?

Cheers!

I’ve discussed it with the ERC editors before, and as far as I know there is no appropriate ERC category or editorial process yet for this kind of thing.

If the goal is to specify NetSpec and want to allow it to be Live, I think it can be a Informational ERC for non-binding proposals, or a Meta ERC if there is a strong enough consensus what we make it a rule. Yes it’s the first time I see a proposal spe

$Contract() naming convention is awesome.

For the reports on the storage layout I replaced ERC7201_MOCK

ProfileStorage private ERC7201_MOCK;

with namespace label, e.g.:

ProfileStorage private ERC7201_zeroekkusu_magicians_Profile;

forge inspect then gives a nice report like this:

Name Type Slot Offset Bytes
ERC7201_XERC20_lockbox_storage struct XERC20WithLockboxUpgradeable.XERC20WithLockboxStorage 0 0 64