Replies: 13 comments 36 replies
-
Isn't it possible to have an empty implementation of new metadata for old macro? so that pallet with old macro can still be used but only provide old metadata? |
Beta Was this translation helpful? Give feedback.
-
|
Yes it is possible, I originally expected that we would have everything converted sooner which is why I didn't consider it originally. |
Beta Was this translation helpful? Give feedback.
-
|
Even if we manage to convert in due time, it will take time for the ecosystem to do the same. |
Beta Was this translation helpful? Give feedback.
-
|
In the end I converted the legacy macros to produce the new metadata, so now we have metadata even for the non converted pallets. |
Beta Was this translation helpful? Give feedback.
-
That sounds reasonable. I think we could use some existing Substrate's Off-Chain capabilities to store the metadata. In any case I think it's essential to split runtime code and the metadata into two separate blobs.
Having the metadata in the Off-Chain DB makes it easy for all consumers (like UI) to query it using built-in RPC method. Usage of Indexing API ensures correctness of the data. |
Beta Was this translation helpful? Give feedback.
-
Since the data you receive should be on-chain verifiable anyway (i.e. compare with hash) I can imagine this being either:
In case of using Off-Chain DB, not at all. Each metadata can have it's own entry (hash being the key for instance). If we store on-chain, I don't think we can afford that though.
Sure that's the most straight-forward, although it means that: (1) the metadata needs to be delivered on-chain (large transaction) (2) and it has to be stored in the trie (we need to avoid shared prefix, but obviously doable if we don't store historic data). |
Beta Was this translation helpful? Give feedback.
-
|
Why not put the runtime code itself in the offchain storage, then? |
Beta Was this translation helpful? Give feedback.
-
|
Could work, yeah. I guess the main difference is that runtime code is essential for full nodes (network security), while metadata is only required for user interactions, hence I'm more on the safe side with runtime code, but not metadata. |
Beta Was this translation helpful? Give feedback.
-
|
Reviving this thread, given results of polkadot runtime binary size increase (2M -> 2.4M minimum) #8370 (comment). We need to consider whether we need to do the separation, if so where to store it, and whether to do it as part of #8615, or as a follow up. |
Beta Was this translation helpful? Give feedback.
-
|
@bkchr Seems like this would create further issues with oversized runtimes? |
Beta Was this translation helpful? Give feedback.
-
Can you elaborate how is this supposed to look like? If it's just about adding new version, then I think it's acceptable - i.e. metadata was designed to be upgraded. If the compatibility is trivial though, we might implement it to make life of client developers easier, especially to not break support for hardware wallets. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe we can even modularize the metadata in order to allow for a smaller payload for constrained signer devices. Signers would only be interested in metadata regarding extrinsincs but not the new type data that also covers RPC signatures. |
Beta Was this translation helpful? Give feedback.
-
|
RPC signatures? :D They are clearly not part of the metadata :D |
Beta Was this translation helpful? Give feedback.
-
|
No they are not, but probably should have the same structure. They have types as well, and the intent has always been to make them self-descriptive, hence the version in That may actually be a small, less-impact place to start. (Certainly less annotations all over the codebase for RPC-only and can probably keep the existing info alongside providing types on the same RPC there) |
Beta Was this translation helpful? Give feedback.
-
I thought this is meant by "adding rich type information to substrate". |
Beta Was this translation helpful? Give feedback.
-
|
No :) These are the same type information as extracted from ink contracts. |
Beta Was this translation helpful? Give feedback.
-
|
Just one thing about timing. If we can get this out before Kusama parachain launch, this will mean all parachain projects will not need to start with the legacy metadata system. This definitely will make a lot of people's life better (including myself) so that we don't need to have single line of code dealing with the legacy system and all the migration. So I am keen to do whatever possible to get this out. Currently parachain custom types are manually generated, hard to validate, centralized discovery & distribution, and no cryptographic authentication anyway, so I don't think we can do anything worse than that. My answer to the questions: Should the metadata still be stored in the runtime itself? No. The types are not stored in current runtime anyway so that's not a regression. Metadata are huge and we already have enough trouble to keep our wasm size small so it play nicely with parachain production code. I think it is a mistake to have metadata include all the comments already. Backwards compatibility Bumping metadata version will causing similar breakage everywhere. We already have multiple runtime breaks balances display (all the ref count changes). I know wallet & exchanges won't like this, but this is the solution to reduce breaking changes so I will 100% say just break it. |
Beta Was this translation helpful? Give feedback.
-
|
+1 for keeping WASM size tiny, I like the idea of only keeping a hash that points to e.g. an IPFS CID. Would the storage chains stuff be helpful there if nodes become "IPFS ready"? could metadata be something kept only in the node client side only to be served via RPC or to other IPFS nodes and the runtime not care about it? |
Beta Was this translation helpful? Give feedback.
-
|
This is not the first metadata bump and won't be the last. For instance any API out there (Go, Ruby, Python, JS, ...) already needs to support at least metadata versions 9, 10, 11 and 12 (current) to be able to parse all historic data from Kusama. The sad thing is that because of the historic data, v13 won't remove the old custom types from any of the APIs (historic support), however at least going forward we won't need to add more. Which is an absolutely brilliant place to be. (And all new chain, as @xlc said, won't have this burden at all) |
Beta Was this translation helpful? Give feedback.
-
|
Polkascan Foundation fully supports Bryan's (@xlc) suggestions. |
Beta Was this translation helpful? Give feedback.
-
Update: I have now converted the legacy frame macro to produce the new metadata in https://github.com/paritytech/substrate/compare/aj-metadata-vnext, so it can now produce the scale-info enriched metadata for all pallets. See here for the resulting metadata for |
Beta Was this translation helpful? Give feedback.
-
|
Example of complete |
Beta Was this translation helpful? Give feedback.
-
|
This is going to be big for us as well, we are building a plugin based "decentralizable" API for our app and its blockchain/substrate plugin is based on |
Beta Was this translation helpful? Give feedback.
-
|
Early draft PR: #8615 |
Beta Was this translation helpful? Give feedback.
-
|
Any updates @ascjones? We (@interlay) are anxious to integrate these changes into our clients before Kusama. |
Beta Was this translation helpful? Give feedback.
-
|
My goal is to have the PR ready to review by the end of this week, just starting on the |
Beta Was this translation helpful? Give feedback.
-
|
Yeah okay it took a bit longer than I expected 🙈 paritytech/polkadot#3336 |
Beta Was this translation helpful? Give feedback.
-
|
First run of generating new metadata for polkadot: https://gist.github.com/ascjones/726caaf29eff76ca72f142f419e9c6a9 |
Beta Was this translation helpful? Give feedback.
-
|
When it says for example |
Beta Was this translation helpful? Give feedback.
-
|
Exactly it's the index into the |
Beta Was this translation helpful? Give feedback.
-
|
Regarding Wasm binary size, here are some numbers for the polkadot runtime:
So a minimum of a 20% increase in the size of the runtime which includes this type metadata. |
Beta Was this translation helpful? Give feedback.
-
|
I suppose I do not have to go on the record that this feature is at the very top of our wish list. Documentation is not functional in the metadata and the type registry/decomposition is functional in the metadata. |
Beta Was this translation helpful? Give feedback.
-
How do I run the compression? I built these with https://github.com/paritytech/polkadot/blob/c69ba7e2d4b4032dab11939129d890bfc88e210e/scripts/build-only-wasm.sh |
Beta Was this translation helpful? Give feedback.
-
|
The |
Beta Was this translation helpful? Give feedback.
-
|
I just used the
|
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Marked #8615 ready for review, still working on fixing some tests and size benchmarking, but ready for some feedback. |
Beta Was this translation helpful? Give feedback.
-
Opening a discussion for tracking and feedback on the proposed integration of
scale-infointo substrate.Currently the substrate metadata doesn't contain full type information, so extra mappings to type definitions are required in client implementations e.g. polkadot.js in order to to provide compatible encoding/decoding of types when interacting with substrate.
The goal is to be able to have enough information embedded in the metadata in order to generate compatible types in any language. This will greatly reduce the maintenance burden for client implementations.
The plan is to use
scale-infoto derive type information for all types that are exposed in a substrate runtime via calls, events. storage, constants etc.Current status
pallet!#7882scale-infoitself still needs a feature or two before it is ready: Substrate compatibility scale-info#36Once all FRAME V2 macros are converted and final
scale-infofeatures are added I will open a substrate PR based on my PoC branch.Open questions
Should the metadata still be stored in the runtime itself?
Currently the metadata is stored in the runtime Wasm blob, however this could become unwieldy with the extra type information. Still need to measure the size difference, it might not be that bad but something to keep in mind. One idea just store a hash of the metadata in the runtime itself and the metadata itself somewhere else (where?).
Backwards compatibility
It would be possible to provide a conversion from the new style metadata (with full type info) to the deprecated style (without full type info) to ease the transition in the client ecosystem. Do we want to do this?
Related issues:
#917
#8158
#8083
/cc @Robbepop @dvdplm @thiolliere @tomusdrw @kianenigma @bkchr @jacogr @xlc
Please tag anyone else who would be interested in this discussion.
Beta Was this translation helpful? Give feedback.
All reactions