Allow runtime to hold multiple version of metadata

[gui1117]

When we want to improve metadata, it would be great to not break off chain tools which depend on some previous version.
Holding multiple version while letting the time for third parties to update is preferred.

Maybe this issue is as simple as adding a new method to sp_api::Metadata each time we bump the metadata, (like metadata_v2 and then metadata_v3) and deprecate old version by returning an empty one when after some time.

But this is a bit annoying for runtime declaration, another solution could be to have Metadata being a vec of (version, Metadata), allowing the runtime to return multiple version in the same api call.

Not sure what is best. and there can be better ideas.

[xlc]

When I was working on #1328, I had an idea that to expose the metadata of metadata. i.e. when scale-info is available, we can expose the scale-info of the Metadata type, so now the SDK will always be able to decode metadata. Because scale info includes field name etc, SDK will be able to work with the new metadata object as long as it don't rename / remove fields / change field type completely.

[bkchr]

When I was working on #1328, I had an idea that to expose the metadata of metadata. i.e. when scale-info is available, we can expose the scale-info of the Metadata type, so now the SDK will always be able to decode metadata. Because scale info includes field name etc, SDK will be able to work with the new metadata object as long as it don't rename / remove fields / change field type completely.

That is really to much meta :P We clearly don't need to provide scale-info for metadata.

Metadata is nowadays relative stable and we don't break it that often. I think we just need to be more careful when doing this. Currently it isn't the case anymore that we update metadata today and tomorrow it is live on some production chain, we left the fast moving days already for quite some time ;)

I think we should just announce the metadata early enough before it gets merged into Substrate. By early enough I would say that 1 month should way enough time.

Maybe this issue is as simple as adding a new method to sp_api::Metadata each time we bump the metadata, (like metadata_v2 and then metadata_v3) and deprecate old version by returning an empty one when after some time.

We could pass some version to the method, but for the client side, the metadata is opaque and it doesn't know what version the runtime is using etc. So, we should not start adding methods per version to call them. But given what I said above, I'm also not in favor of providing this support. Because we would run into the next problem of "when do deprecate an old metadata"? And not all projects follow Substrate master, so they would probably never have this period in time where both versions could be returned.

TLDR: I think we should announce a new metadata version one month before with all the changes to the format and inform the relevant people. As we move metadata out of Substrate, it will probably also be much easier to keep these people informed, as they could watch the other repo.

[lucgerrits]

I have the same issue where the metadata version is outdated since V6 (for Substrate) in polkadot_api_cpp. If the metadata detection doesn't work, the C++ API can't return storage data.
I'm trying to add V12 to polkadot_api_cpp, but:

• Where are the definitions of metadata versions?
• My Rust is bad, so I can't understand Rust defined versions

Rel usetech-llc/polkadot_api_cpp#32.

[gui1117]

@lucgerrits you can find the doc of metadata here: https://substrate.dev/rustdocs/v3.0.0/frame_metadata/enum.RuntimeMetadata.html#variant.V12

It is encoded using parity-scale-codec. Maybe there is better doc about the encoding somewhere I'm not aware of in the https://substrate.dev/ website.

For older version you can look at rust code at old commits.

Not sure how to help you more, maybe you can ask in substrate technical channel in element.

[lucgerrits]

I have also seen exactly that doc. The doc on substrate.dev (here) is missing some details that help to make V12 work.
Thanks for the tip, I'll try the substrate technical channel!

[stale]

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

[gui1117]

I think is what Basti wrote:

TLDR: I think we should announce a new metadata version one month before with all the changes to the format and inform the relevant people. As we move metadata out of Substrate, it will probably also be much easier to keep these people informed, as they could watch the other repo.