Significantly increased query response time for rpc.state.getMetadata #10056
Description
Since the upgrade of the metadata from 13 to 14 some teams are reporting that the time to query rpc.state.getMetadata has increased significantly from ~500ms to ~3-4s. Is there anything that can be done to optimize this besides caching the metadata and reading it from local memory and only querying the node when they know the metadata has changed? It looks like this wait time is clogging the queue in the cases that it is necessary to query the node for metadata.
Suggestion from @ascjones on Element:
It's possibly the routine which constructs the metadata (previously it was mostly static construction). It needs some investigation to pinpoint the cause. The short term solution would be caching as you say, the longer term solution would be to pre-generate the metadata for a runtime and store it somewhere else other than in the runtime binary. If size is an issue then we could run compression on it.