add cross rt execution code

[adriangb]

#12393 (comment)

I'm not sure where to put this or what the right API is, but I'm putting this up as a first draft for discussion.

[adriangb]

I suspect the right way to implement this would be to make SessionContext a trait and offer this as an implementation of it? There might even be a way to automatically wrap all ObjectStores in CrossRtObjectStore. The only manual bit left then would be to ensure any non-ObjectStore IO is wrapped in spawn_io, that's made easier by the fact that we don't allow IO on the CPU runtime so you'd know if you forgot (you get a panic).

[tustvold]

See also #13424 which contains some prior discussion, including why shimming ObjectStore in this way is potentially fraught (#13424 (comment) in particular)

[adriangb]

Interesting, thanks! We don't use list (only get) so maybe that's why we haven't run into any issues. It sounds like AsyncFileReader is a better place for the shim. I wonder if it could be put in there in such a way that it falls back to running on the current runtime, such that we don't need a separate implementation?

[alamb]

See also #13424 which contains some prior discussion, including why shimming ObjectStore in this way is potentially fraught (#13424 (comment) in particular)

FWIW I tried what I think is the alternate suggestion in #13690 and can't seem to make it work realistically: #13690 (comment)

So TLDR unless someone can come up with anything else, the approach in this PR is the only one that seems practical to me

[alamb]

See also

• Example for using a separate threadpool for CPU bound work (try 2) #14286

Which contains something similar to this cross runtime (IoObjectStore)

[adriangb]

closing in favor of #14286