Subscription model

[Nycander]

Stages should register their queries for subscription instead of polling Hydra core with queries.

This would enable a more streamlined message flow where hydra core is in control. It would also enable other backends to be implemented (such as message brokers).

[remen]

👍

[laserval]

When implementing this, we have to consider how the result of the processing is transmitted back to the core.
With the HTTP transport (RemotePipeline), I'm not sure which is the best approach:

• Replace polling and save(): Core POSTs documents to the Stage, which processes the document and returns the modifications in a response body
• Replace polling only: Core POSTs documents to the Stage, which responds directly. When the document has been processed, the stage uses the current HTTP API to send the document back to the Core

In the first case, the Core can easily handle timeouts and document state changes just by checking the response from the stage.