[Experimental] Support direct query data sources

[lezzago]

Description

Support for queries other data sources like prometheus via REST API.
PR is a merge from feature/direct-query-prometheus. It is not a straight merge as that PR shows more changes than there are: #4357.

Most changes are isolated in the direct-query and direct-query-core modules

Introduces 3 new APIs:

• Direct query: Query non-opensearch datasources like Prometheus with PromQL
  • uri: GET /_plugins/_directquery/_query/{dataSources}
• Read resources: Fetch resources from non-opensearch datasources like Prometheus to fetch labels/series/alerts
  • base uri: GET /_plugins/_directquery/_resources/{dataSource}
    • example: GET /_plugins/_directquery/_resources/{dataSource}/api/v1/{resourceType}/{resourceName}/values
• Write resources: Write resources to non-opensearch datasources like Prometheus to put alert manager silences
  • base uri: POST /_plugins/_directquery/_resources/{dataSource}
    • example: POST /_plugins/_directquery/_resources/{dataSource}/alertmanager/api/v2/{resourceType}

Related Issues

#4229

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[joshuali925]

It is not a straight merge as that PR shows more changes than there are: #4357

Probably because I squashed #4374, GitHub had conflict to rebase it and merge is disabled here. Squash doesn't really merge it with main, and that commit content will show in the diff main...feature/direct-query-prometheus during review.

If you want to send the PR from feature branch let me know, i might be able to push the merge commit. But if not this works as well

[joshuali925]

i had the question since beginning: are these actionable TODO items, or are we ok to just leave them? I believe this is for transport layer communications, which we don't have a use case, so "real implementation" would mean when this request is being used at transport laayer?

[lezzago]

We have no current requirements for transport layer. We want to support this when its GA, but there will be a bit of refactoring needed, so was not worth to implement this yet.

[joshuali925]

direct query (non async query) shouldn't have a query id?

[lezzago]

I was thinking of keeping this for now for the purpose of tracing the call to debug errors.

[lezzago]

Probably because I squashed #4374, GitHub had conflict to rebase it and merge is disabled here. Squash doesn't really merge it with main, and that commit content will show in the diff main...feature/direct-query-prometheus during review.

If you want to send the PR from feature branch let me know, i might be able to push the merge commit. But if not this works as well

I am fine with either choice. If this would be a preferred approach, feel free to try to push a merge commit to fix that.

[Swiddis]

I looked over many of the original PRs going into this and don't have major architectural concerns. I'm okay to merge this as-is and address any smaller issues later. (Note: haven't carefully looked over the code here, this is a very large change.)

I'd like if you could add testing details for this though: what's the testing plan for the new module? How is it different from our current one? Any known bugs/operational changes we should be aware of?