[Feature] Improving arrow flight sql with Kubernetes setup

[jonasbrami]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Related to this merge request #43281.

I am afraid of the performance degradation of using Arrow Flight SQL from outside the Kubernetes cluster because from my understanding, it would be like setting enable_parallel_result_sink = False . Would it be theoretically possible, instead of having random BE resolution for the reverse proxy, have a smarter reverse proxy that could be, for example, based on path prefix (doris-url/be1, doris-url/be2) ? Then those doris-url/be{i} would become the "advertised" urls for the grpc client and would be reachable from outside the kubernetes cluster. Do you think it would be a good idea?

Use case

Faster queries when using arrow flight sql from outside Kubernetes cluster

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[xinyiZzz]

Hi @jonasbrami , thanks for your suggestion.

Are you talking about implementing a custom reverse proxy in Kubernetes?

That's a great idea, when you're ready, you can describe how to do it, which hopefully will be a common feature.

[jonasbrami]

I was thinking about the following:

1. Doris Operator exposes configuration of a base public hostname for the Doris backends. For example public-hostname-doris.dev.
2. The operator would set in the be conf the public host to be something like public-hostname-doris.dev/be{I}.
3. Then it is possible to leverage something like Traefik to route all HTTP calls to public-hostname-doris.dev/be{I} to the right internal service.

Maybe it is more an issue to open for Doris Kubernetes operator than for Doris itself. The only requirement for Doris is to expose the public hostname config but it is already the case

[xinyiZzz]

I was thinking about the following:

1. Doris Operator exposes configuration of a base public hostname for the Doris backends. For example public-hostname-doris.dev.
2. The operator would set in the be conf the public host to be something like public-hostname-doris.dev/be{I}.
3. Then it is possible to leverage something like Traefik to route all HTTP calls to public-hostname-doris.dev/be{I} to the right internal service.

Maybe it is more an issue to open for Doris Kubernetes operator than for Doris itself. The only requirement for Doris is to expose the public hostname config but it is already the case

Sounds like a good idea, I will contact the developers of Doris Kubernetes Operator to confirm this.

[jonasbrami]

Coool! Thank you! I would be happy to try implementing this

[hechao-ustc]

I was thinking about the following:

1. Doris Operator exposes configuration of a base public hostname for the Doris backends. For example public-hostname-doris.dev.
2. The operator would set in the be conf the public host to be something like public-hostname-doris.dev/be{I}.
3. Then it is possible to leverage something like Traefik to route all HTTP calls to public-hostname-doris.dev/be{I} to the right internal service.

Maybe it is more an issue to open for Doris Kubernetes operator than for Doris itself. The only requirement for Doris is to expose the public hostname config but it is already the case

Hi, @jonasbrami, #42881 can this PR meet your 1~2 requirements. If yes, you can wait for this PR to be merged before doing the third requirement.

[github-actions]

We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!