feat: add TCP advertise address support for cross-cluster deployments#2
Merged
Conversation
Add support for separating bind and advertise addresses on both the request plane and response plane TCP servers. This enables deployments where the frontend pod IP is not directly routable by workers (e.g., multi-cluster setups with NAT or VPN). New environment variables: Response TCP server (TcpStreamServer): - DYN_TCP_RESP_HOST: bind host override (default: auto-detect) - DYN_TCP_RESP_PORT: bind port override (default: OS-assigned) - DYN_TCP_RESP_ADVERTISE_HOST: advertise host override (default: bind host) - DYN_TCP_RESP_ADVERTISE_PORT: advertise port override (default: bind port) Request plane TCP (endpoint registration): - DYN_TCP_RPC_ADVERTISE_HOST: advertise host override (default: DYN_TCP_RPC_HOST) - DYN_TCP_RPC_ADVERTISE_PORT: advertise port override (default: DYN_TCP_RPC_PORT) When advertise vars are set, the server binds locally as usual but communicates the advertise address to peers via connection_info and etcd registration. This allows exposing the TCP ports via a NodePort, LoadBalancer, or VPN endpoint without changing how the server binds.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34147ea983
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Reject port 0 from DYN_TCP_RPC_ADVERTISE_PORT and DYN_TCP_RESP_ADVERTISE_PORT so templated configs that default to 0 fall through to the actual bound port instead of publishing an unconnectable address.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for separating bind and advertise addresses on the TCP response and request plane servers. This is needed for multi-cluster dynamo deployments where the frontend pod IP is not directly routable by remote workers.
Problem
In a cross-cluster setup (e.g., frontend on GKE, workers on bare-metal MicroK8s connected via Tailscale/VPN), the frontend's response TCP server auto-detects its pod IP and embeds it in
connection_infosent to workers. Workers then try to connect back to that pod IP, which isn't routable from outside the cluster.Solution
New environment variables allow the advertise address to differ from the bind address:
Response TCP server:
DYN_TCP_RESP_HOST— bind host overrideDYN_TCP_RESP_PORT— bind port overrideDYN_TCP_RESP_ADVERTISE_HOST— what gets sent to workers in connection_infoDYN_TCP_RESP_ADVERTISE_PORT— advertise port (e.g., NodePort)Request plane TCP:
DYN_TCP_RPC_ADVERTISE_HOST— overrides host in etcd registrationDYN_TCP_RPC_ADVERTISE_PORT— overrides port in etcd registrationWhen advertise vars are unset, behavior is unchanged (bind address = advertise address).
Example usage
Frontend exposed via a Kubernetes NodePort on port 30100:
Workers connect to
frontend.example.com:30100, which routes to the frontend pod's actual TCP port.Test plan
DYN_TCP_RESP_ADVERTISE_HOSTset to a routable address