Support for multiple networks

[karolz-ms]

Motivation

The purpose of this feature is to allow for modeling of applicatons that span multiple networks. This allows to cleanly solve existing problems with non-uniform networks, and opens up new possibilities for modeling more complex cloud application topologies.

Existing problems include

1. Handling container networks. We do not have a clean way to model the fact that Executables and Containers need to be addressed differently depending on what network (host vs container) the client vs the server are connected to. In some configurations (vanilla Docker, not "desktop" products) container-to-host communication is not possible at all.

2. Handling cloud development environment like GH codespaces. The problem with those is that there is a difference between the "application host" where the application workload runs vs "developer hsot" where user browser and some other tools run. Intra-app traffic can use direct "localhost" addressing, but everything coming from developer host needs proxying/tunneling and different addressing. Again, we do not have a clean way to model that.

3. Services can request binding to pseudo-addresses such as localhost, 0.0.0.0 (IPv4 "all available interfaces") or [::] (IPv6 "all available interfaces"). With Service spec having a single EffectiveAddress property, it does not allow us to properly report what addresses a Service is actually using once it completes network binding using a pseudo-address. Nor is it possible to smoothly handle the situation when the client requests specific port, but only some interfaces have that port available and others don't.

Design Outline

1. We introduce a new Network DCP primitive that allows us to model networks explicitly.
2. Endpoints become Network-specific (are associated with one or more networks).
3. Executables and Containers become attached to one or more Networks.
   • A client (Executable or Container) can only reach a Service if that Service has an Endpoint on the Network the client is connected to.
4. A new Gateway concept models facility for communication between networks that require addressing changes and/or traffic proxying.
   • Gateway controllers are responsible for setting up proxies and creating (endpoints) Endpoints that represent services that are not directly reachable by clients on different networks.
   • "Setting up proxies" might mean literally running proxy/tunnel software, or might involve detection/leveraging a proxy/tunnel that a developer has pre-configured outside Aspire.
5. Services that exist before application is started ("external services") can be modeled using existing Service and Endpoint objects.
6. The existing EffectiveAddress Service spec property is deprecated and replaced with EffectiveAddresses list that holds (address, port) tuples, solving problem 3. In the same work item, the existing AddressAllocationMode should be simplified and stop accepting modes that are identical to setting the Address spec property explicitly to some pseudo-address value (localhost, 0.0.0.0 or [::]). That would leave only Proxyless and likely AllInterfaces with its semantics of "bind to all interfaces but using single IP protocol version, i.e. IPv4 or IPv6).

Additional notes

Aspire will be the first consumer of this feature. We should prototype it together with corresponding Aspire model API and dashboard changes, to ensure that resulting set of APIs makes sense for all Aspire deployment environments.

This feature set involves several breaking changes. It probably makes sense to introduce a new version of the DCP API to accommodate all the changes, with all resources living in a namespace.

[karolz-ms]

@danegsta @dbreshears @davidfowl @mitchdenny please share your thoughts

[danegsta]

A few incredibly random thoughts:

I think we need to start thinking about service discovery semi-independent of the individual resource models. There's discrete resource types (a container, a container network, an executable, etc.) and while service discovery is something they need to do, the resource itself is primarily focused on managing the lifecycle of said resource. Create it, update it, clean it up at the end.

On the other hand, service discovery/networking is really building one (or more) directed graphs of connections between resources and querying that graph to find out how exactly your service A can talk to service B (port, address, etc.). We should model our service discovery types as a graph as much as possible (perhaps going so far as to actually build an in-memory graph model).

Beyond that, new "connection string" resources for client service discovery could be helpful. A resource that could accept a template string in the spec and the status would be the evaluated string (once all services required to populate the template were available).

[dbreshears]

Putting in 9.4 for investigation to determine where parts of this land

[dbreshears]

going to address this post 9.4

[karolz-ms]

For comparison, the following was my proposal for modeling multiple networks in Aspire

1. Aspire gets a concept of a Network. It is a logical grouping of resources with direct connectivity (no routing/NAT needed between them) and DNS/service discovery addressability.

2. During development run there is an implicit "local host Network", one or more container Networks (if containers are used by the application), and a "public Internet" Network.

3. At publish time a Network instance gets interpreted in the context of target deployment environment. I did some research and the proposed concept matches quite well the VNet/VPC/Google VPC/Kubernetes cluster network for Azure/AWS/Google cloud/vanilla Kubernetes respectively. They all provide flat address space and direct connectivity via IP addresses and DNS names.

4. Aspire resources can be associated with one or more Networks. This association comes in two forms: "reachable" and "connected". If the Network is reachable from a (client) resource, it means the client can access server resources on that Network directly, via DNS name or IP address. If a resource is "connected" to the Network, it can provide an Endpoint on that Network by listening on a TCP/UDP socket. Connected implies reachable. For example, localhost (loopback) network is both reachable and connected with respect to Executable resources. Public Internet and corporate VPNs are reachable but (typically) not connected with respect to Executable resources.

5. Each Endpoint is associated with exactly one Network and one "provider" Aspire resource. The provider resource is the resource that implements the service (contract) that the Endpoint represents.

6. EndpointReference resolution happens in the context of the client resource. If the targeted Endpoint belongs to a Network that is reachable from client resource, the resolution works pretty much as it does today, that is DNS/service discovery is used to resolve Endpoint properties. If the EndpointReference is resolved for an Endpoint that is on a Network which is NOT REACHABLE from the client resource, that is a run-time error and resolution fails.

7. Containers get special treatment during development run

   • If a container port is mapped into host network, the Container gets an additional host network Endpoint that matches the container network Endpoint.
   • If a Container takes a reference to a (server) resource on the host network, we create additional Endpoint on the container network that represents the server and matches the server Endpoint on the host network. At run time ApplicationOrchestrator/DcpExecutor create appropriate container tunnel DCP resource that facilitates this connectivity. This replaces the current container host name workaround, which is deprecated.

Note that in this model EndpointReference resolution is "one way" (is done in the context of the client resource), and an Endpoint can be associated with a Network that not connected to-, or even not reachable from the provider resource. This allows us to correctly model the Docker/Podman networking behavior with host-mapped ports, or locked down cloud environments where there might be inbound connectivity to a service, but the service itself cannot make outbound connections.

[MedAnd]

I’d like to mention two additional networking‑related scenarios:

1. Modeling the Sidecar pattern to better represent common service mesh architectures.
2. Chaos testing capabilities (e.g., fault injection, network disruptions) with integrated telemetry collection for analysis.