Lazy loading (on demand) for Routes, Clusters and Endpoints

[nt]

For requests that do not have routes, I would need to write a filter that can send xDS queries to get a Route Added with the associated configuration (cluster def and endpoints).

The use case is for deployments with a very high cluster cardinality and failing the request is not an option.

I do not know if this should be an envoy feature but it does not seem like it can be implemented as a custom filter as there are not xDS headers in include/envoy.

Any pointers on how to achieve this would be greatly appreciated.

cc @mattklein123 @htuch

[ramaraochavali]

@nt I have opened similar issue in the past. See if this #2022 helps. Looks very close though.

[mattklein123]

Here are my rough implementation notes for how I would go about doing this:

Initially I would probably implement a new gRPC API for this purpose. I think you could use the existing CDS API roughly, but operate it yourself by running your own gRPC fetcher on the main thread (created during filter initialization -- there are singleton examples of this already including RDS itself). The API you want might actually be unary vs. streaming also. Roughly you want to fetch a cluster given a name, without any streaming. I think ultimately we can figure out how to fold this back into the main CDS API definition as well as internal Envoy implementation, but my recommendation is to initially implement out of band while we explore the design space. Implementing an API like this in Envoy is pretty trivial with the interfaces we have.

The request flow looks something like:

1. When in request context, call ClusterManager::get() to get the thread local cluster that you are looking for. If it already exists, proceed per normal.
2. If the cluster does not exist, post a message to the main thread asking to acquire the cluster, and block the request.
3. The main thread will make a unary call to your management server to fetch the cluster. This will return a envoy::api::v2::Cluster which can be fed directly into ClusterManager::addOrUpdatePrimaryCluster().
4. Add thread local callbacks that can be subscribed to on the worker threads for when clusters are added or removed (as an aside we need this anyway to make Redis work with CDS). The request can unblock itself when the cluster is created or some timeout occurs.
5. I would recommend also adding some type of timeout/TTL logic on the main thread which cleans up clusters that have not been used in a while. They can be removed with ClusterManager::removePrimaryCluster()

A few implementation notes:

1. Initially, to get things running, I would probably try to return static clusters using CDS and not use EDS. Ultimately though you should use EDS such that once a node is using a cluster, it can subscribe to streaming EDS updates until the cluster TTL expires causing its removal. Note that until CDS: Support cluster update warming #1930 is implemented, your API+EDS in this case won't work seamlessly since the cluster will get created before there are any hosts in it.
2. If you go with the TTL approach, you will need to let the main thread know when there are requests. This can be done using the existing cluster request stats which could be snapped occasionally by the main thread to see the last time they changed.
3. We could go straight to exposing the CDS API out of cluster manager, and allowing a filter to modify the set of watches (CDS today is an empty request, expecting all clusters in the response). This is the other way to implement this. It's not too terrible to expose this out of cluster manager, as cluster manager owns CDS, but like I was saying before, for simplicity I might just explore a prototype with a new API and then once we figure it out we can fold it back in.

@htuch is very familiar with all of this and should be able to help while I'm out on leave. LMK if what I wrote above makes sense or if you want any additional specific pointers and I can provide them.

[nt]

Thank you for the notes, I'll get started on this. I do not plan to implement TTL though as part of this change.

[htuch]

I'd add that code structure wise, I think it makes sense to have some kind of LazyClusterLoader object owned by the main thread that is responsible for doing the post (to itself) and maintaining any state associated with the in-flight pseudo CDS. I.e. there should be a separation of concerns between the filter and the code that is responsible for doing the fetch (which might be serving lazy loads from multiple worker threads and requests).

[htuch]

One middle ground from modifying xDS would be to establish an independent CDS channel, using the same streaming protocol, from the LazyClusterLoader, using code in https://github.com/envoyproxy/envoy/blob/master/source/common/config/grpc_mux_subscription_impl.h and https://github.com/envoyproxy/envoy/blob/master/source/common/config/grpc_subscription_impl.h.

That way, you can keep the CDS protocol, but not use the existing CDS implementation which will be not work well as structured today. You can post independent updates from LazyConfigLoader as Matt describes. The key difference will be that in onConfigUpdate(), you can process just singleton resources for cluster addition, rather than update the entire state of the world (which would cause cluster draining etc.).

[mattklein123]

+1 to all of @htuch's comments above.

[nt]

I have an initial PR out: #2740

[mattklein123]

Initial partial implementation of cross thread cluster creation here: #3479 cc @ramaraochavali

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.