router: only cache single route for the active request

[wbpcode]

Commit Message: router: only cache single route for the active request
Additional Description:

In the previous impl of the route table, every active request will keep a reference of the whole route table. If there route table is updated frequently and there are some long-live http request, the old versions route tables cannot be released timely and increase the overhead of memory.

This PR try to fix this problem by cache single route for active request.

See #20800 for more detailed background.

Risk Level: High. Core code patch. Change the lifetime of route table.
Testing: Added/fixed.
Docs Changes: n/a.
Release Notes: n/a.
Platform Specific Features: n/a.

[alyssawilk]

/wait on CI - I think the router drop is real
Or were you waiting for first pass before writing tests? If so I think @adisuissa could do a round as this looks like it's a config lifetime change

[JakeCooper]

Hello from the Railway folks (original filers of #20800)

Patch is looking great! Will circle back if anything changes but, hopefully that provides a nice little datapoint "in the wild"

CC @char8

[wbpcode]

Will to write some tests to improve the coverage of the common/router.

It would be great if more experts could help to review this PR. This lifetime change always be dangerous patch.

[wbpcode]

cc @JakeCooper Thanks for your feedback. Note this patch is no completed and it couldn't be used in the production. Especially if scoped rds & ondemand rds are used by you.

[wbpcode]

I rechecked the code of ondemand rds. Seems the ondemand callback is managed by the route config provider. So this patch will not effect it.

[adisuissa]

Thanks! Overall, the code changes LGTM, but would add a few tests that validate that the new behavior is maintained (if the config is changed during execution time, the new one is latched onto).

IMO there is a fundamental question here: when should the filter-manager latch to the route table (see Matt's comment?
Prior to this PR the routing table is latched during the decodeHeaders() call.
This PR changes it to latch later when route() is called.
Changing the routing table view in the midst of routing is possible, but it should be reviewed by someone that understands the details of the router.
Also cc @envoyproxy/senior-maintainers to chime in on this.

Another alternative to consider is to explicitly release the routing table, once a request "realizes" that the route is fixed (i.e., no need to reroute). It may be a bit easier to understand the code, and clearly find the cases where there could be an old-new view conflict of the routing table. WDYT?

[wbpcode]

Prior to this PR the routing table is latched during the decodeHeaders() call. This PR changes it to latch later when route() is called.

In fact, we still will get the route table in the decodeHeaders() to get the initial route.

Another alternative to consider is to explicitly release the routing table, once a request "realizes" that the route is fixed (i.e., no need to reroute). It may be a bit easier to understand the code, and clearly find the cases where there could be an old-new view conflict of the routing table. WDYT

Sounds a good idea. Then we can keep current behaviors completely. Typically, we can do this in the router filter. And I think we can also make this a interface of StreamFilterCallback. May be call it freezeRouteCache or something.
cc @mattklein123 @alyssawilk

[repokitteh-read-only]

CC @envoyproxy/runtime-guard-changes: FYI only for changes made to (source/common/runtime/runtime_features.cc).

🐱

Caused by: #26045 was synchronize by wbpcode.

see: more, trace.

[mattklein123]

LGTM with a few comments, thank you for tackling this tough problem! Awesome.

/wait

[mattklein123]

Awesome work. LGTM!

[wbpcode]

Hi, @mattklein123 could you give a stamp again? 😸 @ Conflict resolved and minor update to the changelog format.