EG high memory usage

[qicz]

Version

the main branch

Resources Settings

limits: memory 1Gi, CPU 500m

Platform

Openshift

Description:
The current EG implementation watches a lot of resources, such as HTTPRoute, GPCRoute, EnvoyProxy, etc., and a copy of these resources will be reserved in the EG memory, causing the EG memory usage to be very high. When the user's HTTPRoute and other resources continue to increase, the memory of EG will take up more, and the user needs to adjust the resource limits dynamically ?

When any resource changes, EG will trigger the behavior of these changes, such as translation and infrastructure reconstruction.

gateway/internal/provider/kubernetes/controller.go

Lines 287 to 290 in 6af516d

	// The Store is triggered even when there are no Gateways associated to the
	// GatewayClass. This would happen in case the last Gateway is removed and the
	// Store will be required to trigger a cleanup of envoy infra resources.
	r.resources.GatewayAPIResources.Store(acceptedGC.Name, resourceTree)

gateway/internal/gatewayapi/runner/runner.go

Lines 51 to 66 in 6af516d

	func (r *Runner) subscribeAndTranslate(ctx context.Context) {
	message.HandleSubscription(r.ProviderResources.GatewayAPIResources.Subscribe(ctx),
	func(update message.Update[string, *gatewayapi.Resources]) {
	val := update.Value
	if update.Delete || val == nil {
	return
	}
	// Translate and publish IRs.
	t := &gatewayapi.Translator{
	GatewayControllerName: r.Server.EnvoyGateway.Gateway.ControllerName,
	GatewayClassName: v1beta1.ObjectName(update.Key),
	GlobalRateLimitEnabled: r.EnvoyGateway.RateLimit != nil,
	}

These are actually unnecessary. Is it possible to separate watch for related resources or separate store and subscribe when resources really change?

like this:

if val, ok := r.resources.GatewayAPIHTTPRoutesResources.Load(acceptedGC.Name); !ok || cmp.Diff(val, httpRoutes) != "" {
   r.resources.GatewayAPIHTTPRoutesResources.Store(acceptedGC.Name, httpRoutes)
}

[arkodg]

thanks for raising this issue @qicz . Can you also highlight

1. Number of resources you have running - Gateways, HTTPRoutes, etc ?
2. Version of EG you are running

Why is the memory increasing when the number of resources are increasing ?

• As you highlighted above, EG store relevant resources in the watchable store, so increase in resources increases the memory usage of the EG.
• This is an intentional design decision to
  • Reduce CPU - Caching helps us avoid translations if input resources haven't changed
  • Reduce I/O - Caching helps us avoid writes to Envoy (xds) or K8s API Server (status) if output resources haven't changed
    which comes at a cost of memory usage.

How can we deal with this memory usage at high scale ?

• Lets set the default memory limits to a higher value

  gateway/charts/gateway-helm/values.tmpl.yaml

  Line 8 in a5982c3

  limits:

  128M is too low imo
• Reduce Controller Runtime Cache - limit EG to a subset of namespaces Implement Watch Namespaces #1610. This won't affect the watchable store since its only saving the
  relevant/accepted resources
• @chauhanshubham are they any further optimizations we can make in the provider / reconcile logic ?

the change you've highlighted above with cmp.Diff() is inherently handled by watchable which is one of the primary reasons we chose to use it in this project. https://github.com/envoyproxy/gateway/blob/main/docs/latest/design/watching.md

[qicz]

thanks for raising this issue @qicz . Can you also highlight

1. Number of resources you have running - Gateways, HTTPRoutes, etc ?
2. Version of EG you are running

Why is the memory increasing when the number of resources are increasing ?

• As you highlighted above, EG store relevant resources in the watchable store, so increase in resources increases the memory usage of the EG.

• This is an intentional design decision to

  • Reduce CPU - Caching helps us avoid translations if input resources haven't changed
  • Reduce I/O - Caching helps us avoid writes to Envoy (xds) or K8s API Server (status) if output resources haven't changed
    which comes at a cost of memory usage.

How can we deal with this memory usage at high scale ?

• Lets set the default memory limits to a higher value

  gateway/charts/gateway-helm/values.tmpl.yaml

  Line 8 in a5982c3

  limits:

  
  128M is too low imo
• Reduce Controller Runtime Cache - limit EG to a subset of namespaces Implement Watch Namespaces #1610. This won't affect the watchable store since its only saving the
  relevant/accepted resources
• @chauhanshubham are they any further optimizations we can make in the provider / reconcile logic ?

the change you've highlighted above with cmp.Diff() is inherently handled by watchable which is one of the primary reasons we chose to use it in this project. https://github.com/envoyproxy/gateway/blob/main/docs/latest/design/watching.md

1. just 69 HTTPRoute

2. EG version: the main branch 3. resources limits settings:

IMO #1617 can not resolve this thoroughly.

[qicz]

fix the sa create or update bug like #1609, the limits just need 1Gi, but the EG occupies the same high mem.

[arkodg]

@qicz would be great if you can attach the pprof you shared earlier which highlights the high mem usage due to the secrets being cached. We can then see if

• we are caching relevant secrets or all secrets

  gateway/internal/provider/kubernetes/predicates.go

  Line 71 in 708287e

  func (r *gatewayAPIReconciler) validateSecretForReconcile(obj client.Object) bool {

• if above logic is in fact right, we can attempt to skip caching secrets and configmaps and directly retrieve them from API server instead of using the Cache

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

has this issue been resolved @qicz ?

[qicz]

has this issue been resolved @qicz ?

fixed by #1594 and #1609 , and I will continue to follow EG performance issues