BackendTlsPolicy specify multiple targetRefs of the same service, only one will work

[haorenfsa]

Description:

What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.

When BackendTlsPolicy specify multiple targetRefs of the same service with different port, only the first one will work.

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

• start a service with 2 ports 8080 & 8081, and enable TLS with self-signed localhost certificate.
• create Gateway CR & 2 HTTPRoute CR for both ports.
• create ConfigMap my-ca with self-signed ca.crt.
• create BackendTLSPolicy

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: my-service
  namespace: default
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: my-service
    sectionName: "8080"
  - group: ''
    kind: Service
    name: my-service
    sectionName: "8081"
  validation:
    caCertificateRefs:
    - name: my-ca
      group: ''
      kind: ConfigMap
    hostname: localhost

• curl gateway 8080 service with https, ok
• curl gateway 8081 service with https
• got following error:

Client sent an HTTP request to an HTTPS server.

Environment:

Include the environment like gateway version, envoy version and so on.

gateway: v1.1.0

Logs:

Include the access logs and the Envoy logs.

[zhaohuabing]

Client sent an HTTP request to an HTTPS server.

@haorenfsa BackendTLSPolicy applies to the connection between the Envoy and the backend services. This error came from the connection between the client and the Envoy, so it has nothing to do with the BackendTLSPolicy.

Looks like the Gateway listener is HTTP protocol, so you got this error when sending an HTTP request to it.

Could you please paste the Gateway and HTTPRoute resources here?

[haorenfsa]

Em... Seems it has nothing todo with HTTPRoute or Gateway CR. I resolve it by removing the SectionName field.

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: my-service
  namespace: default
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: my-service
-    sectionName: "8080"
-  - group: ''
-    kind: Service
-    name: my-service
-    sectionName: "8081"
  validation:
    caCertificateRefs:
    - name: my-ca
      group: ''
      kind: ConfigMap
    hostname: localhost

[haorenfsa]

By the way, the error comes from envoy to its backend. The link route is like:

client ---- https (SNI: myservice.com) ----> envoy ---- https (SNI: localhost) ----> backend

This happens when I'm using GCP L7 LoadBalancer with HTTP2/GRPC backend, and try migrate to L4 LoadBalancer with envoy gateway. GCP forces its backend to enable tls.

I created A Gateway with listener set to tls.mode= Terminate. Then I add HTTPRoute, and this BackendTLSPolicy.

[arkodg]

reopening the issue, because the config you specified, although verbose, is still valid, and should work

the other issue around upstream tls may be related to SNI and SAN, you may find the cert generation section in https://gateway.envoyproxy.io/docs/tasks/security/backend-tls/ useful

[haorenfsa]

envoy has some special constraints aside from xDS protocol definition, as other updates must arrive after CDS udpates:

https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#xds-ack-nack

envoyproxy/go-control-plane#1035