gRPC-web and Knative compatibility question

[feclist]

Ask your question here:

Hey all,

We are trying to get a gRPC python server to run as a Knative service that supports gRPC-web. Now after browsing around the docs/the overall internet we couldn't find many resources as to how to get this to work.

A brilliant blog written by Venil Noronha offers a guide as to deploy a gRPC-web application using k8s, istio and Envoy. Having read that blog and trying to replicate it with Knative doesn't seem to be as easy as Knative abstracts parts of this process away. What I suspect to be the issue with this transfer is that the k8s service port is not picked up by istio as a gRPC port hence rendering the EnvoyFilter useless. I might be wrong with this hypothesis, since my knowledge is not that thorough in this domain yet..

We did, however, get the protocol to work with gRPC-node following the gRPC PR in combination with the knative gRPC server - Go example. I am aware that gRPC-web works differently as it relies on an Envoy proxy to have the HTTP1 connection to be "upgraded" to a HTTP2 connection.

Bottomline, we need to figure out if there is a way to use an EnvoyFilter to make gRPC-web work with Knative. Our attempts thus far haven't been fruitful.

Thanks for taking time to read through this :) and hopefully anyone has been through this kind of problem already..

Cheers!

[feclist]

After more tinkering around we're almost confident in the fact that we can't specify a port for the k8s Service as istio would need to have a port defined with the name grpc-port. This is described in the blog by Venil Noronha.

I cannot find any resources on how this would be possible as the knative spec doesn't allow for a service to get a port assigned (which is sensible I guess..)

[tcnghia]

You can specify port name h2c. Can you please try our gRPC sample https://knative.dev/docs/serving/samples/grpc-ping-go/index.html and see if you can use that for gRPC-web?

[feclist]

Thanks for looking into this @tcnghia.

We followed the example step by step. Now trying to reach the endpoint we set up with gRPC-web comes back with a CORS policy error. This makes me think that the request is not 'upgraded' by the Envoy Proxy or some similar issue to that. There might be a missing link that understands these incoming requests to be upgraded from HTTP1.1 to HTTP/2, but that is merely a guess on my end as I don't know what h2c solves out of the box.

I've tried to connect the client through both ports 80 and 8080. Is there another port we need to connect to rather than those two in order to make it work?

Thanks :)

[tcnghia]

No, port 8080 is correct.

Let's turn this issue into a feature request, since base on what you describe we probably aren't able to support grpc-web right now.

[feclist]

Sounds great to me.

Should I change something on the description/title to change it into a feature request or open a new issue?

[nak3]

/kind proposal
/assign

@feclist No problem. I am adding the label and will look into this. (sorry, it looks like I failed to add the label but never mind 😰)

[knative-prow-robot]

@nak3: The label(s) kind/proposal cannot be applied, because the repository doesn't have them

Details

In response to this:

/kind proposal
/assign

@feclist No problem. I am adding the label and will look into this.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[knative-prow-robot]

@nak3: The label(s) kind/proposal cannot be applied, because the repository doesn't have them

Details

In response to this:

/kind proposal
/assign

@feclist No problem. I am adding the label and will look into this. (sorry, it looks like I failed to add the label but never mind 😰)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nak3]

I've verified the issue. And here is the quick summary:

Issue summary

• The current Serving does not work grpc-web.
• There are two missing configs EnvoyFilter and corsPolicy in VirtualService. (both are configured in the blog post @feclist mentioned.)
• After adding the two configs, hello world example worked fine.

Need to implement

• VirtualService adds corsPolicy when grpc-web is used in port name.
• For EnvoyFilter, we do not need to create it if we use port name w/ grpc-web prefix, thanks to pilot: add support for grpc-web prefixed ports istio/istio#10064. (I've verified this as well.)

For more detail about port name, I verified that we can use http2(h2c) for grpc-web, but we need to create EnvoyFilter manually. So, using grpc-web would be better.

[nak3]

Just a small update.
The usage in the blog adds envoy filter to side car. But it means we can use gRPC web only with mesh mode. For Knative, the access is always uses gateway, so we should add envoy filter to the Gateway instead of side car. I filed istio/istio#20537 and will consider to add envoyfilter to Knative's Gateway.

[tcnghia]

@nak3 since istio/istio#20537 is merged, I think we can do this as soon as we use Istio 1.15?

[nak3]

Yes, that is correct. We need to wait for Istio v1.15 release. And once 1.15 is released, we need to implement the logic which adds following corsPolicy in vitualservice when GRPC-WEB is used.

corsPolicy for grpc-web in VirtualService

    corsPolicy:
      allowOrigin:
        - "*"
      allowMethods:
        - POST
        - GET
        - OPTIONS
        - PUT
        - DELETE
      allowHeaders:
        - grpc-timeout
        - content-type
        - keep-alive
        - user-agent
        - cache-control
        - content-type
        - content-transfer-encoding
        - custom-header-1
        - x-accept-content-transfer-encoding
        - x-accept-response-streaming
        - x-user-agent
        - x-grpc-web
      maxAge: 1728s
      exposeHeaders:
        - custom-header-1
        - grpc-status
        - grpc-message
      allowCredentials: true