req.read_body() api block HTTP2 GRPC request

[oowl]

we found req.read_body() api block HTTP2 GRPC request forever.

reproduce config

worker_processes 1;

events {
    worker_connections 1024;
}

http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';

    access_log logs/access.log main;

    # Add any necessary SSL settings here

    # Define the default server block
    server {
        listen       11080 http2;
        server_name  localhost;

        # Add any necessary location blocks here

        location / {
            content_by_lua_block {
                ngx.log(ngx.ERR, "start read.body()")
                ngx.req.read_body()  -- explicitly read the req body
                ngx.log(ngx.ERR, "end read.body()")
                local data = ngx.req.get_body_data()
                if data then
                    ngx.log(ngx.ERR, "body data:", data)
                end

                -- body may get buffered in a temp file:
                local file = ngx.req.get_body_file()
                if file then
                    ngx.log(ngx.ERR, "body data:", file)
                end
                                ngx.log(ngx.ERR, "Hello, world2!")
            }

            # Define your gRPC backend here
            # grpc_pass grpc://192.168.233.45:50051;
        }

        # Add any other necessary directives here
    }
}

and then execute

grpcurl -vv -d '{"name":"Kong"}' -plaintext 192.168.233.45:11080 helloworld.Greeter/SayHello

and error log

2023/03/28 15:28:15 [error] 868115#868115: *46 [lua] content_by_lua(nginx.conf:39):2: start read.body(), client: 192.168.0.4, server: localhost, request: "POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0", host: "192.168.233.45:11080"

as we can see, grpcurl bolck not return anymore, and error log can prove that the Lua code has been stuck in read_body there.

[oowl]

We carefully analyzed this issue and found that the problem is Openresty can not read the whole request body in any phase of the HTTP2 connection. This will cause Nginx to block the special Lua coroutine and prevent it from executing another phase.

In HTTP1, Nginx uses the content_len or chunked header to verify the request body is read completely. But in HTTP2, Nginx use HTTP2 data frame flag NGX_HTTP_V2_END_STREAM_FLAG flag to do it
https://github.com/nginx/nginx/blob/master/src/http/v2/ngx_http_v2.c#L1137

But In GRPC request, the HTTP2 stream may forever exist, it can not get the end of stream flag in the program lifetime, So openresty will spike this connection when the req.read_body() API was executed in the HTTP2 GRPC request.

[oowl]

So I think we must disable the req.read_body() API in the HTTP2 context(or fixing this problem, but I have not any idea to fix it), this will avoid some problems in the Openresty system. And HTTP2 is stream base protocol, read the whole body is theoretically impossible, and it does not make sense.

[zhuizhuhaomeng]

Your analysis is great.
OpenResty does not fully support HTTP2, and some APIs have been disabled.
Would you please create a PR for this?

[oowl]

Ok, I will create PR for this fix. @zhuizhuhaomeng

[oowl]

HTTP2 read_body API has disabled in lua-nginx-module master branch.