Trouble sending JSON post data to a service

[amitsaha]

Hi all, posted my query about having trouble sending JSON body in a POST request here. Wondering if I am doing something wrong here or perhaps a hidden actual issue here?

Summarized the problem here:

I am just writing a simple "API Gateway" using the different go-kit libraries. The current code is at https://github.com/amitsaha/apigatewaydemo/blob/master/src/apigateway/main.go

This gateway currently routes HTTP POST requests to 127.0.0.1:8000/projects to another service I have running (a Python web app on port 5000) which I discover via consul. The routing happens correctly, however, I am not able to send the JSON body through to my web app. The encodeJSONRequest() function is the one which is supposed to do this correctly: https://github.com/amitsaha/apigatewaydemo/blob/master/src/apigateway/main.go#L48 however it doesn't seem to be working.

Example request:

$ http POST 127.0.0.1:8000/projects/ Auth-Header-V1:123 title=MyProject121

I modified the httpclient to dump the request it is sending and this is what it looks like:

"POST /create HTTP/1.1\r\nHost: 127.0.0.1:5000\r\nUser-Agent: Go-http-client/1.1\r\nTransfer-Encoding: chunked\r\nAccept-Encoding: gzip\r\n\r\n1\r\n{\r\n18\r\n\"title\":\"MyProject121\"}\n\r\n0\r\n\r\n"

It seems like the body is not being encoded correctly?

Thanks.

[peterbourgon]

I don't know what your projects service is, so I've created this mock implementation in Go. I start that up and observe that it registers in Consul:

ugh ~/tmp/projects ./projects
service=projects tags=[] address=127.0.0.1 action=register
HTTP=8080

    2016/11/14 20:05:44 [DEBUG] agent: Service 'consul' in sync
    2016/11/14 20:05:44 [INFO] agent: Synced service 'projects_id'
    2016/11/14 20:05:44 [DEBUG] http: Request GET /v1/health/service/projects?index=141&passing=1 (4.656141725s) from=127.0.0.1:63804
    2016/11/14 20:05:44 [DEBUG] agent: Node info in sync
    2016/11/14 20:05:44 [DEBUG] http: Request PUT /v1/agent/service/register (520.496µs) from=127.0.0.1:63810
    2016/11/14 20:05:44 [DEBUG] http: Request GET /v1/health/service/verification?index=141&passing=1 (4.656154616s) from=127.0.0.1:63798
    2016/11/14 20:05:44 [DEBUG] agent: Service 'consul' in sync
    2016/11/14 20:05:44 [DEBUG] agent: Service 'projects_id' in sync
    2016/11/14 20:05:44 [DEBUG] agent: Node info in sync
    2016/11/14 20:05:44 [DEBUG] agent: Service 'consul' in sync
    2016/11/14 20:05:44 [DEBUG] agent: Service 'projects_id' in sync
    2016/11/14 20:05:44 [DEBUG] agent: Node info in sync

I then start your apigateway with the defaults, and observe that it finds the projects service.

ugh ~/src/github.com/amitsaha/apigatewaydemo (master) apigateway
ts=2016-11-15T01:06:15Z caller=subscriber.go:48 service=projects tags=[] instances=1
ts=2016-11-15T01:06:15Z caller=subscriber.go:48 service=verification tags=[] instances=0
ts=2016-11-15T01:06:15Z caller=main.go:254 transport=HTTP addr=:8000

Finally, I send a request to the apigateway. It appears to successfully go thru the apigateway, to the projects service, and a request comes back the other side. (It's an empty JSON object because I'm returning an arbitrary JSON map from my projects service, and it's not decoding properly into the expected struct. But that's fine.)

ugh ~ curl -Ss -H 'Auth-Header-V1: Foo' -d'{"title":"bar"}'  -XPOST 'http://localhost:8000/projects/'
{}

ugh ~/tmp/projects ./projects
service=projects tags=[] address=127.0.0.1 action=register
HTTP=8080
Title: "bar"

So, I suspect there's some problem in your Python service. I think I've given you enough information — can you reproduce my results? Can you identify any problem with your Python service that might lead to this behavior?

[amitsaha]

Hi Peter, thanks a lot for your detailed reply. I will do some more troubleshooting.

However, my Python web application could successfully decode/recieve the JSON data when I talked to it via httpie or another HTTP client. I will take a look once again though.

[amitsaha]

Hello @peterbourgon I finally sat down with this. When I replaced my Python service via your Golang server, it works, all is great. However, I do think there is something wrong in how I am encoding my JSON request to the downstream service (backend) when it is written in a non-Golang language. For example, I tried creating a proxy for httpbin.org, so basically, when I make a request to my API gateway to "/projects", it POSTs a request to httpbin.org/post. I have the following consul service registered:

{"service": {"name": "projects", "tags": [""], "address": "54.175.219.8", "port": 80}}

(The IP address above is that of httpbin.org)

Here is what I see when I do so via the API gateway:

$ http POST 127.0.0.1:8000/projects/ Auth-Header-V1:3121 title=myProject2
HTTP/1.1 200 OK
Content-Length: 35
Content-Type: application/json; charset=utf-8
Date: Fri, 25 Nov 2016 01:44:18 GMT

{
    "url": "http://54.175.219.8/post"
}

This basically means that I am not sending any JSON data from my API gateway. However, when I call the service directly:

$ http POST 54.175.219.8/post title=project
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 425
Content-Type: application/json
Date: Fri, 25 Nov 2016 01:44:52 GMT
Server: nginx

{
    "args": {},
    "data": "{\"title\": \"project\"}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "Content-Length": "20",
        "Content-Type": "application/json",
        "Host": "54.175.219.8",
        "User-Agent": "HTTPie/0.9.2"
    },
    "json": {
        "title": "project"
    },
    "origin": "58.173.10.146",
    "url": "http://54.175.219.8/post"
}

I see the same result in another server I wrote in Crystal

Do let me know how I can help debug this issue or solve my problem..

[peterbourgon]

I do think there is something wrong in how I am encoding my JSON request to the downstream service (backend) when it is written in a non-Golang language.

I don't doubt that you're experiencing trouble, but I'm not really sure what you're asking me, and I'm not really sure how I can help. I think we've proven that the apigateway is sending correct/proper JSON requests to its downstream services. If you believe it should be doing something else, can you please be more specific?

[peterbourgon]

Maybe this helps:

Here is what I see when I do so via the API gateway:
snip
This basically means that I am not sending any JSON data from my API gateway.

No, I think this means you're not decoding the response from httpbin into a complete structure. That is, it looks like you're decoding into

type Response struct {
    URL string `json:"url"`
}

when you rather want something like

type Response struct {
    URL     string            `json:"url"`
    Args    map[string]string `json:"args"`
    Data    string            `json:"data"`
    Headers map[string]string `json:"headers"`
    // etc.
}

[amitsaha]

I don't doubt that you're experiencing trouble, but I'm not really sure what you're asking me, and I'm not really sure how I can help. I think we've proven that the apigateway is sending correct/proper JSON requests to its downstream services. If you believe it should be doing something else, can you please be more specific?

Sorry, I don't mean to imply that I want you to fix this without properly understanding what the issue is. If you can, can I please suggest you to write a downstream service in another language (not Golang) and see if it gets the JSON data correctly? I tried with another web application in another language, and it gets an empty JSON body.

Alternatively, if you don't have the time, feel free to let me know and I will ask around elsewhere.

[amitsaha]

Once I set the Content-Type to application/json, my second web application can see the JSON body now. So, I guess that was it. My Python web application still cannot decode the JSON correctly, though. But that is not for this issue. I will close it.

Thanks a lot for your help!

[amitsaha]

Finally! I had to set the Content-Length in my encoding function for Python to read the request body correctly..

Relevant line: https://github.com/amitsaha/apigatewaydemo/blob/master/apigateway/src/apigateway/main.go#L58