Functions Framework does not work with the Pub/Sub emulator

[RomikimoR]

Hello,

I am trying to emulate the behavior of my application.
To do so I use the Pub/Sub Emulator and the Google cloud function emulator.
The cloud function I created looks like the image following and is supposed to be triggered by an event :

Both emulators are communicating without issue.
I followed the example of this repository: https://github.com/GoogleCloudPlatform/functions-framework-nodejs to link them.

When, following the example, I use this command line:

curl -d "@mockPubsub.json" -X POST
-H "Ce-Type: true"
-H "Ce-Specversion: true"
-H "Ce-Source: true"
-H "Ce-Id: true"
-H "Content-Type: application/json"
http://localhost:8080

On my Cloud function emulator I do receive this :

So this is exactly what I want.
Yet, I do not want to use the command line to send messages but I want the cloud function to be triggered by a Pub/Sub event.

Here you can see the methods creating:
I. a pull subscription
II. a push subsciption
III. publishing messages

However When I want to publish messages, the Pub/Sub emulator detect the connection:

[pubsub] INFO: Detected HTTP/2 connection.

But then the Cloud Function emulator does not receive the data, although it detects an event occurred:

I think it is probably because POSTING to a Gcloud Function needs some headers with the 'Ce-' prefix.
But searching a bit on how to add headers to my push subscriber I found out this wasn't possible.
It's possible I totally missed something...

I hope this is understandable.
Thank you

[di]

Thanks for the report, I'll take a look at this and see what's going on.

[wojtek-viirtue]

+1 on this. I can confirm I'm seeing the same exact result for the same exact use case.

[isaldana]

It looks like when setting Ce-Type, Ce-Specversion, Ce-Source, and Ce-Id the request body is not JSON parsed. Unfortunately, when they are not set, only the data portion of the request are passed on the first parameter to the function. According to the samples for Pub/Sub-triggered functions, the first parameter should be a hash with the data and attributes fields. As is, it looks like attributes could not be accessed. I used the following command to test:

curl --header "Content-Type: application/json" --request POST --data '{"data":"This is a message", "attributes":{"k1":"v1"}}' http://0.0.0.0:8091/

on the following function

def main(event, context):
    print(event, context)

with the following output

127.0.0.1 - - [01/Mar/2020 01:40:42] "POST / HTTP/1.1" 200 -
This is a message {event_id: , timestamp: , event_type: , resource: }

[yurkovoznyak]

I've made fork of this repo, that supports events from Pub/Sub. Not the cleanest/best solution, but it works for my needs. Hope this may help
https://github.com/uraniun/functions-framework-python

[RomikimoR]

I will try the fork!! Thank you a lot I really needed a patch even not the cleanest for this !

[grant]

Thanks for the report and workaround. As Dustin stated, we'll look into this (it may be a similar issue with other frameworks).

[di]

Just a quick update here: we've decided to add support for these legacy-style Pub/Sub events. Hang tight while we roll this out, and thanks for your patience.

[dror-g]

Hi, not sure but I think it's related:
When using the Python function code example from Pub/Sub Tutorial
I get an error when using curl to emulate an event (from nodejs framework, same as OP).

main.py", line 17, in hello if 'data' in event: TypeError: a bytes-like object is required, not 'str'

cheers

[di]

@dror-g It looks like your function expects a Pub/Sub message but you're calling it with headers that indicate that it's a binary CloudEvent instead.

Try calling it like so:

$ curl -d '{"data": {"hi": "there"}}' -X POST -H "Content-Type: application/json" http://localhost:8080

Where the value of data is what eventually becomes the event object in your function.

[dror-g]

Hi @di , yes, like OP I am trying to emulate Pub/Sub triggered function, and we both tried to post a mock.json file.
functions-framework seems to be working as long as you BASE64 encode the body (as it is in the cloud) and enclose it in {"data": {"data": "$B64_string"} }.

Tried this function:

from json import loads
from base64 import b64decode


def wak(event, context):
    print("EVENT", event)
    if 'data' in event:
        data = loads(b64decode(event['data']).decode('utf-8'))
        print('data: ', data)
    else:
        raise RuntimeError('no data in event')
    return

What works:

• started with functions-framework --target wak --signature-type event
  calling with: DATA=$(printf '{"some": "thing"}'|base64 -w 0) && curl -d '{"data": {"data": "'$DATA'"}}' -X POST -H "Content-Type: application/json" http://localhost:8080

• Deploying to the cloud and sending a message from Pub/Sub web UI (message body: {"some": "thing"} ).

• calling from gcloud cli with following command:
  ` DATA=$(printf '{"some": "thing"}'|base64 -w 0) && gcloud functions call testFunc --region europe-west2 --data '{"data": "'$DATA'"}'

• created notification on a bucket, uploaded file to bucket (gsutil notification create -t test-topic -f json gs://test-func-bucket2).

What doesn't:
started with functions-framework --target wak --signature-type event

mock.json:

{
  "message": {
    "attributes": {
      "key": "value"
    },
    "data": "SGVsbG8gQ2xvdWQgUHViL1N1YiEgSGVyZSBpcyBteSBtZXNzYWdlIQ==",
    "messageId": "136969346945"
  },
  "subscription": "projects/myproject/subscriptions/mysubscription"
} 

• curl -d "@mock.json" -X POST -H "Content-Type: application/json" http://localhost:8080 -
  EVENT is null, getting:

RuntimeError: no data in event
127.0.0.1 - - [21/Apr/2020 20:36:30] "POST / HTTP/1.1" 500 -

• curl -d '{"hi": "there"}' -X POST -H "Content-Type: application/json" http://localhost:8080 -
  EVENT is null, getting:

RuntimeError: no data in event
127.0.0.1 - - [21/Apr/2020 20:36:30] "POST / HTTP/1.1" 500 -

• curl --data '{"data": {"hi": "there"}}' -X POST -H "Content-Type: application/json" http://localhost:8080
  Returns:

EVENT {'hi': 'there'}
RuntimeError: no data in event
127.0.0.1 - - [21/Apr/2020 20:50:45] "POST / HTTP/1.1" 500 -

[27Bslash6]

Thanks for the report and workaround. As Dustin stated, we'll look into this (it may be a similar issue with other frameworks).

I had this same issue working with the nodejs pubsub client for cloud functions

[mikebridge]

Just so I'm clear, is this an issue with the functions framework, the pubsub emulator, someone's understanding of the correct payload format, or the (adapted) documentation around the technique for connecting the emulator to the functions framework? I'd like to try using the functions framework as part of an end-to-end test with the pubsub emulator but I'm not clear whether this would be a blocking issue.

Or, after reading @di 's comment, does this only affect legacy triggers?

[mikebridge]

Google's "getting started" documentation describes the event as a python dict, but I can see that when posting via curl with the OP's test script will pass the event as a str. So is the bug then with the http testing endpoint?

Maybe the easiest workaround is to write a wrapper function just for testing that json.loads the event and passes it to the function?

def hello_pubsub_test_wrapper(event_raw, context):
    return hello_pubsub(json.loads(event_raw), context)

def hello_pubsub(event: dict, context):
    """Background Cloud Function to be triggered by Pub/Sub.
    Args:
         event (dict):  The dictionary with data specific to this type of
         event. The `data` field contains the PubsubMessage message. The
         `attributes` field will contain custom attributes if there are any.
         context (google.cloud.functions.Context): The Cloud Functions event
         metadata. The `event_id` field contains the Pub/Sub message ID. The
         `timestamp` field contains the publish time.
    """
    import base64

    print("""This Function was triggered by messageId {} published at {}
    """.format(context.event_id, context.timestamp))
    if 'data' in event:
        name = base64.b64decode(event['data']).decode('utf-8')
    else:
        name = 'World'
    print('Hello {}!'.format(name))

functions-framework --target=hello_pubsub_test_wrapper --signature-type=event

echo -n "test" | base64 
=>
dGVzdA==

mockPubsub.json:

{
  "data": "dGVzdA==",
  "messageId": "136969346945"
}

curl -d @mockPubsub.json -X POST \ 
    -H "Ce-Type: true" \
    -H "Ce-Specversion: true" \
    -H "Ce-Source: true" \
    -H "Ce-Id: true" \
    -H "Content-Type: application/json" \
    http://localhost:8080

This yields: Hello test!.