Dynamic metadata of type google::protobuf::Struct not working

[NomadXD]

In my scenario I'm setting up dynamic metadata from ext_authz filter and trying to read from my WASM filter as follows. But the method which is defined to get metadata returns false.

google::protobuf::Struct ext_metadata;
if (!getMessageValue<google::protobuf::Struct>(
           {"metadata", "filter_metadata", "envoy.filters.http.ext_authz"}, &ext_metadata)) {
     LOG_ERROR(std::string("filter_metadata Error ") + std::to_string(id()));
   }

That logs the error because the method getMessageValue returns false. I tried to debug the issue and the following section of logs shows the issue.

envoyproxy-websocket_1  | [2021-03-29 10:56:54.300][29][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm->host] env.proxy_get_property(5458080, 54, 5345664, 5345616)
envoyproxy-websocket_1  | [2021-03-29 10:56:54.300][29][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [host->vm] malloc(177)
envoyproxy-websocket_1  | [2021-03-29 10:56:54.300][29][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [host<-vm] malloc return: 5346384
envoyproxy-websocket_1  | [2021-03-29 10:56:54.300][29][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm<-host] env.proxy_get_property return: 0
envoyproxy-websocket_1  | [2021-03-29 10:56:54.300][29][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm->host] env.proxy_log(2, 5449512, 92)

I just tried the following to check whether a value exists in the ext_authz metadata and it returns true. Also the trace shows the metadata.

auto buf1 = getProperty<std::string>({"metadata", "filter_metadata", "envoy.filters.http.ext_authz"});
  if (buf1.has_value()) {
    LOG_INFO("Metadata exist");
  }

[NomadXD]

@PiotrSikora Can you please have a look ? I'm stuck with this for several days. And this is not specific to my use case. More like a common issue for someone trying to access dynamic meta data from WASM

[PiotrSikora]

What's the content of buf1? Could you print the value? If it's garbage, could you try to print buf1.pairs()?

[NomadXD]

@PiotrSikora The value for buf1 is as follows.
I did,

 if (buf1.has_value()) {
    LOG_INFO(buf1.value()->toString());
  }

envoyproxy-websocket_1  | [2021-03-31 10:29:07.543][26][info][wasm] [source/extensions/common/wasm/context.cc:1171] wasm log: [examples/wasm-websocket/filter.cc:90]::onRequestHeaders() $$$apiKey0263bc2e-9be3-4494-a3bc-2e9be32494d5subscriptionKeybc5f2c82-a4f7-4d34-9f2c-82a4f7bd34a5applicationKey7c4b8f93-8d5f-43c4-8b8f-938d5f93c436

So all my meta data are there. Any way I can get those as protobuf struct and assign it to a local variable ?

[PiotrSikora]

Could you try deserialize it as a pair?

if (buf1.has_value()) {
  auto pairs = buf1.value()->pairs();
  for (auto &p : pairs) {
    LOG_INFO(std::string(p.first) + std::string(" -> ") + std::string(p.second));
  }
}

Since getMessageValue<google::protobuf::Struct> doesn't work, I don't think this is received as a protobuf.

[NomadXD]

Yeah i could deserialize it as a pair.

envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][info][wasm] [source/extensions/common/wasm/context.cc:1171] wasm log: [examples/wasm-websocket/filter.cc:102]::onRequestHeaders() subscriptionKey -> bc5f2c82-a4f7-4d34-9f2c-82a4f7bd34a5
envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm<-host] env.proxy_log return: 0
envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm->host] env.proxy_log(2, 5374600, 114)
envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][info][wasm] [source/extensions/common/wasm/context.cc:1171] wasm log: [examples/wasm-websocket/filter.cc:102]::onRequestHeaders() applicationKey -> 7c4b8f93-8d5f-43c4-8b8f-938d5f93c436
envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm<-host] env.proxy_log return: 0
envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][trace][wasm] [source/extensions/common/wasm/wasm_vm.cc:40] [vm->host] env.proxy_log(2, 5374576, 106)
envoyproxy-websocket_1  | [2021-03-31 10:47:15.364][30][info][wasm] [source/extensions/common/wasm/context.cc:1171] wasm log: [examples/wasm-websocket/filter.cc:102]::onRequestHeaders() apiKey -> 0263bc2e-9be3-4494-a3bc-2e9be32494d5

So I'm get setting up dynamic metadata from the response of ext_authz. I think filter_metadata is in the form of map<string, google::protobuf::Struct> and when I consider metadata of one specific filter it's a protobuf struct. Here's what I set up from ext_authz response.

envoyproxy-websocket_1  | [2021-03-31 10:47:15.363][30][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:52] Received CheckResponse: status {
envoyproxy-websocket_1  | }
envoyproxy-websocket_1  | ok_response {
envoyproxy-websocket_1  | }
envoyproxy-websocket_1  | dynamic_metadata {
envoyproxy-websocket_1  |   fields {
envoyproxy-websocket_1  |     key: "apiKey"
envoyproxy-websocket_1  |     value {
envoyproxy-websocket_1  |       string_value: "0263bc2e-9be3-4494-a3bc-2e9be32494d5"
envoyproxy-websocket_1  |     }
envoyproxy-websocket_1  |   }
envoyproxy-websocket_1  |   fields {
envoyproxy-websocket_1  |     key: "applicationKey"
envoyproxy-websocket_1  |     value {
envoyproxy-websocket_1  |       string_value: "7c4b8f93-8d5f-43c4-8b8f-938d5f93c436"
envoyproxy-websocket_1  |     }
envoyproxy-websocket_1  |   }
envoyproxy-websocket_1  |   fields {
envoyproxy-websocket_1  |     key: "subscriptionKey"
envoyproxy-websocket_1  |     value {
envoyproxy-websocket_1  |       string_value: "bc5f2c82-a4f7-4d34-9f2c-82a4f7bd34a5"
envoyproxy-websocket_1  |     }
envoyproxy-websocket_1  |   }
envoyproxy-websocket_1  | }

[PiotrSikora]

There was a breaking change at some point that got sneaked past us, and google::protobuf::Struct (effectively, a list of key-value pairs) is now emitted as a serialized list of key-value pairs.

Generally, this is a good change, since it removes the unnecessary dependency on protobuf library inside Wasm.

[NomadXD]

Oh okay. So should I create a protobuf using these key value pairs (if possible) or go with these key value pairs ? I want to send these to a grpc service through a grpc bidi stream which accepts these as protobuf(I can change it to to a map also).

[PiotrSikora]

You have to use pairs(), because I don't think you can get the original google::protobuf::Struct via getProperty anymore.

[NomadXD]

@PiotrSikora okay thanks a lot for all the info and help. There was another issue I got when I try to use grpc bidi streams and I managed to make it work using a work around. I would really appreciate if you can give me any feedback or your opinions about that too. I know you guys are probably busy with work and all , just have a look when u get some time.

envoyproxy/envoy#15712

[NomadXD]

@PiotrSikora Just one more thing regarding the dynamic metadata. So if I set up a a boolean value in ext_authz , I get an empty field for that ??? All the string values works fine for me. But for other values, it's either empty or not in a proper format like this \000\000\000\000\000\000\000\000

[PiotrSikora]

@NomadXD the pairs serialization was designed specifically for string pairs, and if it's used for anything else, then it's most likely broken. See: envoyproxy/envoy#15196 (comment).