Need specification for how to parse unrecognized enums in JSON

[srcreigh]

I was forwarded to here by @tbkka over in the Swift protobuf library from this issue: apple/swift-protobuf#972

As it stands the JSON serialization option in the swift-protobuf library is unusable because when it sees an unrecognized enum value that has been serialized to a JSON string, it throws an exception. There is no option to ignore the unrecognized string enum value, the only behaviour that swift-protobuf offers is to throw an exception.

To me it is abundantly obvious that any protobuf library has to ignore information that was generated using a newer schema.

For example, if a new enum field is shipped on a proto with v12.5 of an iOS app but is totally ignored (maybe it's a new feature under a shared schema that is implemented on a web app and is not implemented on the mobile app for a while). Then when this field, which is totally ignored, gets a new enum value, the v12.5 version of the app breaks because the only thing this that swift-protobuf can do is throw an exception when it sees the unrecognized enum value.

Of course this scenario is the core problem that protocol buffers as a whole set out to solve.

Protocol buffers were designed to solve many of these problems:

• New fields could be easily introduced, and intermediate servers that didn't need to inspect the data could simply parse it and pass through the data without needing to know about all the fields.
  https://developers.google.com/protocol-buffers/docs/overview#a-bit-of-history

(Of course an iOS app is not a server but it's the same thing as far as this is concerned because it participates in a distributed protocol.)

Could someone please provide a specification for what a protobuf library should do when it sees an unrecognized Enum value serialized in a JSON string?

I can't imagine that every other protobuf client that supports JSON serialized string enum values here crashes when it sees something unrecognized. Can someone point me to the behaviour of some of the other client libraries so that the folks over at swift-protobuf can have some precedence for how to handle this situation?

Thank you.

[acozzette]

I'm not sure about other languages, but at least for C++ you can set an option to ignore unknown fields during JSON parsing, and this includes known fields with unknown enum values. The documentation here says: "Proto3 JSON parser should reject unknown fields by default but may provide an option to ignore unknown fields in parsing."

[tbkka]

Could you please update the spec to explicitly call out enums as falling into this case? It would also help if you could clarify expectations here for repeated enum fields.

And of course, expanding the conformance tester to verify these cases would be a nice treat!

[thomasvl]

Also please document (add conformance tests) for what this means when enums are integer (vs. strings), and the value isn't a known value.

[thomasvl]

@acozzette the only place I can find code for the C++ side seems to be within JsonToBinaryStream in json_utils.cc -

protobuf/src/google/protobuf/util/json_util.cc

Lines 186 to 187 in 7bff839

	proto_writer_options.ignore_unknown_enum_values =
	options.ignore_unknown_fields;

Does that come into play in normal parsing (as opposed to transcoding which is what that api sorta looks like)?

Tracing the code a little further, there is a listener to get some feedback for that, but a normal parse of json to an object doesn't have a listener, so it is a silent data loss?

[thomasvl]

Oh, and an enum field within a oneof also needs a spec. In all these cases, does the value get ignored or does it get the default value?

Oops, and for map<> usage, how does that end up working?

[lc-nuorder]

Hello all! Does anyone have any updates for the questions asked by @thomasvl? I've currently encountered the same error as detailed in apple/swift-protobuf#972 and would love to see how we can move this forward to get it merged in?

[doom-goober]

Oops, and for map<> usage, how does that end up working?

Enums are not allowed to be keys for a map. From https://developers.google.com/protocol-buffers/docs/proto3#maps:

Note that enum is not a valid key_type

Here is the change the Java team made to support unknown enums and ignore unknown fields: #4825. It seems they just explicitly check the field is an enum and rather than throwing exception, they parse unknown enums as default enum value.

[thomasvl]

Enums are only allowed as map values, not keys.

[kristianmo]

The same issue exists in C#.