UTF8 byte order marker breaks jsoncpp

[psifertex]

Describe the bug
The documentation says that jsoncpp supports valid UTF-8. Unfortunately the optional Byte Order Mark allowed by the spec causes jsoncpp to fail to parse otherwise valid utf8.

To Reproduce

1. Build example readFromStream.cpp
2. Validate it works on included withComment.json via: ./readFromStream withComment.json
3. Create a new file: (echo -en '\xef\xbb\xbf'; cat withComment.json) > withBom.json
4. Run: ./readFromStream withBom.json

Expected behavior
The same output on step 4 and step 2. Instead, you get:

* Line 1, Column 1
  Syntax error: value, object or array expected.

Desktop (please complete the following information):

• MacOS
• (brew installed jsoncpp)

[dota17]

RFC 8259, Section 8.1:

Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a networked-transmitted JSON text.

The BOM character is illegal in JSON.

[psifertex]

And yet that doesn't stop some implementations from doing it wrong (https://stackoverflow.com/questions/26746498/remove-byte-order-mark-in-r-c?lq=1)

I agree, it should never be included by proper generators but that doesn't mean it won't accidentally happen some times (some copy/paste workflows introduce it which is how we discovered it in our use of jsoncpp).

Given the general guidance of RFC 1122 it seems appropriate to more gracefully skip/ignore it as opposed to failing to parse entirely.

[schroeder-]

This is the workaround I use against illegal UTF-8 BOM:

bool json_open(const std::string& file, Json::Value& root){
	std::ifstream config_doc(file.c_str(), std::ifstream::binary);
	//check for UTF-8 BOM and remove if needed
	char b[4] = {0, 0, 0, 0};
	char bom[] = {(char)239, (char)187, (char)191, 0};
	config_doc.read(b, 3);
	if(strcmp(b, bom) != 0){
		// If no UTF-8 BOM is found reposition stream
		config_doc.seekg(0, std::ios::beg);
	}
	std::string errs;
	Json::CharReaderBuilder builder;
	bool res = Json::parseFromStream(builder, config_doc, &root, &errs);
        ......
}

[psifertex]

Thanks @schroeder- that's the route we're planning on taking, may just be everyone needs to do their own pre-filtering. Appreciate the example.

[dota17]

jsoncpp/include/json/reader.h

Line 252 in 3beb37e

* \param beginDoc Pointer on the beginning of the UTF-8 encoded string

Except UTF8-BOM, other charset should be translated to UTF8.

[jfreilly]

RFC 8259, Section 8.1:

Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a networked-transmitted JSON text.

The BOM character is illegal in JSON.

Please read what you have written/copied. Its illegal in a specific context not in general if that is true. The specification goes on to say implementations may ignore bom marks in the interests of interoperability. In saying that the specification itself is flawed in the sense that a valid parser may fail on a valid file containing bom marks for example. That is possibly not the intention of the author.

[dota17]

The specification goes on to say implementations may ignore bom marks in the interests of interoperability. In saying that the specification itself is flawed in the sense that a valid parser may fail on a valid file containing bom marks for example. That is possibly not the intention of the author.

@jfreilly
Right, if this library can ignore bom marks, i think it can be helpful.
Also, i think that there would be some messy if we ignore bom marks so that people would think the datas what they copy are UTF-8 encoded.

[dota17]

FWIW, Python's json parser seems that it does not support this directly either:
https://stackoverflow.com/questions/13156395/python-load-json-file-with-utf-8-bom-header

So, I think there is a better way to solve this probleam.
We can add a option features, just like emitUTF8. When it is true, we will ignore bom marks, and if not true, the input data must be UTF-8 encoded strictly.

[psifertex]

This would be a great option (pun intended).