BUG - json dump field with unicode -> array of ints (instead of string)

[jossef]

I'm expecting the following code:

json result;
result["name"] = L"אינעל דינק";
std::cout << result.dump(4) << std::endl;

to output:

{
    "name": "\u05D0\u05D9\u05E0\u05E2\u05DC \u05D3\u05D9\u05E0\u05E7"
}

However, it treats the unicode string as json array, outputs instead:

{
    "name": [
        1488,
        1497,
        1504,
        1506,
        1500,
        32,
        1491,
        1497,
        1504,
        1511,
        0
    ]
}

BTW Thanks for this project. cool stuff happen here.

[nlohmann]

Hm. It works when you remove the wide string literal:

json result;
result["name"] = "אינעל דינק";
std::cout << result.dump(4) << std::endl;

->

{
    "name": "אינעל דינק"
}

To get the code points escaped, you need to use the ensure_ascii parameter:

std::cout << result.dump(4, ' ', true) << std::endl;

->

{
    "name": "\u05d0\u05d9\u05e0\u05e2\u05dc \u05d3\u05d9\u05e0\u05e7"
}

[nlohmann]

Wide strings are not supported by the library. An array is created, because under the hood this constructor is chosen:

template<typename BasicJsonType, typename T, std::size_t N,
         enable_if_t<not std::is_constructible<typename BasicJsonType::string_t, T (&)[N]>::value, int> = 0>
void to_json(BasicJsonType& j, T (&arr)[N])
{
    external_constructor<value_t::array>::construct(j, arr);
}

I am afraid that right now there is not much that we can do. The library uses UTF-8 internally, and to process wide strings, we would need to translate UTF-16 or UTF-32 to UTF-8. While we have some code for this in the input adapters (allowing to parse from wide strings), it is currently not planned to also have a conversion between different string types.

Well... Of course, pull requests would be more than welcome!

[jossef]

thanks @nlohmann for the comments.

I'm using a workaround for now (until this will be supported in this library)

string wstringToString(const wstring& input) {
	stringstream ss;

	for (auto item : input)
	{
		if (item <= 127)
		{
			ss << (char)item;
		}
		else
		{
			ss << "\\u" << std::hex << std::setfill('0') << std::setw(4) << (int)item;
		}
	}

	return ss.str();
}

// ....

json result;
result["name"] = wstringToString(L"אינעל דינק");
std::cout << result.dump(4) << std::endl;

• source

->

{
    "name": "\\u05d0\\u05d9\\u05e0\\u05e2\\u05dc \\u05d3\\u05d9\\u05e0\\u05e7"
}

[jaredgrubb]

The simple "this library only supports UTF8" is a good policy, and adding support to this library for std::wstring doesn't feel right. It may be better to try to disable the array overload so that it fails to compile if you try to use a std::wstring with this library.

[nlohmann]

What is wrong with also supporting UTF-16 and UTF-32 (even though we only store UTF-8 internally)?

[jaredgrubb]

On the face of it, supporting this would be a good thing.

My biggest worry (but not opposition) is that this will distract this library from focusing on the core idea of doing JSON well by having to deal with bugs/issues that come about from supporting multiple encodings properly. For example:

• should json<wstring> work?
• or is the goal just allowing json<string> = wstring to work? Would that mean wstring = json<string> should also work?
• Converting string<->wstring by itself is not obvious. For example, an official C++11 way of doing it got deprecated in C++17, without replacement, because the "general feeling that [the STL's approach] is not the best way to address unicode transcoding".
• What are the portability issues? For example, Linux has a 32-bit wstring, but Windows has a 16-bit wstring (note there is no implicit encoding, just explicit bit-width). The library would have to take a position that wstring must hold UTF-32 (-16 on Windows) just as it says string must hold UTF-8, but will this open up the library to even more issue reports around encoding handling than it already gets with the very simple "just UTF-8" rule?

I'm a very-biased adherent to "UTF-8 Everywhere", so take my advice with that grain of salt. If people ever suggest to me anything but string+utf8, I say don't do that. But I understand others do not share that view. Handling encoding issues sounds like the job of a dedicated library for that specific purpose and I don't think it's worth nlohmann::json's time to tackle it -- but maybe it's far easier than I'm painting it out to be! I'm not opposed, just skeptical.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[igoloe]

But supporting unicode sequences from char16_t and char32_t cannot be misinterpreted. wstring is just a wchar_t container with no associated encoding.

[nlohmann]

The library currently treats std::wstring with 2-byte characters as UTF-16 and those with 4-byte characters as UTF-32.

[nlohmann]

I added an FAQ entry: https://json.nlohmann.me/home/faq/#wide-string-handling