byte[] de/serialization behaviour change

[petrna]

Library

System.Text.Json

Summary

Some JSON containing null, where value maps to type byte[] - initially serialized by 4.7.2 (or Newtonsoft) - can't be deserialized with versions after 4.7.2 without a custom converter as exceptions are thrown.

Description

Behaviour discrepancy after upgrading from 4.7.2.
The discrepancy is present as early as 5.0.0-preview.1.20120.5. Same result with latest 6.0.0-preview.2.21154.6

4.7.2 Serialization behaviour:

        [Fact]
        public void Serialize_ByteArray_ShouldBeNull()
        {
            var obj = (byte[]) null;
            var result = JsonSerializer.Serialize(obj);
            Assert.Equal("null", result);
        }
        
        [Fact]
        public void Serialize_ByteArrayProperty_ShouldBeNull()
        {
            byte[] data = null;
            var objc = new {MyData = data};
            var result = JsonSerializer.Serialize(objc);
            Assert.Equal("{\"MyData\":null}", result);
        }

        [Fact] // Fails after 4.7.2
        public void Serialize_ByteArrayOfArray_ShouldBeArrayOfNull()
        {
            var obj = new[] {(byte[]) null};
            var result = JsonSerializer.Serialize(obj);
            Assert.Equal("[null]", result);
        }

        [Fact] // Fails after 4.7.2
        public void Serialize_ByteArrayDictValue_ShouldBeNull()
        {
            var obj = new Dictionary<string, byte[]>()
            {
                {"test", (byte[])null}
            };
            var result = JsonSerializer.Serialize(obj);
            Assert.Equal("{\"test\":null}", result);
        }

Any 5.x and 6.x package since then converts null of type byte[] to "" instead of null within a collection.
4.7.2 has identical behaviour in this regard to Newtonsoft.Json (12.0.3)

Deserialization is also no longer working, as the following code now throws exception:

        [Fact]
        public void Deserialize_ByteArrayDictValue_ShouldBeNull()
        {
            var result = JsonSerializer.Deserialize<Dictionary<string, byte[]>>("{\"test\":null}");
            Assert.Equal(null, result["test"]);
        }

Exception thrown:
System.InvalidOperationException
Cannot get the value of a token type 'Null' as a string.
   at System.Text.Json.Utf8JsonReader.TryGetBytesFromBase64(Byte[]& value)
   at System.Text.Json.Utf8JsonReader.GetBytesFromBase64()
   at System.Text.Json.Serialization.Converters.ByteArrayConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Library

System.Text.Json

Summary

Some JSON containing null, where value maps to type byte[] - initially serialized by 4.7.2 (or Newtonsoft) - can't be deserialized with versions after 4.7.2 without a custom converter as exceptions are thrown.

Description

Behaviour discrepancy after upgrading from 4.7.2.
The discrepancy is present as early as 5.0.0-preview.1.20120.5. Same result with latest 6.0.0-preview.2.21154.6

4.7.2 Serialization behaviour:

        [Fact]
        public void Serialize_ByteArray_ShouldBeNull()
        {
            var obj = (byte[]) null;
            var result = JsonSerializer.Serialize(obj);
            Assert.Equal("null", result);
        }

        [Fact] // Fails after 4.7.2
        public void Serialize_ByteArrayOfArray_ShouldBeArrayOfNull()
        {
            var obj = new[] {(byte[]) null};
            var result = JsonSerializer.Serialize(obj);
            Assert.Equal("[null]", result);
        }
        
        [Fact] // Fails after 4.7.2
        public void Serialize_ByteArrayProperty_ShouldBeNull()
        {
            byte[] data = null;
            var objc = new {MyData = data};
            var result = JsonSerializer.Serialize(objc);
            Assert.Equal("{\"MyData\":null}", result);
        }

        [Fact]
        public void Serialize_ByteArrayDictValue_ShouldBeNull()
        {
            var obj = new Dictionary<string, byte[]>()
            {
                {"test", (byte[])null}
            };
            var result = JsonSerializer.Serialize(obj);
            Assert.Equal("{\"test\":null}", result);
        }

Any 5.x and 6.x package since then converts null of type byte[] to "" instead of null within a collection.
4.7.2 has identical behaviour in this regard to Newtonsoft.Json (12.0.3)

Deserialization is also no longer working, as the following code now throws exception:

        [Fact]
        public void Deserialize_ByteArrayDictValue_ShouldBeNull()
        {
            var result = JsonSerializer.Deserialize<Dictionary<string, byte[]>>("{\"test\":null}");
            Assert.Equal(null, result["test"]);
        }

Exception thrown:
System.InvalidOperationException
Cannot get the value of a token type 'Null' as a string.
   at System.Text.Json.Utf8JsonReader.TryGetBytesFromBase64(Byte[]& value)
   at System.Text.Json.Utf8JsonReader.GetBytesFromBase64()
   at System.Text.Json.Serialization.Converters.ByteArrayConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

Author:	petrna
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

The behaviour change was made intentionally in dotnet/corefx#38106 to support serializing byte arrays as base64 encoded strings.

@steveharter do you think we should update the code so that null is tolerated on deserialization?

[eiriktsarpalis]

Related to #30456.

It looks the null deserialization failure is a bug specific to Dictionary deserialization:

JsonSerializer.Deserialize<Dictionary<string, byte[]>>("{\"test\":null}");

As far as making byte[] serialization more compatible with how Newtonsoft works, the recommended approach would be to write your own custom JsonConverter that does this.

[petrna]

I see you asked Steve for his input, but if it was up to me at least backward compatibility for deserializing would be the minimum solution here. Under backward compatibility I mean versions of System.Text.Json until 4.7.2 and any Newtonsoft.Json.

As you said, I can always write a JsonConverter for serializing into null instead of default empty string - which is exactly how I noticed the deserialization issue -, but that is kind of pointless if other solutions not controlled by me can't read it out of the box.

Thanks for the updates.

[eiriktsarpalis]

I see you asked Steve for his input, but if it was up to me at least backward compatibility for deserializing would be the minimum solution here. Under backward compatibility I mean versions of System.Text.Json until 4.7.2 and any Newtonsoft.Json.

This has been the behaviour of System.Text.Json since serialization support was released in .NET Core 3. Note that System.Text.Json is a different library than Newtonsoft.Json, compatibility never was a goal and in some cases divergences are intentional.

[eiriktsarpalis]

Something I had overlooked when first triaging this issue is that we are also inconsistently serializing null byte arrays depending on whether they are POCO properties/root-level values or collection elements (null in the former case, empty string in the latter). This happens because collection converters skip null checks for elements when elementConverter.CanUseDirectReadOrWriter is set to true, which in turn is enabled because ByteArrayConverter is defined as a "value" converter. This is the same root cause as what is causing the null value deserialization errors in collection types.

My conclusion then is that serializing null byte array items in collections as empty string is accidental rather than intentional (there are no tests validating this behavior) and should be fixed to be consistent with how POCOs are being handled.