From 85259e9b4e04a506aa9bfd1646ff4e2dc274ab25 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Wed, 27 Nov 2019 12:01:39 -0800 Subject: [PATCH 01/11] Add specification for ReferenceHandling --- .../docs/ReferenceHandling_spec.md | 768 ++++++++++++++++++ 1 file changed, 768 insertions(+) create mode 100644 src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md new file mode 100644 index 00000000000000..45d4038a92a916 --- /dev/null +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -0,0 +1,768 @@ +# Terminology + +**Reference loops**: also refered as circular references, occur when a property of the object refers to the object itself, directly (a -> a) or indirectly (a -> b -> a). They also occur when the element of an array refers to the array itself (arr[0] -> arr). Multiple ocurrences of the same reference does not imply circularity. + +**Preserve duplicated references**: Semantically represent objects and/or arrays whose have been previously written, with a reference to them in subsequent founds. + +**Metadata**: Extra properties on JSON objects and/or arrays (that may change thir schema) to enable reference preservation when round-tripping, those properties are only meant to be understand by the `JsonSerializer`. + +# Motivation + +Currently there is no mechanism to prevent infinite looping in circular objects (while serializing) nor to preserve references that round-trip when using System.Text.Json. + +This is a heavily requested feature since it is consider by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though the JSON specification does not support reference loops by default. Therefore, this will be implemented as an opt-in feature (for both serialization and deserialization). + +The current solution to deal with reference loops is to rely in MaxDepth and throw a JsonException after it is exceeded. Now, this is a decent and cheap solution but we will also offer other not-so-cheap options to deal with this problem while keeping the current one in order to not affect the out-of-the-box performance. + +# Other languages + +## Newtonsoft.Json +Newtonsoft.Json contains settings that you can enable to deal with such problems. +* For Serialization: + * [`ReferenceLoopHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm) + * [`PreserveReferencesHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm) +* For Deserialization: + * [`MetadataPropertyHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_MetadataPropertyHandling.htm) + +When using `ReferenceLoopHandling.Ignore`, other objects that were already seen on the current graph branch will be ignored on serialization. + +When using `PreserveReferencesHandling.All` you are signaling that your resulting JSON will contain *metadata* properties `$ref`, `$id` and `$values` which are going to act as reference identifiers (`$id`) and pointers (`$ref`). +Now, to read back those references, you have to use `MetadataPropertyHandling.Default` to indicate that *metadata* is expected in the payload passed to the `Deserialize` method. + +* Pros + * If we opt-in for this we could provide compatibility with Newtonsoft which is always desired by the community. +* Cons + * Quite invasive, (it affects `JsonException.Path`, `JsonSerializerOptions.IgnoreNullValues`, `JsonPropertyNameAttribute`, and Converters). + * This would break existing converters i.e: an array converter may expect the first token to be "[" and a preserved array starts with "{". + * perhaps converters are more feasible with the JSON path impl. + * We will now accept that an array comes in valid format when starts with a curly brace "{"; below issue is related to guard against NRE when this happens: + * https://github.com/dotnet/corefx/issues/41839 + +## dojo toolkit (JavaScript framework) +https://dojotoolkit.org/reference-guide/1.10/dojox/json/ref.html + +Similar: https://www.npmjs.com/package/json-cyclic + +* id-based (ignore this approach since is the same the one of Newtonsoft.Json) +* path-based + * "\#" denotes the root of the object and then uses semantics inspired in JSONPath. + * It does not uses `$id` nor `$values` metadata, therefore, everything can be referenced. + * Pros + * It looks cleaner. + * Only disruptive (weird) edge case would be a reference to an array i.e: { "MyArray": { "$ref": "#manager.subordinates" } }. + * Cons + * Path value will become too long on very deep objects. + * Storing all the complex types could become very expensive, are we going to store also primitive types? + * This would break existing converters when handling reference to an array. + * Not compatible with Newtonsoft.Json. + +## flatted (JavaScript module) (probably not worth it) +https://github.com/WebReflection/flatted + +* While stringifying, all Objects, including Arrays, and strings, are flattened out and replaced as unique index. +* Once parsed, all indexes will be replaced through the flattened collection. +* It has 23M downloads per month. +* Every single value (primitive and complex) is preserved. +* Cons: + * It does not look like JSON anymore. + + +## Jackson (Java) +https://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion + +* Let you annotate your class with @JsonIdentityInfo where you can define a class property that will be used to further represent the object. + +## golang + +* (https://go-review.googlesource.com/c/go/+/187920/), The fix is about detecting circular references after a threshold of 1,000 and throw when found in order to prevent a non-recoverable Stack Overflow. + +# Proposal + +```cs +namespace System.Text.Json.Serialization +{ + /// + /// This class defines the various ways the can deal with references on Serialization and Deserialization. + /// + public sealed class ReferenceHandling + { + public static ReferenceHandling Default { get; } + // TODO: decide if we keep or remove this option. + public static ReferenceHandling Ignore { get; } + public static ReferenceHandling Preserve { get; } + } +} + +namespace System.Text.Json +{ + public partial class JsonSerializerOptions + { + public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default; + } +} +``` +See also the [internal implementation details](https://gist.github.com/Jozkee/b0922ef609f7a942f00ac2c93a976ff1). + +## In depth +* **Default**: + * **On Serialize**: Throw a JsonException when MaxDepth is exceeded, this may occur by either a Reference Loop or by passing a very deep object. This option will not affect the performance of the serializer. + * **On Deserialize**: No effect. + +* **Ignore**: + * **On Serialize**: Ignores (skips writing) the property/element where the reference loop is detected. + * **On Deserialize**: No effect. + +* **Preserve**: + * **On Serialize**: When writing complex types, the serializer also writes them metadata ($id, $values and $ref) properties in order re-use them by writing a reference to the object or array. + * **On Deserialize**: While the other options show no effect on Deserialization, `Preserve` does affect its behavior with the following: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it. + +For System.Text.Json, the goal is to stick to the same *metadata* semantics for preserve from Newtonsoft.Json and provide a similar usage in `JsonSerializerOptions` that encompasses the needed options (i.e. provide reference preservation). + +This API is exposing the `ReferenceHandling` property as a class, to be extensible in the future; and provide built-in static instances of `Default` and `Preserve` that are useful to enable the most common behaviors by just setting those in `JsonSerializerOptions.ReferenceHandling`. + +With `ReferenceHandling` being a class, we can exclude things that, as of now, we are not sure are required and add them later based on customer feedback. For example, the `Object` and `Array` granularity of `Newtonsoft.Json's` `PreserveReferencesHandling` or the `ReferenceLoopHandling.Ignore` option. + +## Future + +Things that may build on top based on customer feedback: + +* (De)Serialize can define its own Preserve References Handling behavior each (i.e: you could opt-out form Preserve Reference on serialization but opt-in for read them on deserialization). + +* Expose a `ReferenceResolver` to override the logic that preserves references (Create your own implementation of a reference resolver). + +* Expose the `ReferenceResolver` in Converters to have access to the map of references. + +* Create `JsonReferenceHandlingAttribute` to enable to annotate properties and classes with their own isolated ReferenceHandling behavior (I cut-off the support for this due the constant checking for attributes was causing too much perf overhead on the main path, maybe we can try moving the attribute check to the warm-up method to reduce the runtime increase). +```cs +// Example of a class annotated with JsonReferenceHandling attributes. +[JsonReferenceHandling(ReferenceHandling.Preserve)] +public class Employee { + string Name { get; set; } + + [JsonReferenceHandling(ReferenceHandling.Ignore)] + Employee Manager { get; set; } + + List Subordinates { get; set; } +} +``` + +## Compatibility + +The next table show the combination of Newtonsoft's **ReferenceLoopHandling** and **PreserveReferencesHandling** and how to get its equivalent on System.Text.Json's *ReferenceHandling*: + +| RLH\PRH | None | All | Objects | Arrays | +|--------------:|----------:|-----------------:|-----------------:|-----------------:| +| **Error** | *Default* | future (overlap) | future (overlap) | future (overlap) | +| **Ignore** | *Ignore* | future (overlap) | future (overlap) | future (overlap) | +| **Serialize** | future | *Preserve* | future | future | + +Notes: +* Newtonsoft's `MetadataPropertyHandling.ReadAhead` will not be supported in this first effort. +* `Objects` and `Arrays` granularity may apply to both, Serialization and Deserialization. +* (overlap) means that Preserve References will co-exist with Reference Loop Handling; see [example](#using-a-custom-referencehandling-to-show-possible-future-usage). + + +# Examples + +Having the following class: +```cs +class Employee +{ + string Name { get; set; } + Employee Manager { get; set; } + List Subordinates { get; set; } +} +``` + +## Using Ignore on Serialize +```cs +private Employee bob = new Employee { Name = "Bob" }; +private Employee angela = new Employee { Name = "Angela" }; + +angela.Manager = bob; +bob.Subordinates = new List{ angela }; +``` + +On System.Text.Json: +```cs +public static void WriteIgnoringReferenceLoops() +{ + var options = new JsonSerializerOptions + { + ReferenceHandling = ReferenceHandling.Ignore + WriteIndented = true, + }; + + string json = JsonSerializer.Serialize(angela, options); + Console.Write(json); +} +``` + +On Newtonsoft.Json: +```cs +public static void WriteIgnoringReferenceLoops() +{ + var settings = new JsonSerializerSettings + { + ReferenceLoopHandling = ReferenceLoopHandling.Ignore + Formatting = Formatting.Indented + }; + + string json = JsonConvert.SerializeObject(angela, settings); + Console.Write(json); +} +``` + +Output: +```jsonc +{ + "Name": "Angela", + "Manager": { + "Name": "Bob", + // Note how subordinates is empty due Angela is being ignored. + "Subordinates": [] + } +} +``` + +## Using Preserve on Serialize +```cs +private Employee bob = new Employee { Name = "Bob" }; +private Employee angela = new Employee { Name = "Angela" }; + +angela.Manager = bob; +bob.Subordinates = new List{ angela }; +``` + +On System.Text.Json: +```cs +public static void WritePreservingReference() +{ + var options = new JsonSerializerOptions + { + ReferenceHandling = ReferenceHandling.Preserve + WriteIndented = true, + }; + + string json = JsonSerializer.Serialize(angela, options); + Console.Write(json); +} +``` + +On Newtonsoft.Json: +```cs +public static void WritePreservingReference() +{ + var settings = new JsonSerializerSettings + { + PreserveReferencesHandling = PreserveReferencesHandling.All + Formatting = Formatting.Indented + }; + + string json = JsonConvert.SerializeObject(angela, settings); + Console.Write(json); +} +``` + +Output: +```jsonc +{ + "$id": "1", + "Name": "Angela", + "Manager": { + "$id": "2", + "Name": "Bob", + "Subordinates": { + // Note how the Subordinates' square braces are replaced with curly braces + // in order to include $id and $values properties, + // $values will now hold whatever value was meant for the Subordinates list. + "$id": "3", + "$values": [ + { // Note how this object denotes reference to Angela that was previously serialized. + "$ref": "1" + } + ] + } + } +} +``` + +## Using Preserve on Deserialize +```cs +private const string json = + @"{ + ""$id"": ""1"", + ""Name"": ""Angela"", + ""Manager"": { + ""$id"": ""2"", + ""Name"": ""Bob"", + ""Subordinates"": { + ""$id"": ""3"", + ""$values"": [ + { + ""$ref"": ""1"" + } + ] + } + } + }"; +``` +On System.Text.Json: +```cs +public static void ReadJsonWithPreservedReferences(){ + var options = new JsonSerializerOptions + { + ReferenceHandling = ReferenceHandling.Preserve + }; + + Employee angela = JsonSerializer.Deserialize(json, options); + Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true. +} +``` + +On Newtonsoft.Json: +```cs +public static void ReadJsonWithPreservedReferences(){ + var options = new JsonSerializerSettings + { + //Newtonsoft.Json reads metadata by default, just setting the option for ilustrative purposes. + MetadataPropertyHanding = MetadataPropertyHandling.Default + }; + + Employee angela = JsonConvert.DeserializeObject(json, settings); + Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true. +} +``` + +## Using a custom `ReferenceHandling` (to show possible future usage). +```cs +public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() +{ + var bob = new Employee { Name = "Bob" }; + var angela = new Employee { Name = "Angela" }; + + angela.Manager = bob; + bob.Subordinates = new List{ angela }; + + var allEmployees = new List + { + angela, + bob + }; + + var options = new JsonSerializerOptions + { + ReferenceHandling = new ReferenceHandling( + PreserveReferencesHandling.All, // Preserve References Handling on serialization. + PreserveReferencesHandling.All, // Preserve References Handling on deserialization. + ReferenceLoopHandling.Ignore) // Reference Loop Handling on serialization. + WriteIndented = true, + }; + + string json = JsonSerializer.Serialize(allEmployees, options); + Console.Write(json); + + /* Output: + [ + { + "$id": "1", + "Name": "Angela", + "Manager": { + "$id": "2", + "Name": "Bob", + "Subordinates": { + "$id": "3", + // Note how subordinates is empty due Angela is being ignored. + "$values": [] + } + } + }, + { + // Note how element 2 is written as a reference + // since was previously seen in allEmployees[0].Manager + "$ref": "2" + } + ] + */ + + allEmployees = JsonSerializer.Deserialize>(json, options); + Console.WriteLine(allEmployees[0].Manager == allEmployees[1]); + /* Output: true */ +} +``` + +# Ground rules + +As a rule of thumb, we throw on all cases where the JSON payload being read contains any metadata that is impossible to create with the `JsonSerializer` (i.e. it was hand modified). However, this conflicts with feature parity in Newtonsoft.Json; those scenarios are described below. + +## Reference objects ($ref) + +* Regular property **before** `$ref`. + * **Newtonsoft.Json**: `$ref` is ignored if a regular property is previously found in the object. + * **S.T.Json**: Throw - Reference objects cannot contain other properties. + +```json +{ + "$id": "1", + "Name": "Angela", + "ReportsTo": { + "Name": "Bob", + "$ref": "1" + } +} +``` + +* Regular property **after** `$ref`. + * **Newtonsoft.Json**: Throw - Additional content found in JSON reference object. + * **S.T.Json**: Throw - Reference objects cannot contain other properties. + +```json +{ + "$id": "1", + "Name": "Angela", + "ReportsTo":{ + "$ref": "1", + "Name": "Angela" + } +} +``` + +* Metadata property **before** `$ref`: + * **Newtonsoft.Json**: `$id` is disregarded and the reference is set. + * **S.T.Json**: Throw - Reference objects cannot contain other properties. +```json +{ + "$id": "1", + "Name": "Angela", + "ReportsTo": { + "$id": "2", + "$ref": "1" + } +} +``` + +* Metadata property **after** `$ref`: + * **Newtonsoft.Json**: Throw with the next message: 'Additional content found in JSON reference object'. + * **S.T.Json**: Throw - Reference objects cannot contain other properties. +```json +{ + "$id": "1", + "Name": "Angela", + "ReportsTo": { + "$ref": "1", + "$id": "2" + } +} +``` + +* Reference object is before preserved object (or preserved object was never spotted): + * **Newtonsoft.Json**: Reference object evaluates as `null`. + * **S.T.Json**: Reference object evaluates as `null`. +```json +[ + { + "$ref": "1" + }, + { + "$id": "1", + "Name": "Angela" + } +] +``` + +## Preserved objects ($id) + +* Having more than one `$id` in the same object: + * **Newtonsoft.Json**: last one wins, in the example, the reference object evaluates to `null` (if `$ref` would be `"2"`, it would evaluate to itself). + * **S.T.Json**: Throw - Object already defines a reference identifier. +```json +{ + "$id": "1", + "$id": "2", + "Name": "Angela", + "ReportsTo": { + "$ref": "1" + } +} +``` + +* `$id` is not the first property: + * **Newtonsoft.Json**: Object is not preserved and cannot be referenced, therefore any reference to it would evaluate as null. + + * **S.T.Json**: Throw - Object $id is not the first property. + Note: In case we would want to switch, we can handle the `$id` not being the first property since we store the reference at the moment we spot the `$id` property, we throw to honor the rule of thumb. +```json +{ + "Name": "Angela", + "$id": "1", + "ReportsTo": { + "$ref": "1" + } +} +``` + +* `$id` is duplicated (not necessarily nested): + * **Newtonsoft.Json**: Throws - Error reading object reference '1'- Inner Exception: ArgumentException: A different value already has the Id '1'. + * **S.T.Json**: Throws - Duplicated id found while preserving reference. +```json +[ + { + "$id": "1", + "Name": "Angela" + }, + { + "$id": "1", + "Name": "Bob" + } +] +``` + +## Preserved arrays +A regular array is `[ elem1, elem2 ]`. +A preserved array is written in the next format `{ "$id": "1", "$values": [ elem1, elem2 ] }` + +* Preserved array does not contain any metadata: + * **Newtonsoft.Json**: Throws - Cannot deserialize the current JSON object into type 'System.Collections.Generic.List`1 + * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. + + ```json + {} + ``` + +* Preserved array only contains $id: + * **Newtonsoft.Json**: Throws - Cannot deserialize the current JSON object into type 'System.Collections.Generic.List`1 + * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. + + ```json + { + "$id": "1" + } + ``` + +* Preserved array only contains `$values`: + * **Newtonsoft.Json**: Does not throw and the payload evaluates to the array in the property. + * **S.T.Json**: Throw - Preserved arrays cannot lack an identifier. + + ```json + { + "$values": [] + } + ``` + +* Preserved array $values property contains null + * **Newtonsoft.Json**: Throw - Unexpected token while deserializing object: EndObject. Path ''. + * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. + + ```json + { + "$id": "1", + "$values": null + } + ``` + +* Preserved array $values property contains value + * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. + * **S.T.Json**: Throw - The JSON value could not be converted to TArray. Path: $.$values + + ```json + { + "$id": "1", + "$values": 1 + } + ``` + +* Preserved array $values property contains object + * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. + * **S.T.Json**: Throw - The property is already part of a preserved array object, cannot be read as a preserved array. + + ```json + { + "$id": "1", + "$values": {} + } + ``` + +## JSON Objects if not Collection (Class | Struct | Dictionary) - On Deserialize (and Serialize?) + +* `$ref` **Valid** under conditions: + * must be the only property in the object. + +* `$id` **Valid** under conditions: + * must be the first property in the object + +* `$values` **Not Valid** + +* `$.*` **Valid** + +Note: For Dictionary keys on serialize, should we allow serializing keys `$id`, `$ref` and `$values`? if we allow it, then there is a potential round-tripping issue. +Sample of similar issue with `DictionaryKeyPolicy`: +```cs +public static void TestDictionary_Collision() +{ + var root = new Dictionary(); + root["helloWorld"] = 100; + root["HelloWorld"] = 200; + + var opts = new JsonSerializerOptions + { + DictionaryKeyPolicy = JsonNamingPolicy.CamelCase + }; + + string json = JsonSerializer.Serialize(root, opts); + Console.WriteLine(json); + /* Output: + {"helloWorld":100,"helloWorld":200} */ + + // Round tripping issue + root = JsonSerializer.Deserialize>(json); +} +``` + + +## JSON Object if Collection - On Deserialize + +* `$ref` **Valid** under conditions: + * must be the only property in the object. + +* `$id` **Valid** under conditions: + * must be the first property in the object. + +* `$values` **Valid** under conditions: + * must be after `$id` + +* `$.*` **Not Valid** + + +## Immutable types +Since these types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes, nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. + +With that said, the deserializer will throw when it reads `$id` on any of these types. + +* **Immutable types**: i.e: `ImmutableList` and `ImmutableDictionary` +* **System.Array** + +## Value types + +* **Serialization**: +The serializer emits an `$id` for every JSON complex type, that means that if you have a custom struct, the serializer will append an id to it, however, there will never be a reference to these ids, since by default it uses `ReferenceEquals` when checking for references. + +```cs +public static void SerializeStructs() +{ + EmployeeStruct angela = new EmployeeStruct + { + Name = "Angela" + }; + + List employees = new List + { + angela, + angela + }; + + var options = new JsonSerializerOptions + { + ReferenceHandling = ReferenceHandling.Preserve + }; + + string json = JsonSerializer.Serialize(employees, options); + Console.WriteLine(json); +} + +Output: +```json +{ + "$id": "1", + "$values": [ + { + "$id": "2", + "Name": "Angela" + }, + { + "$id": "3", + "Name": "Angela" + } + ] +} +``` + +* **Deserialization**: +The deserializer will throw when it reads `$ref` within a property that matches to a value type (such as a struct) and `ReferenceHandling.Preserve` is set. + +Example: +```cs +public static void DeserializeStructs() +{ + string json = @" + { + ""$id"": ""1"", + ""$values"": [ + { + ""$id"": ""2"", + ""Name"": ""Angela"" + }, + { + ""$ref"": ""2"" + } + ] + }"; + + var options = new JsonSerializerOptions + { + ReferenceHandling = ReferenceHandling.Preserve + }; + + List root = JsonSerializer.Deserialize>(json, options); + // Throws JsonException. +} +``` + +In other words, having a `$ref` property in a struct, is never emitted by the serializer and read such thing (by manually changing the JSON payload) is not supported by the deserializer. + +## Interaction with JsonPropertyNameAttribute +Having the following class: + +```cs +private class EmployeeAnnotated +{ + [JsonPropertyName("$id")] + public string Identifier { get; set; } + [JsonPropertyName("$ref")] + public string Reference { get; set; } + [JsonPropertyName("$values")] + public List Values { get; set; } + + public string Name { get; set; } +} +``` + +Either on Serialization or Deserialization: + +```cs +public static void DeSerializeWithPreserve() +{ + var root = new EmployeeAnnotated(); + var opts = new JsonSerializerOptions + { + ReferenceHandling = ReferenceHandling.Preserve + }; + + //Throw JsonException - PropertyName cannot start with '$' when Preserve References is enabled. + string json = JsonSerializer.Serialize(root, opts); + + //Also throws the same exception. + EmployeeAnnotated obj = JsonSerializer.Deserialize(json, opts); +} +``` + + + + +# Notes + +1. MaxDepth validation will not be affected by `ReferenceHandling.Preserve`. +2. We are merging the Newtonsoft.Json types [`ReferenceLoopHandling`]("https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm"), [`MetadataPropertyHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_MetadataPropertyHandling.htm) (without `ReadAhead`), and [`PreserveReferencesHandling`]("https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm") (without the granularity of `Objects` and `Arrays`) into one single class; `ReferenceHandling`. +3. While Immutable types and `System.Array`s can be Serialized with Preserve semantics, they will not be supported when trying to Deserialize them as a reference; those types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes, nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. +4. Value types, such as structs that contain preserve semantics, will not be supported when Deserialized as well; this is because the serializer will never signal a reference object to those types, doing such thing implies boxing of value types. +5. Additional features, such as Converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. +6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`, this option will not ship if said evidence is not found. \ No newline at end of file From f5f85c1b88e6a994a0ab0591ac30459e0e0563d7 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Wed, 27 Nov 2019 18:43:41 -0800 Subject: [PATCH 02/11] Address more feedback, corrected types, and included table of contents --- .../docs/ReferenceHandling_spec.md | 348 ++++++++++-------- 1 file changed, 195 insertions(+), 153 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 45d4038a92a916..abf9d2ee024339 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -1,10 +1,41 @@ +# Table of Contents + +- [Terminology](#terminology) +- [Motivation](#motivation) +- [Proposal](#proposal) + - [In depth](#in-depth) + - [Compatibility](#compatibility) +- [Examples](#examples) + - [Using Default on Serialize](#using-default-on-serialize) + - [Using Ignore on Serialize](#using-ignore-on-serialize) + - [Using Preserve on Serialize](#using-preserve-on-serialize) + - [Using Preserve on Deserialize](#using-preserve-on-deserialize) +- [Other languages](#other-languages) + - [Newtonsoft.Json](#newtonsoftjson) + - [dojo toolkit (JavaScript framework)](#dojo-toolkit-javascript-framework) + - [flatted (JavaScript module) (probably not worth it)](#flatted-javascript-module-probably-not-worth-it) + - [Jackson (Java)](#jackson-java) + - [golang](#golang) +- [Ground rules](#ground-rules) + - [Reference objects ($ref)](#reference-objects-ref) + - [Preserved objects ($id)](#preserved-objects-id) + - [Preserved arrays](#preserved-arrays) + - [JSON Objects if not Collection (Class | Struct | Dictionary) - On Deserialize (and Serialize?)](#json-objects-if-not-collection-class--struct--dictionary---on-deserialize-and-serialize) + - [JSON Object if Collection - On Deserialize](#json-object-if-collection---on-deserialize) + - [Immutable types](#immutable-types) + - [Value types](#value-types) + - [Interaction with JsonPropertyNameAttribute](#interaction-with-jsonpropertynameattribute) +- [Future](#future) +- [Notes](#notes) + + # Terminology -**Reference loops**: also refered as circular references, occur when a property of the object refers to the object itself, directly (a -> a) or indirectly (a -> b -> a). They also occur when the element of an array refers to the array itself (arr[0] -> arr). Multiple ocurrences of the same reference does not imply circularity. +**Reference loops**: also referred as circular references, occur when a property of the object refers to the object itself, directly (a -> a) or indirectly (a -> b -> a). They also occur when the element of an array refers to the array itself (arr[0] -> arr). Multiple occurrences of the same reference does not imply circularity. **Preserve duplicated references**: Semantically represent objects and/or arrays whose have been previously written, with a reference to them in subsequent founds. -**Metadata**: Extra properties on JSON objects and/or arrays (that may change thir schema) to enable reference preservation when round-tripping, those properties are only meant to be understand by the `JsonSerializer`. +**Metadata**: Extra properties on JSON objects and/or arrays (that may change their schema) to enable reference preservation when round-tripping, those properties are only meant to be understand by the `JsonSerializer`. # Motivation @@ -14,68 +45,6 @@ This is a heavily requested feature since it is consider by many as a very commo The current solution to deal with reference loops is to rely in MaxDepth and throw a JsonException after it is exceeded. Now, this is a decent and cheap solution but we will also offer other not-so-cheap options to deal with this problem while keeping the current one in order to not affect the out-of-the-box performance. -# Other languages - -## Newtonsoft.Json -Newtonsoft.Json contains settings that you can enable to deal with such problems. -* For Serialization: - * [`ReferenceLoopHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm) - * [`PreserveReferencesHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm) -* For Deserialization: - * [`MetadataPropertyHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_MetadataPropertyHandling.htm) - -When using `ReferenceLoopHandling.Ignore`, other objects that were already seen on the current graph branch will be ignored on serialization. - -When using `PreserveReferencesHandling.All` you are signaling that your resulting JSON will contain *metadata* properties `$ref`, `$id` and `$values` which are going to act as reference identifiers (`$id`) and pointers (`$ref`). -Now, to read back those references, you have to use `MetadataPropertyHandling.Default` to indicate that *metadata* is expected in the payload passed to the `Deserialize` method. - -* Pros - * If we opt-in for this we could provide compatibility with Newtonsoft which is always desired by the community. -* Cons - * Quite invasive, (it affects `JsonException.Path`, `JsonSerializerOptions.IgnoreNullValues`, `JsonPropertyNameAttribute`, and Converters). - * This would break existing converters i.e: an array converter may expect the first token to be "[" and a preserved array starts with "{". - * perhaps converters are more feasible with the JSON path impl. - * We will now accept that an array comes in valid format when starts with a curly brace "{"; below issue is related to guard against NRE when this happens: - * https://github.com/dotnet/corefx/issues/41839 - -## dojo toolkit (JavaScript framework) -https://dojotoolkit.org/reference-guide/1.10/dojox/json/ref.html - -Similar: https://www.npmjs.com/package/json-cyclic - -* id-based (ignore this approach since is the same the one of Newtonsoft.Json) -* path-based - * "\#" denotes the root of the object and then uses semantics inspired in JSONPath. - * It does not uses `$id` nor `$values` metadata, therefore, everything can be referenced. - * Pros - * It looks cleaner. - * Only disruptive (weird) edge case would be a reference to an array i.e: { "MyArray": { "$ref": "#manager.subordinates" } }. - * Cons - * Path value will become too long on very deep objects. - * Storing all the complex types could become very expensive, are we going to store also primitive types? - * This would break existing converters when handling reference to an array. - * Not compatible with Newtonsoft.Json. - -## flatted (JavaScript module) (probably not worth it) -https://github.com/WebReflection/flatted - -* While stringifying, all Objects, including Arrays, and strings, are flattened out and replaced as unique index. -* Once parsed, all indexes will be replaced through the flattened collection. -* It has 23M downloads per month. -* Every single value (primitive and complex) is preserved. -* Cons: - * It does not look like JSON anymore. - - -## Jackson (Java) -https://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion - -* Let you annotate your class with @JsonIdentityInfo where you can define a class property that will be used to further represent the object. - -## golang - -* (https://go-review.googlesource.com/c/go/+/187920/), The fix is about detecting circular references after a threshold of 1,000 and throw when found in order to prevent a non-recoverable Stack Overflow. - # Proposal ```cs @@ -87,9 +56,9 @@ namespace System.Text.Json.Serialization public sealed class ReferenceHandling { public static ReferenceHandling Default { get; } + public static ReferenceHandling Preserve { get; } // TODO: decide if we keep or remove this option. public static ReferenceHandling Ignore { get; } - public static ReferenceHandling Preserve { get; } } } @@ -105,47 +74,23 @@ See also the [internal implementation details](https://gist.github.com/Jozkee/b0 ## In depth * **Default**: - * **On Serialize**: Throw a JsonException when MaxDepth is exceeded, this may occur by either a Reference Loop or by passing a very deep object. This option will not affect the performance of the serializer. - * **On Deserialize**: No effect. - -* **Ignore**: - * **On Serialize**: Ignores (skips writing) the property/element where the reference loop is detected. - * **On Deserialize**: No effect. + * **On Serialize**: Throw a `JsonException` when `MaxDepth` is exceeded, this may occur by either a reference loop or by passing a very deep object. This option will not affect the performance of the serializer. + * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` dictionary. * **Preserve**: * **On Serialize**: When writing complex types, the serializer also writes them metadata ($id, $values and $ref) properties in order re-use them by writing a reference to the object or array. * **On Deserialize**: While the other options show no effect on Deserialization, `Preserve` does affect its behavior with the following: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it. +* **Ignore**: + * **On Serialize**: Ignores (skips writing) the property/element where the reference loop is detected. + * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` dictionary. + For System.Text.Json, the goal is to stick to the same *metadata* semantics for preserve from Newtonsoft.Json and provide a similar usage in `JsonSerializerOptions` that encompasses the needed options (i.e. provide reference preservation). This API is exposing the `ReferenceHandling` property as a class, to be extensible in the future; and provide built-in static instances of `Default` and `Preserve` that are useful to enable the most common behaviors by just setting those in `JsonSerializerOptions.ReferenceHandling`. With `ReferenceHandling` being a class, we can exclude things that, as of now, we are not sure are required and add them later based on customer feedback. For example, the `Object` and `Array` granularity of `Newtonsoft.Json's` `PreserveReferencesHandling` or the `ReferenceLoopHandling.Ignore` option. -## Future - -Things that may build on top based on customer feedback: - -* (De)Serialize can define its own Preserve References Handling behavior each (i.e: you could opt-out form Preserve Reference on serialization but opt-in for read them on deserialization). - -* Expose a `ReferenceResolver` to override the logic that preserves references (Create your own implementation of a reference resolver). - -* Expose the `ReferenceResolver` in Converters to have access to the map of references. - -* Create `JsonReferenceHandlingAttribute` to enable to annotate properties and classes with their own isolated ReferenceHandling behavior (I cut-off the support for this due the constant checking for attributes was causing too much perf overhead on the main path, maybe we can try moving the attribute check to the warm-up method to reduce the runtime increase). -```cs -// Example of a class annotated with JsonReferenceHandling attributes. -[JsonReferenceHandling(ReferenceHandling.Preserve)] -public class Employee { - string Name { get; set; } - - [JsonReferenceHandling(ReferenceHandling.Ignore)] - Employee Manager { get; set; } - - List Subordinates { get; set; } -} -``` - ## Compatibility The next table show the combination of Newtonsoft's **ReferenceLoopHandling** and **PreserveReferencesHandling** and how to get its equivalent on System.Text.Json's *ReferenceHandling*: @@ -159,7 +104,7 @@ The next table show the combination of Newtonsoft's **ReferenceLoopHandling** an Notes: * Newtonsoft's `MetadataPropertyHandling.ReadAhead` will not be supported in this first effort. * `Objects` and `Arrays` granularity may apply to both, Serialization and Deserialization. -* (overlap) means that Preserve References will co-exist with Reference Loop Handling; see [example](#using-a-custom-referencehandling-to-show-possible-future-usage). +* (overlap) means that preserve references will co-exist with Reference Loop Handling and we will need to define how to resolve that (On Newtonsoft.Json, `PreserveReferencesHandling` takes precedence); see [example](#using-a-custom-referencehandling-to-show-possible-future-usage). # Examples @@ -174,6 +119,20 @@ class Employee } ``` +## Using Default on Serialize +```cs +private Employee bob = new Employee { Name = "Bob" }; +private Employee angela = new Employee { Name = "Angela" }; + +public static void WriteObject() +{ + string json = JsonSerializer.Serialize(angela, options); + // Throws JsonException - + // "A possible object cycle was detected which is not supported. + // This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 64." +} +``` + ## Using Ignore on Serialize ```cs private Employee bob = new Employee { Name = "Bob" }; @@ -334,62 +293,67 @@ public static void ReadJsonWithPreservedReferences(){ } ``` -## Using a custom `ReferenceHandling` (to show possible future usage). -```cs -public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() -{ - var bob = new Employee { Name = "Bob" }; - var angela = new Employee { Name = "Angela" }; +# Other languages - angela.Manager = bob; - bob.Subordinates = new List{ angela }; - - var allEmployees = new List - { - angela, - bob - }; +## Newtonsoft.Json +Newtonsoft.Json contains settings that you can enable to deal with such problems. +* For Serialization: + * [`ReferenceLoopHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm) + * [`PreserveReferencesHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm) +* For Deserialization: + * [`MetadataPropertyHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_MetadataPropertyHandling.htm) - var options = new JsonSerializerOptions - { - ReferenceHandling = new ReferenceHandling( - PreserveReferencesHandling.All, // Preserve References Handling on serialization. - PreserveReferencesHandling.All, // Preserve References Handling on deserialization. - ReferenceLoopHandling.Ignore) // Reference Loop Handling on serialization. - WriteIndented = true, - }; +When using `ReferenceLoopHandling.Ignore`, other objects that were already seen on the current graph branch will be ignored on serialization. - string json = JsonSerializer.Serialize(allEmployees, options); - Console.Write(json); +When using `PreserveReferencesHandling.All` you are signaling that your resulting JSON will contain *metadata* properties `$ref`, `$id` and `$values` which are going to act as reference identifiers (`$id`) and pointers (`$ref`). +Now, to read back those references, you have to use `MetadataPropertyHandling.Default` to indicate that *metadata* is expected in the payload passed to the `Deserialize` method. - /* Output: - [ - { - "$id": "1", - "Name": "Angela", - "Manager": { - "$id": "2", - "Name": "Bob", - "Subordinates": { - "$id": "3", - // Note how subordinates is empty due Angela is being ignored. - "$values": [] - } - } - }, - { - // Note how element 2 is written as a reference - // since was previously seen in allEmployees[0].Manager - "$ref": "2" - } - ] - */ +* Pros + * If we opt-in for this we could provide compatibility with Newtonsoft which is always desired by the community. +* Cons + * Quite invasive, (it affects `JsonException.Path`, `JsonSerializerOptions.IgnoreNullValues`, `JsonPropertyNameAttribute`, and Converters). + * This would break existing converters i.e: an array converter may expect the first token to be "[" and a preserved array starts with "{". + * perhaps converters are more feasible with the JSON path implementation. + * We will now accept that an array comes in valid format when starts with a curly brace "{"; below issue is related to guard against NRE when this happens: + * https://github.com/dotnet/corefx/issues/41839 - allEmployees = JsonSerializer.Deserialize>(json, options); - Console.WriteLine(allEmployees[0].Manager == allEmployees[1]); - /* Output: true */ -} -``` +## dojo toolkit (JavaScript framework) +https://dojotoolkit.org/reference-guide/1.10/dojox/json/ref.html + +Similar: https://www.npmjs.com/package/json-cyclic + +* id-based (ignore this approach since is the same the one of Newtonsoft.Json) +* path-based + * "\#" denotes the root of the object and then uses semantics inspired in JSONPath. + * It does not uses `$id` nor `$values` metadata, therefore, everything can be referenced. + * Pros + * It looks cleaner. + * Only disruptive (weird) edge case would be a reference to an array i.e: { "MyArray": { "$ref": "#manager.subordinates" } }. + * Cons + * Path value will become too long on very deep objects. + * Storing all the complex types could become very expensive, are we going to store also primitive types? + * This would break existing converters when handling reference to an array. + * Not compatible with Newtonsoft.Json. + +## flatted (JavaScript module) (probably not worth it) +https://github.com/WebReflection/flatted + +* While stringifying, all Objects, including Arrays, and strings, are flattened out and replaced as unique index. +* Once parsed, all indexes will be replaced through the flattened collection. +* It has 23M downloads per month. +* Every single value (primitive and complex) is preserved. +* Cons: + * It does not look like JSON anymore. + + +## Jackson (Java) +https://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion + +* Let you annotate your class with @JsonIdentityInfo where you can define a class property that will be used to further represent the object. + +## golang +* Circularity detection will start to occur after a fixed threshold of 1,000 depth. + * (https://go-review.googlesource.com/c/go/+/187920/), The fix is about detecting circular references after a threshold of 1,000 and throw when found in order to prevent a non-recoverable Stack Overflow. # Ground rules @@ -405,7 +369,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont { "$id": "1", "Name": "Angela", - "ReportsTo": { + "Manager": { "Name": "Bob", "$ref": "1" } @@ -420,7 +384,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont { "$id": "1", "Name": "Angela", - "ReportsTo":{ + "Manager":{ "$ref": "1", "Name": "Angela" } @@ -434,7 +398,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont { "$id": "1", "Name": "Angela", - "ReportsTo": { + "Manager": { "$id": "2", "$ref": "1" } @@ -448,7 +412,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont { "$id": "1", "Name": "Angela", - "ReportsTo": { + "Manager": { "$ref": "1", "$id": "2" } @@ -480,7 +444,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont "$id": "1", "$id": "2", "Name": "Angela", - "ReportsTo": { + "Manager": { "$ref": "1" } } @@ -495,7 +459,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont { "Name": "Angela", "$id": "1", - "ReportsTo": { + "Manager": { "$ref": "1" } } @@ -755,8 +719,86 @@ public static void DeSerializeWithPreserve() } ``` +# Future +Things that may build on top based on customer feedback: +* (De)Serialize can define its own Preserve References Handling behavior each (i.e: you could opt-out form Preserve Reference on serialization but opt-in for read them on deserialization). + +* Expose a `ReferenceResolver` to override the logic that preserves references (Create your own implementation of a reference resolver). +* Expose the `ReferenceResolver` in Converters to have access to the map of references. + +* Create `JsonReferenceHandlingAttribute` to enable to annotate properties and classes with their own isolated ReferenceHandling behavior (I cut-off the support for this due the constant checking for attributes was causing too much perf overhead on the main path, maybe we can try moving the attribute check to the warm-up method to reduce the runtime increase). +```cs +// Example of a class annotated with JsonReferenceHandling attributes. +[JsonReferenceHandling(ReferenceHandling.Preserve)] +public class Employee { + string Name { get; set; } + + [JsonReferenceHandling(ReferenceHandling.Ignore)] + Employee Manager { get; set; } + + List Subordinates { get; set; } +} +``` + +## Using a custom `ReferenceHandling` (to show possible future usage). +```cs +public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() +{ + var bob = new Employee { Name = "Bob" }; + var angela = new Employee { Name = "Angela" }; + + angela.Manager = bob; + bob.Subordinates = new List{ angela }; + + var allEmployees = new List + { + angela, + bob + }; + + var options = new JsonSerializerOptions + { + ReferenceHandling = new ReferenceHandling( + PreserveReferencesHandling.All, // Preserve References Handling on serialization. + PreserveReferencesHandling.All, // Preserve References Handling on deserialization. + ReferenceLoopHandling.Ignore) // Reference Loop Handling on serialization. + WriteIndented = true, + }; + + string json = JsonSerializer.Serialize(allEmployees, options); + Console.Write(json); + + /* Output: + [ + { + "$id": "1", + "Name": "Angela", + "Manager": { + "$id": "2", + "Name": "Bob", + "Subordinates": { + "$id": "3", + // Note how subordinates is empty due Angela is being ignored. + // Alternatively: we may let PreserveReferenceHandling take precedence and write the reference instead? + "$values": [] + } + } + }, + { + // Note how element 2 is written as a reference + // since was previously seen in allEmployees[0].Manager + "$ref": "2" + } + ] + */ + + allEmployees = JsonSerializer.Deserialize>(json, options); + Console.WriteLine(allEmployees[0].Manager == allEmployees[1]); + /* Output: true */ +} +``` # Notes From e12a8f6d0a0bcca83a9414b5849f2d4e3f3515a3 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Mon, 2 Dec 2019 20:41:40 -0800 Subject: [PATCH 03/11] Add missing access modifiers to class properties Add note about immutable types --- .../docs/ReferenceHandling_spec.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index abf9d2ee024339..427cad92328986 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -113,9 +113,9 @@ Having the following class: ```cs class Employee { - string Name { get; set; } - Employee Manager { get; set; } - List Subordinates { get; set; } + public string Name { get; set; } + public Employee Manager { get; set; } + public List Subordinates { get; set; } } ``` @@ -598,9 +598,9 @@ public static void TestDictionary_Collision() ## Immutable types -Since these types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes, nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. +Since these types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes; nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. -With that said, the deserializer will throw when it reads `$id` on any of these types. +With that said, the deserializer will throw when it reads `$id` on any of these types; but regardless of that, when writing those types, they are going to be preserved as any other collection type (`{ "$id": "1", "$values": [...] }`) since those types can still being parsed into a collection type that it is supported. * **Immutable types**: i.e: `ImmutableList` and `ImmutableDictionary` * **System.Array** @@ -608,7 +608,7 @@ With that said, the deserializer will throw when it reads `$id` on any of these ## Value types * **Serialization**: -The serializer emits an `$id` for every JSON complex type, that means that if you have a custom struct, the serializer will append an id to it, however, there will never be a reference to these ids, since by default it uses `ReferenceEquals` when checking for references. +The serializer emits an `$id` for every JSON complex type, that means that if you have a custom struct (which is value type), the serializer will append an `$id` to it, however, there will never be a reference to those `$id`s, since by default it uses `ReferenceEquals` when comparing the objects. ```cs public static void SerializeStructs() @@ -733,12 +733,12 @@ Things that may build on top based on customer feedback: // Example of a class annotated with JsonReferenceHandling attributes. [JsonReferenceHandling(ReferenceHandling.Preserve)] public class Employee { - string Name { get; set; } + public string Name { get; set; } [JsonReferenceHandling(ReferenceHandling.Ignore)] - Employee Manager { get; set; } + public Employee Manager { get; set; } - List Subordinates { get; set; } + public List Subordinates { get; set; } } ``` From 24c6d1716e85996ac824826ee2f301b141b277a6 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Mon, 2 Dec 2019 20:57:27 -0800 Subject: [PATCH 04/11] Apply suggestions from code review Co-Authored-By: Ahson Khan --- .../docs/ReferenceHandling_spec.md | 56 ++++++++++--------- 1 file changed, 29 insertions(+), 27 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 427cad92328986..7e86bd18816394 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -31,19 +31,20 @@ # Terminology -**Reference loops**: also referred as circular references, occur when a property of the object refers to the object itself, directly (a -> a) or indirectly (a -> b -> a). They also occur when the element of an array refers to the array itself (arr[0] -> arr). Multiple occurrences of the same reference does not imply circularity. +**Reference loops**: Also referred as circular references, loops occur when a property of a .NET object refers to the object itself, either directly (a -> a) or indirectly (a -> b -> a). They also occur when the element of an array refers to the array itself (arr[0] -> arr). Multiple occurrences of the same reference do not imply a cycle. -**Preserve duplicated references**: Semantically represent objects and/or arrays whose have been previously written, with a reference to them in subsequent founds. +**Preserve duplicated references**: Semantically represent objects and/or arrays that have been previously written, with a reference to them when found again in the object graph (using reference equality for comparison). -**Metadata**: Extra properties on JSON objects and/or arrays (that may change their schema) to enable reference preservation when round-tripping, those properties are only meant to be understand by the `JsonSerializer`. +**Metadata**: Extra properties on JSON objects and/or arrays (that may change their schema) to enable reference preservation when round-tripping. These additional properties are only meant to be understood by the `JsonSerializer`. # Motivation -Currently there is no mechanism to prevent infinite looping in circular objects (while serializing) nor to preserve references that round-trip when using System.Text.Json. +Currently, there is no mechanism to avoid infinite loops while serializing .NET object instances that contain cycles nor to preserve references that round-trip when using `System.Text.Json`. The `JsonSerializer` throws a `JsonException` when a loop is found within the object graph. -This is a heavily requested feature since it is consider by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though the JSON specification does not support reference loops by default. Therefore, this will be implemented as an opt-in feature (for both serialization and deserialization). +This is a heavily requested feature since it is considered by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though the JSON specification does not support reference loops by default. Therefore, this will be implemented as an opt-in feature (for both serialization and deserialization). + +The current solution to deal with cycles in the object graph while serializing is to rely on `MaxDepth` and throw a `JsonException` after it is exceeded. This was done to avoid perf overhead for cycle detection in the common case. The goal is to enable the new opt-in feature with minimal impact to existing performance. -The current solution to deal with reference loops is to rely in MaxDepth and throw a JsonException after it is exceeded. Now, this is a decent and cheap solution but we will also offer other not-so-cheap options to deal with this problem while keeping the current one in order to not affect the out-of-the-box performance. # Proposal @@ -51,7 +52,8 @@ The current solution to deal with reference loops is to rely in MaxDepth and thr namespace System.Text.Json.Serialization { /// - /// This class defines the various ways the can deal with references on Serialization and Deserialization. + /// This class defines the various ways the + /// can deal with references on Serialization and Deserialization. /// public sealed class ReferenceHandling { @@ -74,26 +76,26 @@ See also the [internal implementation details](https://gist.github.com/Jozkee/b0 ## In depth * **Default**: - * **On Serialize**: Throw a `JsonException` when `MaxDepth` is exceeded, this may occur by either a reference loop or by passing a very deep object. This option will not affect the performance of the serializer. - * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` dictionary. + * **On Serialize**: Throw a `JsonException` when `MaxDepth` is exceeded. This may occur by either a reference loop or by passing a very deep object. This option will not affect the performance of the serializer. + * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` overflow dictionary. * **Preserve**: - * **On Serialize**: When writing complex types, the serializer also writes them metadata ($id, $values and $ref) properties in order re-use them by writing a reference to the object or array. - * **On Deserialize**: While the other options show no effect on Deserialization, `Preserve` does affect its behavior with the following: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it. + * **On Serialize**: When writing complex types (i.e. POCOs/non-primitive types), the serializer also writes the metadata ($id, $values and $ref) properties in order to reference them later by writing a reference to the previously written JSON object or array. + * **On Deserialize**: While the other options have no effect on deserialization, `Preserve` does affect its behavior, as follows: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it. * **Ignore**: * **On Serialize**: Ignores (skips writing) the property/element where the reference loop is detected. * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` dictionary. -For System.Text.Json, the goal is to stick to the same *metadata* semantics for preserve from Newtonsoft.Json and provide a similar usage in `JsonSerializerOptions` that encompasses the needed options (i.e. provide reference preservation). +For `System.Text.Json`, the goal is to stick to the same *metadata* syntax used when preserving references using `Newtonsoft.Json` and provide a similar usage in `JsonSerializerOptions` that encompasses the needed options (i.e. provide reference preservation). This way, JSON output produced by `Newtonsoft.Json` can be deserialized by `System.Text.Json` and vice versa. This API is exposing the `ReferenceHandling` property as a class, to be extensible in the future; and provide built-in static instances of `Default` and `Preserve` that are useful to enable the most common behaviors by just setting those in `JsonSerializerOptions.ReferenceHandling`. -With `ReferenceHandling` being a class, we can exclude things that, as of now, we are not sure are required and add them later based on customer feedback. For example, the `Object` and `Array` granularity of `Newtonsoft.Json's` `PreserveReferencesHandling` or the `ReferenceLoopHandling.Ignore` option. +With `ReferenceHandling` being a class, we can exclude things that, as of now, we are not sure are required and add them later based on customer feedback. For example, the `Object` and `Array` granularity of `Newtonsoft.Json's` `PreserveReferencesHandling` feature or the `ReferenceLoopHandling.Ignore` option. ## Compatibility -The next table show the combination of Newtonsoft's **ReferenceLoopHandling** and **PreserveReferencesHandling** and how to get its equivalent on System.Text.Json's *ReferenceHandling*: +The next table show the combination of Newtonsoft's **ReferenceLoopHandling** (RLH) and **PreserveReferencesHandling** (PRH) and how to get its equivalent on System.Text.Json's *ReferenceHandling*: | RLH\PRH | None | All | Objects | Arrays | |--------------:|----------:|-----------------:|-----------------:|-----------------:| @@ -102,14 +104,14 @@ The next table show the combination of Newtonsoft's **ReferenceLoopHandling** an | **Serialize** | future | *Preserve* | future | future | Notes: -* Newtonsoft's `MetadataPropertyHandling.ReadAhead` will not be supported in this first effort. -* `Objects` and `Arrays` granularity may apply to both, Serialization and Deserialization. -* (overlap) means that preserve references will co-exist with Reference Loop Handling and we will need to define how to resolve that (On Newtonsoft.Json, `PreserveReferencesHandling` takes precedence); see [example](#using-a-custom-referencehandling-to-show-possible-future-usage). +* We are deferring adding support for Newtonsoft's `MetadataPropertyHandling.ReadAhead` for now. +* `Objects` and `Arrays` granularity may apply to both, serialization and deserialization. +* (overlap) means that preserve references co-exists along with reference loop handling and we will need to define how to resolve that (On `Newtonsoft.Json`, `PreserveReferencesHandling` takes precedence); see [example](#using-a-custom-referencehandling-to-show-possible-future-usage). # Examples -Having the following class: +Let's assume you have the following class: ```cs class Employee { @@ -142,7 +144,7 @@ angela.Manager = bob; bob.Subordinates = new List{ angela }; ``` -On System.Text.Json: +On `System.Text.Json`: ```cs public static void WriteIgnoringReferenceLoops() { @@ -157,7 +159,7 @@ public static void WriteIgnoringReferenceLoops() } ``` -On Newtonsoft.Json: +On `Newtonsoft.Json`: ```cs public static void WriteIgnoringReferenceLoops() { @@ -178,7 +180,7 @@ Output: "Name": "Angela", "Manager": { "Name": "Bob", - // Note how subordinates is empty due Angela is being ignored. + // Note how subordinates is empty because Angela is being ignored. "Subordinates": [] } } @@ -193,7 +195,7 @@ angela.Manager = bob; bob.Subordinates = new List{ angela }; ``` -On System.Text.Json: +On `System.Text.Json`: ```cs public static void WritePreservingReference() { @@ -208,7 +210,7 @@ public static void WritePreservingReference() } ``` -On Newtonsoft.Json: +On `Newtonsoft.Json`: ```cs public static void WritePreservingReference() { @@ -266,7 +268,7 @@ private const string json = } }"; ``` -On System.Text.Json: +On `System.Text.Json`: ```cs public static void ReadJsonWithPreservedReferences(){ var options = new JsonSerializerOptions @@ -279,12 +281,12 @@ public static void ReadJsonWithPreservedReferences(){ } ``` -On Newtonsoft.Json: +On `Newtonsoft.Json`: ```cs public static void ReadJsonWithPreservedReferences(){ var options = new JsonSerializerSettings { - //Newtonsoft.Json reads metadata by default, just setting the option for ilustrative purposes. + //Newtonsoft.Json reads metadata by default, just setting the option for illustrative purposes. MetadataPropertyHanding = MetadataPropertyHandling.Default }; @@ -807,4 +809,4 @@ public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() 3. While Immutable types and `System.Array`s can be Serialized with Preserve semantics, they will not be supported when trying to Deserialize them as a reference; those types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes, nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. 4. Value types, such as structs that contain preserve semantics, will not be supported when Deserialized as well; this is because the serializer will never signal a reference object to those types, doing such thing implies boxing of value types. 5. Additional features, such as Converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. -6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`, this option will not ship if said evidence is not found. \ No newline at end of file +6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`, this option will not ship if said evidence is not found. From 6b4a91b301dc35654656a6936a468b1e1d52c550 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Mon, 2 Dec 2019 21:26:29 -0800 Subject: [PATCH 05/11] Apply more suggestions from code review Co-Authored-By: Ahson Khan --- .../docs/ReferenceHandling_spec.md | 64 +++++++++---------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 7e86bd18816394..db8a25d9cc72f0 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -41,7 +41,7 @@ Currently, there is no mechanism to avoid infinite loops while serializing .NET object instances that contain cycles nor to preserve references that round-trip when using `System.Text.Json`. The `JsonSerializer` throws a `JsonException` when a loop is found within the object graph. -This is a heavily requested feature since it is considered by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though the JSON specification does not support reference loops by default. Therefore, this will be implemented as an opt-in feature (for both serialization and deserialization). +This is a heavily requested feature since it is considered by many as a very common scenario, especially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though the JSON specification does not support reference loops by default. Therefore, this will be implemented as an opt-in feature (for both serialization and deserialization). The current solution to deal with cycles in the object graph while serializing is to rely on `MaxDepth` and throw a `JsonException` after it is exceeded. This was done to avoid perf overhead for cycle detection in the common case. The goal is to enable the new opt-in feature with minimal impact to existing performance. @@ -80,7 +80,7 @@ See also the [internal implementation details](https://gist.github.com/Jozkee/b0 * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` overflow dictionary. * **Preserve**: - * **On Serialize**: When writing complex types (i.e. POCOs/non-primitive types), the serializer also writes the metadata ($id, $values and $ref) properties in order to reference them later by writing a reference to the previously written JSON object or array. + * **On Serialize**: When writing complex types (i.e. POCOs/non-primitive types), the serializer also writes the metadata (`$id`, `$values` and `$ref`) properties in order to reference them later by writing a reference to the previously written JSON object or array. * **On Deserialize**: While the other options have no effect on deserialization, `Preserve` does affect its behavior, as follows: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it. * **Ignore**: @@ -298,7 +298,7 @@ public static void ReadJsonWithPreservedReferences(){ # Other languages ## Newtonsoft.Json -Newtonsoft.Json contains settings that you can enable to deal with such problems. +`Newtonsoft.Json` contains settings that you can enable to deal with such problems. * For Serialization: * [`ReferenceLoopHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm) * [`PreserveReferencesHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm) @@ -314,7 +314,7 @@ Now, to read back those references, you have to use `MetadataPropertyHandling.De * If we opt-in for this we could provide compatibility with Newtonsoft which is always desired by the community. * Cons * Quite invasive, (it affects `JsonException.Path`, `JsonSerializerOptions.IgnoreNullValues`, `JsonPropertyNameAttribute`, and Converters). - * This would break existing converters i.e: an array converter may expect the first token to be "[" and a preserved array starts with "{". + * This could break existing converters. For example, an array converter may expect the first token to be "[" but a preserved array starts with "{". * perhaps converters are more feasible with the JSON path implementation. * We will now accept that an array comes in valid format when starts with a curly brace "{"; below issue is related to guard against NRE when this happens: * https://github.com/dotnet/corefx/issues/41839 @@ -324,9 +324,9 @@ https://dojotoolkit.org/reference-guide/1.10/dojox/json/ref.html Similar: https://www.npmjs.com/package/json-cyclic -* id-based (ignore this approach since is the same the one of Newtonsoft.Json) +* id-based (ignore this approach since it is the same as `Newtonsoft.Json`) * path-based - * "\#" denotes the root of the object and then uses semantics inspired in JSONPath. + * "\#" denotes the root of the object and then uses semantics inspired by JSONPath. * It does not uses `$id` nor `$values` metadata, therefore, everything can be referenced. * Pros * It looks cleaner. @@ -335,12 +335,12 @@ Similar: https://www.npmjs.com/package/json-cyclic * Path value will become too long on very deep objects. * Storing all the complex types could become very expensive, are we going to store also primitive types? * This would break existing converters when handling reference to an array. - * Not compatible with Newtonsoft.Json. + * Not compatible with `Newtonsoft.Json`. ## flatted (JavaScript module) (probably not worth it) https://github.com/WebReflection/flatted -* While stringifying, all Objects, including Arrays, and strings, are flattened out and replaced as unique index. +* While stringifying, all Objects, including Arrays and strings, are flattened out and replaced with unique index. * Once parsed, all indexes will be replaced through the flattened collection. * It has 23M downloads per month. * Every single value (primitive and complex) is preserved. @@ -351,15 +351,15 @@ https://github.com/WebReflection/flatted ## Jackson (Java) https://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion -* Let you annotate your class with @JsonIdentityInfo where you can define a class property that will be used to further represent the object. +* Let you annotate your class with `@JsonIdentityInfo` where you can define a class property that will be used to further represent the object. ## golang * Circularity detection will start to occur after a fixed threshold of 1,000 depth. - * (https://go-review.googlesource.com/c/go/+/187920/), The fix is about detecting circular references after a threshold of 1,000 and throw when found in order to prevent a non-recoverable Stack Overflow. + * [This fix](https://go-review.googlesource.com/c/go/+/187920/) is about detecting circular references after a threshold of 1,000 and throw when found in order to prevent a non-recoverable stack overflow. # Ground rules -As a rule of thumb, we throw on all cases where the JSON payload being read contains any metadata that is impossible to create with the `JsonSerializer` (i.e. it was hand modified). However, this conflicts with feature parity in Newtonsoft.Json; those scenarios are described below. +As a rule of thumb, we throw on all cases where the JSON payload being read contains any metadata that is impossible to create with the `JsonSerializer` (i.e. it was hand modified). Since `System.Text.Json` is more strict, it means that certain payloads that `Newtonsoft.Json` could process, will fail with `System.Text.Json`. Specific example scenarios where that could happen are described below. ## Reference objects ($ref) @@ -394,7 +394,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont ``` * Metadata property **before** `$ref`: - * **Newtonsoft.Json**: `$id` is disregarded and the reference is set. + * **Newtonsoft.Json**: `$id` is disregarded, and the reference is set. * **S.T.Json**: Throw - Reference objects cannot contain other properties. ```json { @@ -515,7 +515,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem } ``` -* Preserved array $values property contains null +* Preserved array `$values` property is null * **Newtonsoft.Json**: Throw - Unexpected token while deserializing object: EndObject. Path ''. * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. @@ -526,7 +526,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem } ``` -* Preserved array $values property contains value +* Preserved array `$values` property is a primitive value * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. * **S.T.Json**: Throw - The JSON value could not be converted to TArray. Path: $.$values @@ -537,7 +537,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem } ``` -* Preserved array $values property contains object +* Preserved array `$values` property contains object * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. * **S.T.Json**: Throw - The property is already part of a preserved array object, cannot be read as a preserved array. @@ -560,7 +560,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem * `$.*` **Valid** -Note: For Dictionary keys on serialize, should we allow serializing keys `$id`, `$ref` and `$values`? if we allow it, then there is a potential round-tripping issue. +Note: For Dictionary keys on serialize, should we allow serializing keys `$id`, `$ref` and `$values`? If we allow it, then there is a potential round-tripping issue. Sample of similar issue with `DictionaryKeyPolicy`: ```cs public static void TestDictionary_Collision() @@ -610,7 +610,7 @@ With that said, the deserializer will throw when it reads `$id` on any of these ## Value types * **Serialization**: -The serializer emits an `$id` for every JSON complex type, that means that if you have a custom struct (which is value type), the serializer will append an `$id` to it, however, there will never be a reference to those `$id`s, since by default it uses `ReferenceEquals` when comparing the objects. +The serializer emits an `$id` for every JSON complex type. That means that if you have a custom struct (which is value type), the serializer will append an `$id` to the JSON when serializing it. However, there will never be a reference to those `$ids`, since by default it uses `ReferenceEquals` when comparing the objects. ```cs public static void SerializeStructs() @@ -683,10 +683,10 @@ public static void DeserializeStructs() } ``` -In other words, having a `$ref` property in a struct, is never emitted by the serializer and read such thing (by manually changing the JSON payload) is not supported by the deserializer. +In other words, having a `$ref` property in a struct, is never emitted by the serializer and reading such a payload (for instance, if the payload was hand-crafted) is not supported by the deserializer. ## Interaction with JsonPropertyNameAttribute -Having the following class: +Let's say you have the following class: ```cs private class EmployeeAnnotated @@ -702,7 +702,7 @@ private class EmployeeAnnotated } ``` -Either on Serialization or Deserialization: +Both on serialization and deserialization: ```cs public static void DeSerializeWithPreserve() @@ -722,15 +722,15 @@ public static void DeSerializeWithPreserve() ``` # Future -Things that may build on top based on customer feedback: +Things that we may want to consider building on top based on customer feedback: -* (De)Serialize can define its own Preserve References Handling behavior each (i.e: you could opt-out form Preserve Reference on serialization but opt-in for read them on deserialization). +* (De)Serialize can define its own, independently configurable reference handling behavior (for example, you could opt-out from preserve reference on serialization but opt-in for reading them on deserialization). -* Expose a `ReferenceResolver` to override the logic that preserves references (Create your own implementation of a reference resolver). +* Expose a `ReferenceResolver` to override the logic that preserves references (the caller creates their own implementation of a reference resolver). -* Expose the `ReferenceResolver` in Converters to have access to the map of references. +* Expose the `ReferenceResolver` in `Converters` to have access to the map of references. -* Create `JsonReferenceHandlingAttribute` to enable to annotate properties and classes with their own isolated ReferenceHandling behavior (I cut-off the support for this due the constant checking for attributes was causing too much perf overhead on the main path, maybe we can try moving the attribute check to the warm-up method to reduce the runtime increase). +* Create `JsonReferenceHandlingAttribute` to enable annotating properties and classes with their own isolated `ReferenceHandling` behavior (I am deferring this feature because the constant checking for attributes was causing too much perf overhead on the main path, but maybe we can try moving the attribute check to the warm-up method to reduce the runtime increase). ```cs // Example of a class annotated with JsonReferenceHandling attributes. [JsonReferenceHandling(ReferenceHandling.Preserve)] @@ -744,7 +744,7 @@ public class Employee { } ``` -## Using a custom `ReferenceHandling` (to show possible future usage). +## Using a custom `ReferenceHandling` (shows potential API evolution and usage). ```cs public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() { @@ -782,7 +782,7 @@ public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() "Name": "Bob", "Subordinates": { "$id": "3", - // Note how subordinates is empty due Angela is being ignored. + // Note how subordinates is empty because Angela is being ignored. // Alternatively: we may let PreserveReferenceHandling take precedence and write the reference instead? "$values": [] } @@ -804,9 +804,9 @@ public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() # Notes -1. MaxDepth validation will not be affected by `ReferenceHandling.Preserve`. -2. We are merging the Newtonsoft.Json types [`ReferenceLoopHandling`]("https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm"), [`MetadataPropertyHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_MetadataPropertyHandling.htm) (without `ReadAhead`), and [`PreserveReferencesHandling`]("https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm") (without the granularity of `Objects` and `Arrays`) into one single class; `ReferenceHandling`. -3. While Immutable types and `System.Array`s can be Serialized with Preserve semantics, they will not be supported when trying to Deserialize them as a reference; those types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes, nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. -4. Value types, such as structs that contain preserve semantics, will not be supported when Deserialized as well; this is because the serializer will never signal a reference object to those types, doing such thing implies boxing of value types. -5. Additional features, such as Converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. +1. `MaxDepth` validation will not be affected by `ReferenceHandling.Preserve`. +2. We are merging the `Newtonsoft.Json` types [`ReferenceLoopHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_ReferenceLoopHandling.htm), [`MetadataPropertyHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_MetadataPropertyHandling.htm) (without `ReadAhead`), and [`PreserveReferencesHandling`](https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_PreserveReferencesHandling.htm) (without the granularity of `Objects` and `Arrays`) into one single class; `ReferenceHandling`. +3. While immutable types and `System.Arrays` can be serialized with preserve semantics, they will not be supported when trying to deserialize them as a reference. Those types are created with the help of an internal converter and they are not parsed until the entire block of JSON finishes. Nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. +4. Value types, such as structs that contain preserve semantics, will not be supported when deserializing as well. This is because the serializer will never emit a reference object to those types and doing so implies boxing of value types. +5. Additional features, such as converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. 6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`, this option will not ship if said evidence is not found. From 095d5a9e1dd5101cdf12a30e5c46afb85ce6751e Mon Sep 17 00:00:00 2001 From: David Cantu Date: Mon, 2 Dec 2019 22:25:00 -0800 Subject: [PATCH 06/11] Add PR suggestions. --- .../docs/ReferenceHandling_spec.md | 62 ++++++++++++++++--- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index db8a25d9cc72f0..6c1f98b0ee7853 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -49,6 +49,14 @@ The current solution to deal with cycles in the object graph while serializing i # Proposal ```cs +namespace System.Text.Json +{ + public partial class JsonSerializerOptions + { + public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default; + } +} + namespace System.Text.Json.Serialization { /// @@ -63,14 +71,6 @@ namespace System.Text.Json.Serialization public static ReferenceHandling Ignore { get; } } } - -namespace System.Text.Json -{ - public partial class JsonSerializerOptions - { - public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default; - } -} ``` See also the [internal implementation details](https://gist.github.com/Jozkee/b0922ef609f7a942f00ac2c93a976ff1). @@ -111,7 +111,45 @@ Notes: # Examples -Let's assume you have the following class: +## Using Default on Deserialize +```cs +class Employee +{ + [JsonPropertyName("$id")] + public string Identifier { get; set; } + public Employee Manager { get; set; } + + [JsonExtensionData] + public IDictionary ExtensionData { get; set; } +} + +private const string json = + @"{ + ""$id"": ""1"", + ""Name"": ""Angela"", + ""Manager"": { + ""$id"": ""2"", + ""Name"": ""Bob"", + ""Manager"": { + ""$ref"": ""2"" + } + } + }"; +``` + +```cs +public static void ReadObject() +{ + Employee angela = JsonSerializer.Deserialize(json); + Console.WriteLine(angela.Identifier) //prints: "1". + Console.WriteLine(angela.Manager.Identifier) //prints: "2". + Console.WriteLine(angela.Manager.Manager.ExtensionData["$ref"]) //prints: "2". +} +``` + +Note how you can annotate .Net properties to use properties that are meant for metadata and are added to the `JsonExtensionData` overflow dictionary, in case there is any, when opting-out of the `ReferenceHanding.Preserve` feature. + +For the next samples let's assume you have the following class: ```cs class Employee { @@ -126,6 +164,9 @@ class Employee private Employee bob = new Employee { Name = "Bob" }; private Employee angela = new Employee { Name = "Angela" }; +angela.Manager = bob; +bob.Subordinates = new List{ angela }; + public static void WriteObject() { string json = JsonSerializer.Serialize(angela, options); @@ -604,6 +645,9 @@ Since these types are created with the help of an internal converter, and they a With that said, the deserializer will throw when it reads `$id` on any of these types; but regardless of that, when writing those types, they are going to be preserved as any other collection type (`{ "$id": "1", "$values": [...] }`) since those types can still being parsed into a collection type that it is supported. +Note: By the same principle, `Newtonsoft.Json` does not support parsing JSON arrays into immutables as well. +Note 2: When using immutable types and `ReferenceHandling.Preserve`, you will not be able to generate payloads that are capables of round-tripping. + * **Immutable types**: i.e: `ImmutableList` and `ImmutableDictionary` * **System.Array** From bc2701c9a4a70197967458478c75012f270eee5a Mon Sep 17 00:00:00 2001 From: David Cantu Date: Mon, 2 Dec 2019 22:28:09 -0800 Subject: [PATCH 07/11] Apply even more suggestions from code review Co-Authored-By: Ahson Khan --- .../System.Text.Json/docs/ReferenceHandling_spec.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 6c1f98b0ee7853..bf82c086d1334a 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -6,6 +6,7 @@ - [In depth](#in-depth) - [Compatibility](#compatibility) - [Examples](#examples) + - [Using Default on Deserialize](#using-default-on-deserialize) - [Using Default on Serialize](#using-default-on-serialize) - [Using Ignore on Serialize](#using-ignore-on-serialize) - [Using Preserve on Serialize](#using-preserve-on-serialize) @@ -381,7 +382,7 @@ Similar: https://www.npmjs.com/package/json-cyclic ## flatted (JavaScript module) (probably not worth it) https://github.com/WebReflection/flatted -* While stringifying, all Objects, including Arrays and strings, are flattened out and replaced with unique index. +* While stringifying, all Objects, including Arrays and strings, are flattened out and replaced with a unique index. * Once parsed, all indexes will be replaced through the flattened collection. * It has 23M downloads per month. * Every single value (primitive and complex) is preserved. @@ -643,13 +644,13 @@ public static void TestDictionary_Collision() ## Immutable types Since these types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes; nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. -With that said, the deserializer will throw when it reads `$id` on any of these types; but regardless of that, when writing those types, they are going to be preserved as any other collection type (`{ "$id": "1", "$values": [...] }`) since those types can still being parsed into a collection type that it is supported. +With that said, the deserializer will throw when it reads `$id` on any of these types. When serializing (i.e. writing) those types, however, they are going to be preserved as any other collection type (`{ "$id": "1", "$values": [...] }`) since those types can still be parsed into a collection type that is supported. Note: By the same principle, `Newtonsoft.Json` does not support parsing JSON arrays into immutables as well. Note 2: When using immutable types and `ReferenceHandling.Preserve`, you will not be able to generate payloads that are capables of round-tripping. * **Immutable types**: i.e: `ImmutableList` and `ImmutableDictionary` -* **System.Array** +* **System.Array**: i.e: `Array` and `T[]` ## Value types From cc1ac66d0a0df6e9139431a7fa42636050ce9a6d Mon Sep 17 00:00:00 2001 From: David Cantu Date: Mon, 2 Dec 2019 22:46:32 -0800 Subject: [PATCH 08/11] Add missing semi colons on samples and standardize error messages on $values Co-Authored-By: Ahson Khan --- .../docs/ReferenceHandling_spec.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index bf82c086d1334a..568fb13287bfb0 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -142,15 +142,15 @@ private const string json = public static void ReadObject() { Employee angela = JsonSerializer.Deserialize(json); - Console.WriteLine(angela.Identifier) //prints: "1". - Console.WriteLine(angela.Manager.Identifier) //prints: "2". - Console.WriteLine(angela.Manager.Manager.ExtensionData["$ref"]) //prints: "2". + Console.WriteLine(angela.Identifier); //prints: "1". + Console.WriteLine(angela.Manager.Identifier); //prints: "2". + Console.WriteLine(angela.Manager.Manager.ExtensionData["$ref"]); //prints: "2". } ``` Note how you can annotate .Net properties to use properties that are meant for metadata and are added to the `JsonExtensionData` overflow dictionary, in case there is any, when opting-out of the `ReferenceHanding.Preserve` feature. -For the next samples let's assume you have the following class: +For the next samples, let's assume you have the following class: ```cs class Employee { @@ -570,7 +570,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem * Preserved array `$values` property is a primitive value * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. - * **S.T.Json**: Throw - The JSON value could not be converted to TArray. Path: $.$values + * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. ```json { @@ -581,7 +581,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem * Preserved array `$values` property contains object * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. - * **S.T.Json**: Throw - The property is already part of a preserved array object, cannot be read as a preserved array. + * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. ```json { @@ -854,4 +854,4 @@ public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() 3. While immutable types and `System.Arrays` can be serialized with preserve semantics, they will not be supported when trying to deserialize them as a reference. Those types are created with the help of an internal converter and they are not parsed until the entire block of JSON finishes. Nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. 4. Value types, such as structs that contain preserve semantics, will not be supported when deserializing as well. This is because the serializer will never emit a reference object to those types and doing so implies boxing of value types. 5. Additional features, such as converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. -6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`, this option will not ship if said evidence is not found. +6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`. This option will not ship if said evidence is not found. From 95d263697e8e0d84f2c909187771b6b8b26b4a35 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Fri, 13 Dec 2019 15:46:25 -0800 Subject: [PATCH 09/11] Address changes discussed on API review --- .../docs/ReferenceHandling_spec.md | 67 +++++++++++++------ 1 file changed, 47 insertions(+), 20 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 568fb13287bfb0..8bf4b8349990be 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -465,7 +465,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont * Reference object is before preserved object (or preserved object was never spotted): * **Newtonsoft.Json**: Reference object evaluates as `null`. - * **S.T.Json**: Reference object evaluates as `null`. + * **S.T.Json**: Throw - Reference not found. ```json [ { @@ -482,7 +482,7 @@ As a rule of thumb, we throw on all cases where the JSON payload being read cont * Having more than one `$id` in the same object: * **Newtonsoft.Json**: last one wins, in the example, the reference object evaluates to `null` (if `$ref` would be `"2"`, it would evaluate to itself). - * **S.T.Json**: Throw - Object already defines a reference identifier. + * **S.T.Json**: Throw - $id must be the first property. ```json { "$id": "1", @@ -559,7 +559,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem * Preserved array `$values` property is null * **Newtonsoft.Json**: Throw - Unexpected token while deserializing object: EndObject. Path ''. - * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. + * **S.T.Json**: Throw - Invalid token after $values metadata property. ```json { @@ -570,7 +570,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem * Preserved array `$values` property is a primitive value * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. - * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. + * **S.T.Json**: Throw - Invalid token after $values metadata property. ```json { @@ -581,7 +581,7 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem * Preserved array `$values` property contains object * **Newtonsoft.Json**: Unexpected token while deserializing object: EndObject. Path ''. - * **S.T.Json**: Throw - Preserved array $values property was not present or its value is not an array. + * **S.T.Json**: Throw - Invalid token after $values metadata property. ```json { @@ -590,17 +590,33 @@ A preserved array is written in the next format `{ "$id": "1", "$values": [ elem } ``` -## JSON Objects if not Collection (Class | Struct | Dictionary) - On Deserialize (and Serialize?) +* Preserved array contains a property other than `$id` and `$values` + * **Newtonsoft.Json**: Ignores other properties. + * **S.T.Json**: Throw - Invalid property in preserved array. + + ```json + { + "$id": "1", + "$values": [1, 2, 3], + "TrailingProperty": "Hello world" + } + ``` + +## JSON Objects if not Enumerable (Class | Struct | Dictionary) - On Deserialize (and Serialize?) * `$ref` **Valid** under conditions: * must be the only property in the object. * `$id` **Valid** under conditions: - * must be the first property in the object + * must be the first property in the object. + +* `$values` **Not valid** -* `$values` **Not Valid** +* `$.*` **Not valid** -* `$.*` **Valid** +* `\u0024.*` **valid** + +* `\u0024id*` **valid** but not considered metadata. Note: For Dictionary keys on serialize, should we allow serializing keys `$id`, `$ref` and `$values`? If we allow it, then there is a potential round-tripping issue. Sample of similar issue with `DictionaryKeyPolicy`: @@ -626,8 +642,13 @@ public static void TestDictionary_Collision() } ``` +Resolution for above issue: +On serialization, when a JSON property name, that is either a dictionary key or a CLR class property, starts with a '$' character, we must write the escaped character "\u0024" instead. + +On deserialization, metadata will be digested by using only the raw bytes, so no encoded characters are allowed in metadata; to read JSON properties that start with a '$' you will need to pass it with the escaped '$' (\u0024) or turn the feature off. + -## JSON Object if Collection - On Deserialize +## JSON Object if Enumerable - On Deserialize * `$ref` **Valid** under conditions: * must be the only property in the object. @@ -638,7 +659,7 @@ public static void TestDictionary_Collision() * `$values` **Valid** under conditions: * must be after `$id` -* `$.*` **Not Valid** +* `.*` **Not Valid** any property other than above metadata will not be valid. ## Immutable types @@ -655,7 +676,7 @@ Note 2: When using immutable types and `ReferenceHandling.Preserve`, you will no ## Value types * **Serialization**: -The serializer emits an `$id` for every JSON complex type. That means that if you have a custom struct (which is value type), the serializer will append an `$id` to the JSON when serializing it. However, there will never be a reference to those `$ids`, since by default it uses `ReferenceEquals` when comparing the objects. +The serializer emits an `$id` for every JSON complex type. However, to reduce bandwidth, structs will not be written with metadata, since it would be meaningless due `ReferenceEquals` is used when comparing the objects and no backpointer reference would be ever written to an struct. ```cs public static void SerializeStructs() @@ -686,11 +707,9 @@ Output: "$id": "1", "$values": [ { - "$id": "2", "Name": "Angela" }, { - "$id": "3", "Name": "Angela" } ] @@ -728,7 +747,7 @@ public static void DeserializeStructs() } ``` -In other words, having a `$ref` property in a struct, is never emitted by the serializer and reading such a payload (for instance, if the payload was hand-crafted) is not supported by the deserializer. +In other words, having a `$ref` property in a struct, is never emitted by the serializer and reading such a payload (for instance, if the payload was hand-crafted) is not supported by the deserializer. However, since `Newtonsoft.Json` does emit `$id` for value-type objects `System.Text.Json` will allow reading struct objects that contain `$id`, regardless of not being able to create such payloads. ## Interaction with JsonPropertyNameAttribute Let's say you have the following class: @@ -758,14 +777,22 @@ public static void DeSerializeWithPreserve() ReferenceHandling = ReferenceHandling.Preserve }; - //Throw JsonException - PropertyName cannot start with '$' when Preserve References is enabled. + // The property will be emitted with the '$' encoded. string json = JsonSerializer.Serialize(root, opts); - - //Also throws the same exception. - EmployeeAnnotated obj = JsonSerializer.Deserialize(json, opts); + Console.WriteLine(json); } ``` +```json +{ + "\u0024id": null, + "\u0024ref": null, + "\u0024values": null +} +``` + +If the name of your property starts with '$', either by using `JsonPropertyNameAttribute`, by using F#, or by any other reason, that leading '$' (and that one only), will be replaced with its encoded equivalent `\u0024`. + # Future Things that we may want to consider building on top based on customer feedback: @@ -854,4 +881,4 @@ public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() 3. While immutable types and `System.Arrays` can be serialized with preserve semantics, they will not be supported when trying to deserialize them as a reference. Those types are created with the help of an internal converter and they are not parsed until the entire block of JSON finishes. Nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. 4. Value types, such as structs that contain preserve semantics, will not be supported when deserializing as well. This is because the serializer will never emit a reference object to those types and doing so implies boxing of value types. 5. Additional features, such as converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. -6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`. This option will not ship if said evidence is not found. +6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`. This option will not ship if said evidence is not found. \ No newline at end of file From 94021e1249c051a087e3abacf54fb0f94cce71f7 Mon Sep 17 00:00:00 2001 From: David Cantu Date: Tue, 14 Jan 2020 22:17:55 -0800 Subject: [PATCH 10/11] Add note to describe unsupported round-tripping capability on JsonExtensionData. --- src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 8bf4b8349990be..1ee1027381994a 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -881,4 +881,5 @@ public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences() 3. While immutable types and `System.Arrays` can be serialized with preserve semantics, they will not be supported when trying to deserialize them as a reference. Those types are created with the help of an internal converter and they are not parsed until the entire block of JSON finishes. Nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. 4. Value types, such as structs that contain preserve semantics, will not be supported when deserializing as well. This is because the serializer will never emit a reference object to those types and doing so implies boxing of value types. 5. Additional features, such as converter support, `ReferenceResolver`, `JsonPropertyAttribute.IsReference` and `JsonPropertyAttribute.ReferenceLoopHandling`, that build on top of `ReferenceLoopHandling` and `PreserveReferencesHandling` were considered but they can be added in the future based on customer requests. -6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`. This option will not ship if said evidence is not found. \ No newline at end of file +6. We are still looking for evidence that backs up supporting `ReferenceHandling.Ignore`. This option will not ship if said evidence is not found. +7. Round-tripping support for preserved references into the `JsonExtensionData` is currently not supported (we emit the metadata on serialization and we create a JsonElement on deserialization instead), while in `Newtonsoft.Json` they are supported. This may change in a future based on customer feedback. \ No newline at end of file From c23959daba1ccc0f27d74deecf486625304cde7e Mon Sep 17 00:00:00 2001 From: David Cantu Date: Wed, 15 Jan 2020 17:51:17 -0800 Subject: [PATCH 11/11] Replace i.e. for e.g. where meant to quote an example. --- .../docs/ReferenceHandling_spec.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md index 1ee1027381994a..9bfc8895f69b6e 100644 --- a/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md +++ b/src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md @@ -81,14 +81,14 @@ See also the [internal implementation details](https://gist.github.com/Jozkee/b0 * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` overflow dictionary. * **Preserve**: - * **On Serialize**: When writing complex types (i.e. POCOs/non-primitive types), the serializer also writes the metadata (`$id`, `$values` and `$ref`) properties in order to reference them later by writing a reference to the previously written JSON object or array. + * **On Serialize**: When writing complex types (e.g. POCOs/non-primitive types), the serializer also writes the metadata (`$id`, `$values` and `$ref`) properties in order to reference them later by writing a reference to the previously written JSON object or array. * **On Deserialize**: While the other options have no effect on deserialization, `Preserve` does affect its behavior, as follows: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it. * **Ignore**: * **On Serialize**: Ignores (skips writing) the property/element where the reference loop is detected. * **On Deserialize**: Metadata properties will not be consumed, therefore they will be treated as regular properties that can map to a real property using `JsonPropertyName` or be added to the `JsonExtensionData` dictionary. -For `System.Text.Json`, the goal is to stick to the same *metadata* syntax used when preserving references using `Newtonsoft.Json` and provide a similar usage in `JsonSerializerOptions` that encompasses the needed options (i.e. provide reference preservation). This way, JSON output produced by `Newtonsoft.Json` can be deserialized by `System.Text.Json` and vice versa. +For `System.Text.Json`, the goal is to stick to the same *metadata* syntax used when preserving references using `Newtonsoft.Json` and provide a similar usage in `JsonSerializerOptions` that encompasses the needed options (e.g. provide reference preservation). This way, JSON output produced by `Newtonsoft.Json` can be deserialized by `System.Text.Json` and vice versa. This API is exposing the `ReferenceHandling` property as a class, to be extensible in the future; and provide built-in static instances of `Default` and `Preserve` that are useful to enable the most common behaviors by just setting those in `JsonSerializerOptions.ReferenceHandling`. @@ -372,7 +372,7 @@ Similar: https://www.npmjs.com/package/json-cyclic * It does not uses `$id` nor `$values` metadata, therefore, everything can be referenced. * Pros * It looks cleaner. - * Only disruptive (weird) edge case would be a reference to an array i.e: { "MyArray": { "$ref": "#manager.subordinates" } }. + * Only disruptive (weird) edge case would be a reference to an array e.g: { "MyArray": { "$ref": "#manager.subordinates" } }. * Cons * Path value will become too long on very deep objects. * Storing all the complex types could become very expensive, are we going to store also primitive types? @@ -401,7 +401,7 @@ https://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recurs # Ground rules -As a rule of thumb, we throw on all cases where the JSON payload being read contains any metadata that is impossible to create with the `JsonSerializer` (i.e. it was hand modified). Since `System.Text.Json` is more strict, it means that certain payloads that `Newtonsoft.Json` could process, will fail with `System.Text.Json`. Specific example scenarios where that could happen are described below. +As a rule of thumb, we throw on all cases where the JSON payload being read contains any metadata that is impossible to create with the `JsonSerializer` (e.g. it was hand modified). Since `System.Text.Json` is more strict, it means that certain payloads that `Newtonsoft.Json` could process, will fail with `System.Text.Json`. Specific example scenarios where that could happen are described below. ## Reference objects ($ref) @@ -665,13 +665,13 @@ On deserialization, metadata will be digested by using only the raw bytes, so no ## Immutable types Since these types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes; nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive. -With that said, the deserializer will throw when it reads `$id` on any of these types. When serializing (i.e. writing) those types, however, they are going to be preserved as any other collection type (`{ "$id": "1", "$values": [...] }`) since those types can still be parsed into a collection type that is supported. +With that said, the deserializer will throw when it reads `$id` on any of these types. When serializing (e.g. writing) those types, however, they are going to be preserved as any other collection type (`{ "$id": "1", "$values": [...] }`) since those types can still be parsed into a collection type that is supported. Note: By the same principle, `Newtonsoft.Json` does not support parsing JSON arrays into immutables as well. Note 2: When using immutable types and `ReferenceHandling.Preserve`, you will not be able to generate payloads that are capables of round-tripping. -* **Immutable types**: i.e: `ImmutableList` and `ImmutableDictionary` -* **System.Array**: i.e: `Array` and `T[]` +* **Immutable types**: e.g: `ImmutableList` and `ImmutableDictionary` +* **System.Array**: e.g: `Array` and `T[]` ## Value types