Use of ReferenceHandler.Preserve (considerably) increase Memory Pressure

[codingdna2]

Description

I've enabled in my ASP.NET Core project the usage of the built-in ReferenceHandler as following.

services.AddControllers()
    .AddJsonOptions(options =>
    {
        options.JsonSerializerOptions.IgnoreNullValues = true;
        options.JsonSerializerOptions.Converters.Add(new TimeSpanConverter());
        options.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.Preserve;
    });

I've observed my process, which was previously running with a memory consuption of 110-120Mb, running at over 2GB (and increasing) a few seconds minutes after I've connected my client. My client is a C# client generated with NSwag.

Configuration

• NET 5.0
• Windows 10 v20H2 build 19042.746
• Architecture x64

Other information

I've tried to have a look at source code and I see PreserveReferenceHandler have a static reference (ReferenceHandler.cs) and contains a dictionary from which values are never removed (PreserveReferenceResolver.cs).. So I can just guess disposing would occur when the whole serializer is disposed but it's not clear to me when this happens.

Am I using it in a wrong way?
Thanks in advance for any help.
Dan

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I've enable in my ASP.NET Core project the usage of the built-in ReferenceHandler as following.

services.AddControllers()
    .AddJsonOptions(options =>
    {
        options.JsonSerializerOptions.IgnoreNullValues = true;
        options.JsonSerializerOptions.Converters.Add(new TimeSpanConverter());
        options.JsonSerializerOptions.ReferenceHandler = ReferenceHandler.Preserve;
    });

I've observed my process, which was previously running with a memory consuption of 110-120Mb, running at over 2GB (and increasing) after with seconds after I connect my client. My client is a C# client generated with NSwag.

Configuration

• NET 5.0
• Windows 10 v20H2 build 19042.746
• Architecture x64

Other information

I've tried to have a look at source code and I see PreserveReferenceHandler have a static reference (ReferenceHandler.cs) and contains a dictionary from which values are never removed (PreserveReferenceResolver.cs).. So I can just guess disposing would occur when the whole serializer is disposed but it's not clear to me when this happens.

Is this a memory leak? Am I using it in a wrong way?
Thanks in advance for any help.

Author:	codingdna2
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

Hi @codingdna2, would you be able to supply a minimal reproduction that demonstrates the issue? If not, have you tried profiling your application?

[layomia]

cc @jozkee

This reminds me of dotnet/aspnetcore#24230. Are you able to take a memory dump, similar to that issue, and share what you find with us?

How many elements (unique objects with different $ids) are placed in the reference dictionary you mentioned? JsonSerializer, by default, news up an instance of the reference dictionary for every call to the serializer, which is destroyed when (de)serialization is complete. If the problem here is using the preserve-reference feature, this suggests to me that a single call to the serializer is creating a whole lot of objects based on the incoming JSON payload (when deserializing), or based on the size of the instantiated input object graph (when serializing).

Is it possible to check if the same issue happens if Newtonsoft.Json is used instead?

[codingdna2]

Hi @eiriktsarpalis, I'll try my best to create a sample repo.

What I've done was to enable/disable the reference handling and observing the memory increase. My model is quite big and I've tried to use reference handling to decrease the size of my JSONs. The client is also quite complex and makes various call to the APIs in response to some SignalR notifications.

I've also tried to profile the application but I cannot understand exactly what kind of objects get allocated (probably I don't have the experience to carry such investigation myself).

@layomia About object graph, I've checked one of my responses and the biggest $id I see is 3478. But as you said if the dictionary is destroyed at every call, I wouldn't expect them to pile.

Give me some time, I'll try to collect some more informations.

[layomia]

Thanks @codingdna2. I'll start to look at this, but yes a sample repro would be really helpful.

[codingdna2]

Sorry for the delay, here is the sample repro. It's close enough to my configuration. The model is quite simple and the memory grows slowly (but steady).

EDIT: I'm not sure replicates it or if it's a memory leak but enabling/disabling the ReferenceHandler sure makes a big difference. I'm now collecting some statistics..

EDIT2: After 200 minutes, it's now at 540MB. It doesn't feel right but I don't know what to think...

@layomia, @eiriktsarpalis Let me know if you need anything else.

[codingdna2]

Last question asked by @layomia.

Is it possible to check if the same issue happens if Newtonsoft.Json is used instead?

Yes, It seems to have the same effect.

services.AddControllers().AddNewtonsoftJson(
    options =>
    {
        options.SerializerSettings.PreserveReferencesHandling = Newtonsoft.Json.PreserveReferencesHandling.All;
    });

[codingdna2]

I've done an additional test using <ServerGarbageCollection>false</ServerGarbageCollection> which has finally convinced me there's no leak. Memory is stable at 61MB. Surely ReferenceHandler have a strong impact on memory which makes it unusable for me.

EDIT: The (unnecessary) question I'm left with is: Why having entities in a dictionary for the time of the serialization, (apparently) cause GC to move them to an higher generation?

[eiriktsarpalis]

I would recommend closing the issue for now. If you could provide us with a reproduction (e.g. containing the type of object graphs that illustrate the behaviour you are seeing) we can take a second look.

[jozkee]

Why having entities in a dictionary for the time of the serialization, (apparently) cause GC to move them to an higher generation?

@codingdna2 the dictionary holds unique objects found during serialization and helps us to determine if the current object to be serialized was previously serialized, if so, the serializer only writes an object with a pointer property ($ref) instead of writing the whole object for a second time.

This is beneficial for a couple cases:

1. If your object has a large amount of duplicate references, using ReferenceHandler.Preserve can significantly reduce the size of your resulting JSON.
2. If your object happens to contain a cycle, the pointer written by the serializer when Preserve is on serves as a terminator to avoid infinite recursion.

[codingdna2]

I would recommend closing the issue for now. If you could provide us with a reproduction (e.g. containing the type of object graphs that illustrate the behaviour you are seeing) we can take a second look.

@eiriktsarpalis, it was posted above...

https://github.com/codingdna2/JsonReferenceHandlerIssue