Crash after linking code using DataContractSerializer

[marek-safar]

---

Issue moved from dotnet/linker#1460

• Please respond to @rolfbjarne.

---

From @rolfbjarne on Friday, August 28, 2020 2:06:52 PM

Test case: linkertestcase-e6b5538.zip

To repro run test.sh:

$ ./test.sh
+ dotnet --version
5.0.100-rc.1.20426.3
+ dotnet publish -r osx-x64 --self-contained true -p:MonoAOT=true -p:VM=mono -p:PublishTrimmed=true Executable/Executable.csproj /bl:x.binlog
Microsoft (R) Build Engine version 16.8.0-preview-20425-03+384d02a5f for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/local/share/dotnet/sdk/5.0.100-rc.1.20426.3/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/local/share/dotnet/sdk/5.0.100-rc.1.20426.3/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/local/share/dotnet/sdk/5.0.100-rc.1.20426.3/dotnet.dll -maxcpucount -property:RuntimeIdentifier=osx-x64 -property:SelfContained=true -p:MonoAOT=true -p:VM=mono -p:PublishTrimmed=true -restore -target:Publish -verbosity:m /bl:x.binlog Executable/Executable.csproj
  Determining projects to restore...
  Restored /Users/rolf/test/linkertestcase/Executable/Executable.csproj (in 4.87 sec).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  Executable -> /Users/rolf/test/linkertestcase/Executable/bin/Debug/net5.0/osx-x64/Executable.dll
  Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  Executable -> /Users/rolf/test/linkertestcase/Executable/bin/Debug/net5.0/osx-x64/publish/
+ ./Executable/bin/Debug/net5.0/osx-x64/publish/Executable
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Runtime.Serialization.CodeGenerator.VerifyParameterCount(MethodInfo methodInfo, Int32 expectedCount)
   at System.Runtime.Serialization.CodeGenerator.Call(Object thisObj, MethodInfo methodInfo, Object param1, Object param2)
   at System.Runtime.Serialization.XmlFormatWriterGenerator.CriticalHelper.InternalSerialize(MethodInfo methodInfo, LocalBuilder memberValue, Type memberType, Boolean writeXsiType)
   at System.Runtime.Serialization.XmlFormatWriterGenerator.CriticalHelper.WriteValue(LocalBuilder memberValue, Boolean writeXsiType)
   at System.Runtime.Serialization.XmlFormatWriterGenerator.CriticalHelper.WriteMembers(ClassDataContract classContract, LocalBuilder extensionDataLocal, ClassDataContract derivedMostClassContract)
   at System.Runtime.Serialization.XmlFormatWriterGenerator.CriticalHelper.WriteClass(ClassDataContract classContract)
   at System.Runtime.Serialization.XmlFormatWriterGenerator.CriticalHelper.GenerateClassWriter(ClassDataContract classContract)
   at System.Runtime.Serialization.XmlFormatWriterGenerator.GenerateClassWriter(ClassDataContract classContract)
   at System.Runtime.Serialization.ClassDataContract.CreateXmlFormatWriterDelegate()
   at System.Runtime.Serialization.ClassDataContract.get_XmlFormatWriterDelegate()
   at System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator xmlWriter, Object obj, XmlObjectSerializerWriteContext context)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle declaredTypeHandle)
   at System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.DataContractSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph)
   at System.Runtime.Serialization.DataContractSerializer.WriteObject(XmlWriter writer, Object graph)
   at DataContractTest.ToXml[T](T obj) in /Users/rolf/test/linkertestcase/Executable/Program.cs:line 15
   at DataContractTest.Flags() in /Users/rolf/test/linkertestcase/Executable/Program.cs:line 53
   at C.Main() in /Users/rolf/test/linkertestcase/Executable/Program.cs:line 62
./test.sh: line 6: 87351 Abort trap: 6           ./Executable/bin/Debug/net5.0/osx-x64/publish/Executable

This is the C# code:

public class DataContractTest {
	public static string ToXml<T> (T obj)
	{
		var sb = new StringBuilder();
		using (var x = XmlWriter.Create (sb, new XmlWriterSettings ())) {
			var s = new DataContractSerializer (typeof (T));
			s.WriteObject(x, obj);
		}
		return sb.ToString();
	}

	public static T FromXml<T> (string xml)
	{
		using (var r = XmlReader.Create (new StringReader (xml))) {
			var s = new DataContractSerializer (typeof (T));
			return (T) s.ReadObject (r);
		}
	}

	[DataContract (Namespace = "mb")]
	public class TestClass
	{
		public TestClass (SomeTypes types)
		{
			Types = types;
		}

		[DataMember]
		public SomeTypes Types { get; set; }
	}

	[DataContract (Namespace = "mb")]
	[Flags]
	public enum SomeTypes {
		[EnumMember] None = 0,
		[EnumMember] Image = 1,
		[EnumMember] Audio = 2,
		[EnumMember] Video = 4,
		[EnumMember] Document = 8
	}

	public void Flags ()
	{
		var t1 = new TestClass (SomeTypes.Audio | SomeTypes.Image);
		var st = ToXml (t1);
		var t2 = FromXml<TestClass> (st);
		Console.WriteLine ($"{t2.Types}, {t1.Types}");
	}
}

class C {
	static void Main ()
	{
		new DataContractTest ().Flags ();
	}
}

[marek-safar]

/cc @eerhardt

[eerhardt]

Similar to #41389.

In order to make DataContractSerializer serialization work with the ILLinker, we will probably need to address dotnet/linker#1087.

[eerhardt]

I verified that fixing #42754 will also fix this issue. Applying the fix to the above repro project causes it to successfully print out Image, Audio, Image, Audio instead of crashing.