Segfault in libcoreclr.so

[corruptmem]

I'm experiencing a segmentation fault on .NET Core 2 on Linux/docker, under versions 2.1.2 and 2.0.9, when my process is under heavy load and using a lot of memory.

I haven't tried to reproduce outside of Linux/docker, but I have reproduced on both AWS Kubernetes and my local Docker for Windows with identical stack traces.

Here are some details from capturing the segfault under the debugger:

Process 1608 stopped
* thread dotnet/runtime#3876: tid = 1664, 0x00007fff821131b3, name = 'dotnet', stop reason = signal SIGSEGV: address access protected (fault address: 0x7ffd217b05c8)
    frame #0: 0x00007fff821131b3
->  0x7fff821131b3: rep
    0x7fff821131b4: movsq  (%rsi), %es:(%rdi)
    0x7fff821131b6: incl   0x8(%rbx)
    0x7fff821131b9: movl   $0x1, %eax

(lldb) clrstack
OS Thread Id: 0x680 (54)
        Child SP               IP Call Site
00007FFB5AFFA850 00007FFF821131B3 System.Collections.Generic.List`1+Enumerator[[System.ValueTuple`3[[System.DateTime, System.Private.CoreLib],[Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource, Ist.Terraform.TimeSeries],[System.Decimal, System.Private.CoreLib]], System.Private.CoreLib]].MoveNext() [/root/coreclr/src/mscorlib/shared/System/Collections/Generic/List.cs @ 1139]
00007FFB5AFFA870 00007FFF82112F6B System.Linq.Enumerable.Min[[System.ValueTuple`3[[System.DateTime, System.Private.CoreLib],[Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource, Ist.Terraform.TimeSeries],[System.Decimal, System.Private.CoreLib]], System.Private.CoreLib],[System.DateTime, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.ValueTuple`3<System.DateTime,Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource,System.Decimal>>, System.Func`2<System.ValueTuple`3<System.DateTime,Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource,System.Decimal>,System.DateTime>) [/root/corefx-1818530/src/System.Linq/src/System/Linq/Min.cs @ 910]
00007FFB5AFFA990 00007FFF82112CE4 Ist.Terraform.TimeSeries.CompositeCalc.Portfolio.PortfolioBuyHoldReturnCalc+<>c__DisplayClass23_0.<.ctor>b__9(System.ValueTuple`3<System.DateTime,Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource,System.Decimal>)
00007FFB5AFFA9D0 00007FFF82112B6A System.Linq.Enumerable+WhereListIterator`1[[System.ValueTuple`3[[System.DateTime, System.Private.CoreLib],[Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource, Ist.Terraform.TimeSeries],[System.Decimal, System.Private.CoreLib]], System.Private.CoreLib]].ToList() [/root/corefx-1818530/src/System.Linq/src/System/Linq/Where.cs @ 397]
00007FFB5AFFAAC0 00007FFF82100B35 Ist.Terraform.TimeSeries.CompositeCalc.Portfolio.PortfolioBuyHoldReturnCalc..ctor(Ist.Terraform.TimeSeries.Interfaces.ISeriesContainer, System.Collections.Generic.IDictionary`2<Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource,Ist.Terraform.TimeSeries.Interfaces.ISeries`1<System.Nullable`1<Double>>>, System.Collections.Generic.IList`1<Ist.Terraform.TimeSeries.CompositeCalc.CompositeRatio>, System.Collections.Generic.IList`1<Ist.Terraform.TimeSeries.CompositeCalc.CompositeRatio>, System.Nullable`1<System.DateTime>, Ist.Terraform.TimeSeries.Interfaces.ISeries`1<System.Nullable`1<Double>>, System.Nullable`1<Double>) [/app/src/Ist.Terraform.TimeSeries/CompositeCalc/Portfolio/PortfolioBuyHoldReturnCalc.cs @ 64]
00007FFB5AFFABD0 00007FFF820FF575 Ist.Terraform.TimeSeries.CompositeCalc.Portfolio.CalculatedCostReturnCalc..ctor(Ist.Terraform.TimeSeries.Interfaces.ISeriesContainer, System.Collections.Generic.IDictionary`2<Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource,Ist.Terraform.TimeSeries.Interfaces.ISeries`1<System.Nullable`1<Double>>>, System.Collections.Generic.IList`1<Ist.Terraform.TimeSeries.CompositeCalc.CompositeRatio>, Ist.Terraform.TimeSeries.Interfaces.ISeries`1<System.Nullable`1<Double>>, System.Collections.Generic.IList`1<Ist.Terraform.TimeSeries.CompositeCalc.CompositeRatio>, System.Nullable`1<System.DateTime>, System.Nullable`1<Double>) [/app/src/Ist.Terraform.TimeSeries/CompositeCalc/Portfolio/CalculatedCostReturnCalc.cs @ 24]
00007FFB5AFFAC50 00007FFF820FF419 Ist.Terraform.TimeSeries.TimeSeriesModuleFactory.CreatePortfolioModel(Int32, System.String, Ist.Terraform.Data.Models.Enums.CompositeType, System.Collections.Generic.IDictionary`2<Ist.Terraform.TimeSeries.CompositeCalc.CompositeSource,Ist.Terraform.TimeSeries.Interfaces.ISeries`1<System.Nullable`1<Double>>>, Ist.Terraform.Data.Models.Enums.CurrencyCode, Ist.Terraform.Data.Models.Enums.CompositeCalcType, System.Collections.Generic.IList`1<Ist.Terraform.TimeSeries.CompositeCalc.CompositeRatio>, System.Collections.Generic.IList`1<Ist.Terraform.TimeSeries.CompositeCalc.CompositeRatio>, Ist.Terraform.Data.Models.Enums.RebalanceFrequency, System.Nullable`1<System.DateTime>, System.Nullable`1<Double>, Ist.Terraform.TimeSeries.Interfaces.ISeries`1<System.Nullable`1<Double>>) [/app/src/Ist.Terraform.TimeSeries/TimeSeriesModuleFactory.cs @ 261]
00007FFB5AFFAD20 00007FFF817F8544 Ist.Terraform.TimeSeries.TimeSeriesModuleFactory+<CreateSystemPortfolioAsync>d__25.MoveNext() [/app/src/Ist.Terraform.TimeSeries/TimeSeriesModuleFactory.cs @ 865]
00007FFB5AFFAE30 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFAE90 00007FFF817FFE71 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Ist.Terraform.TimeSeries.TimeSeriesModuleFactory+<CreateSystemPortfolioAsync>d__25, Ist.Terraform.TimeSeries]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFAEC0 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFAF00 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFAFA0 00007FFF7CB9FD46 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 421]
00007FFB5AFFAFD0 00007FFF7CBDC60B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 639]
00007FFB5AFFB000 00007FFF818054AC Ist.Terraform.TimeSeries.TimeSeriesModuleFactory+<GetInvestableReturnsAsync>d__15.MoveNext()
00007FFB5AFFB080 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFB0E0 00007FFF820F6031 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Ist.Terraform.TimeSeries.TimeSeriesModuleFactory+<GetInvestableReturnsAsync>d__15, Ist.Terraform.TimeSeries]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFB110 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFB150 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFB1F0 00007FFF7CB9FD46 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 421]
00007FFB5AFFB220 00007FFF7CBDC60B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 639]
00007FFB5AFFB250 00007FFF81806CB4 Ist.Terraform.TimeSeries.TimeSeriesModuleFactory+<GetPricesAsync>d__30.MoveNext()
00007FFB5AFFB360 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFB3C0 00007FFF818097F1 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Ist.Terraform.TimeSeries.TimeSeriesModuleFactory+<GetPricesAsync>d__30, Ist.Terraform.TimeSeries]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFB3F0 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFB430 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFB4D0 00007FFF7CB9FD46 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 421]
00007FFB5AFFB500 00007FFF7CBDC60B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 639]
00007FFB5AFFB530 00007FFF8180BEFA Ist.Terraform.Data.Repositories.PostgresRepository+<GetSeriesPricesAsync>d__24.MoveNext()
00007FFB5AFFBCB0 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFBD10 00007FFF81819751 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Ist.Terraform.Data.Repositories.PostgresRepository+<GetSeriesPricesAsync>d__24, Ist.Terraform.Data]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFBD40 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFBD80 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFBE20 00007FFF7CBD641C System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 424]
00007FFB5AFFBE50 00007FFF7CBA93BF System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 646]
00007FFB5AFFBE70 00007FFF8181BFA0 Npgsql.NpgsqlTransaction+<Commit>d__17.MoveNext()
00007FFB5AFFBEC0 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFBF20 00007FFF80B0D6C0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Npgsql.NpgsqlTransaction+<Commit>d__17, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFBF50 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFBF90 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFC030 00007FFF7CBD641C System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 424]
00007FFB5AFFC060 00007FFF7CBA93BF System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 646]
00007FFB5AFFC080 00007FFF8181CBA4 Npgsql.NpgsqlConnector+<ExecuteInternalCommand>d__204.MoveNext()
00007FFB5AFFC1B0 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFC210 00007FFF80381090 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Npgsql.NpgsqlConnector+<ExecuteInternalCommand>d__204, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFC240 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFC280 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFC320 00007FFF7CB9FD46 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 421]
00007FFB5AFFC350 00007FFF7CBDC60B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 639]
00007FFB5AFFC380 00007FFF80AF9131 Npgsql.NpgsqlConnector+<>c__DisplayClass161_0+<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()
00007FFB5AFFC470 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFC4D0 00007FFF80AFD161 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Npgsql.NpgsqlConnector+<>c__DisplayClass161_0+<<ReadMessage>g__ReadMessageLong|0>d, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFC500 00007FFF7CB9A637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFB5AFFC540 00007FFF7CB5098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFB5AFFC5E0 00007FFF7CBD641C System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 424]
00007FFB5AFFC610 00007FFF7CBA93BF System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 646]
00007FFB5AFFC630 00007FFF80AF9D95 Npgsql.NpgsqlReadBuffer+<>c__DisplayClass31_0+<<Ensure>g__EnsureLong|0>d.MoveNext()
00007FFB5AFFC6C0 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFC720 00007FFF80AFCB70 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Npgsql.NpgsqlReadBuffer+<>c__DisplayClass31_0+<<Ensure>g__EnsureLong|0>d, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFB5AFFC750 00007FFF7FB06D73 System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs @ 219]
00007FFB5AFFC770 00007FFF7FB075F0 System.Net.Sockets.SocketAsyncEventArgs.ExecutionCallback(System.Object) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs @ 439]
00007FFB5AFFC780 00007FFF7CB0CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFB5AFFC7E0 00007FFF7FB08169 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs @ 786]
00007FFB5AFFC800 00007FFF7FB095A4 System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs @ 413]
00007FFB5AFFC820 00007FFF7FB08686 System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, Byte[], Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs @ 122]
00007FFB5AFFC830 00007FFF7FB15202 System.Net.Sockets.SocketAsyncContext+BufferMemoryReceiveOperation.InvokeCallback(Boolean) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 464]
00007FFB5AFFC880 00007FFF7FB15D1B System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].ProcessAsyncOperation(System.__Canon) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 855]
00007FFB5AFFC8B0 00007FFF7FB16318 System.Net.Sockets.SocketAsyncContext+OperationQueue`1+<>c[[System.__Canon, System.Private.CoreLib]].<.cctor>b__18_0(System.Object) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 697]
00007FFB5AFFC8C0 00007FFF7CB9C6E8 System.Threading.ThreadPoolWorkQueue.Dispatch() [/root/coreclr/src/mscorlib/src/System/Threading/ThreadPool.cs @ 588]
00007FFB5AFFCC90 00007ffff6269ecf [DebuggerU2MCatchHandlerFrame: 00007ffb5affcc90]

(lldb) bt
* thread dotnet/runtime#3876: tid = 1664, 0x00007fff821131b3, name = 'dotnet', stop reason = signal SIGSEGV: address access protected (fault address: 0x7ffd217b05c8)
  * frame #0: 0x00007fff821131b3
    frame dotnet/coreclr#1: 0x00007fff82112f6b
    frame dotnet/coreclr#2: 0x00007fff82112ce4
    frame dotnet/coreclr#3: 0x00007fff82112b6a
    frame dotnet/coreclr#4: 0x00007fff82100b35
    frame dotnet/coreclr#5: 0x00007fff820ff575
    frame dotnet/coreclr#6: 0x00007fff820ff419
    frame dotnet/coreclr#7: 0x00007fff817f8544
    frame dotnet/coreclr#8: 0x00007fff7cb0cf9d
    frame dotnet/coreclr#9: 0x00007fff817ffe71
    frame dotnet/coreclr#10: 0x00007fff7cb9a637
    frame dotnet/coreclr#11: 0x00007fff7cb5098e
    frame dotnet/coreclr#12: 0x00007fff7cb9fd46
    frame dotnet/coreclr#13: 0x00007fff7cbdc60b
    frame dotnet/coreclr#14: 0x00007fff818054ac
    frame dotnet/coreclr#15: 0x00007fff7cb0cf9d
    frame dotnet/coreclr#16: 0x00007fff820f6031
    frame dotnet/coreclr#17: 0x00007fff7cb9a637
    frame dotnet/coreclr#18: 0x00007fff7cb5098e
    frame dotnet/coreclr#19: 0x00007fff7cb9fd46
    frame dotnet/coreclr#20: 0x00007fff7cbdc60b
    frame dotnet/coreclr#21: 0x00007fff81806cb4
    frame dotnet/coreclr#22: 0x00007fff7cb0cf9d
    frame dotnet/coreclr#23: 0x00007fff818097f1
    frame dotnet/coreclr#24: 0x00007fff7cb9a637
    frame dotnet/runtime#3858: 0x00007fff7cb5098e
    frame dotnet/runtime#3859: 0x00007fff7cb9fd46
    frame dotnet/coreclr#27: 0x00007fff7cbdc60b
    frame dotnet/runtime#3860: 0x00007fff8180befa
    frame dotnet/runtime#3861: 0x00007fff7cb0cf9d
    frame dotnet/runtime#3862: 0x00007fff81819751
    frame dotnet/runtime#3863: 0x00007fff7cb9a637
    frame dotnet/coreclr#32: 0x00007fff7cb5098e
    frame dotnet/runtime#3864: 0x00007fff7cbd641c
    frame dotnet/runtime#3865: 0x00007fff7cba93bf
    frame dotnet/coreclr#35: 0x00007fff8181bfa0
    frame dotnet/coreclr#36: 0x00007fff7cb0cf9d
    frame dotnet/runtime#3866: 0x00007fff80b0d6c0
    frame dotnet/coreclr#38: 0x00007fff7cb9a637
    frame dotnet/runtime#3867: 0x00007fff7cb5098e
    frame dotnet/runtime#3868: 0x00007fff7cbd641c
    frame dotnet/coreclr#41: 0x00007fff7cba93bf
    frame dotnet/coreclr#42: 0x00007fff8181cba4
    frame dotnet/runtime#3869: 0x00007fff7cb0cf9d
    frame dotnet/runtime#3870: 0x00007fff80381090
    frame dotnet/runtime#3871: 0x00007fff7cb9a637
    frame dotnet/coreclr#46: 0x00007fff7cb5098e
    frame dotnet/runtime#3872: 0x00007fff7cb9fd46
    frame dotnet/coreclr#48: 0x00007fff7cbdc60b
    frame dotnet/runtime#3873: 0x00007fff80af9131
    frame dotnet/coreclr#50: 0x00007fff7cb0cf9d
    frame dotnet/runtime#3874: 0x00007fff80afd161
    frame dotnet/runtime#3875: 0x00007fff7cb9a637
    frame dotnet/coreclr#53: 0x00007fff7cb5098e
    frame dotnet/runtime#3876: 0x00007fff7cbd641c
    frame dotnet/runtime#3877: 0x00007fff7cba93bf
    frame dotnet/coreclr#56: 0x00007fff80af9d95
    frame dotnet/runtime#3878: 0x00007fff7cb0cf9d
    frame dotnet/coreclr#58: 0x00007fff80afcb70
    frame dotnet/runtime#3879: 0x00007fff7fb06d73
    frame dotnet/runtime#3880: 0x00007fff7cb0cf9d
    frame dotnet/runtime#3881: 0x00007fff7fb08169
    frame dotnet/coreclr#62: 0x00007fff7fb095a4
    frame dotnet/coreclr#63: 0x00007fff7fb08686
    frame dotnet/coreclr#64: 0x00007fff7fb15d1b
    frame dotnet/runtime#3882: 0x00007fff7fb16318
    frame dotnet/coreclr#66: 0x00007ffff6269ecf libcoreclr.so`CallDescrWorkerInternal + 124
    frame dotnet/coreclr#67: 0x00007ffff618a5fc libcoreclr.so`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 83 at callhelpers.cpp:78
    frame dotnet/runtime#3883: 0x00007ffff618a5a9 libcoreclr.so`MethodDescCallSite::CallTargetWorker(this=<unavailable>, pArguments=<unavailable>, pReturnValue=0x00007ffb5affcb28, cbReturnValue=8) + 857 at callhelpers.cpp:620
    frame dotnet/coreclr#69: 0x00007ffff630a845 libcoreclr.so`QueueUserWorkItemManagedCallback(void*) [inlined] MethodDescCallSite::Call_RetBool(this=0x00007ffb5affcb30, pArguments=<unavailable>) + 15 at callhelpers.h:433
    frame dotnet/runtime#3884: 0x00007ffff630a836 libcoreclr.so`QueueUserWorkItemManagedCallback(pArg=0x00007ffb5affcebe) + 166 at comthreadpool.cpp:451
    frame dotnet/coreclr#71: 0x00007ffff615bf2d libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchInner(pCallState=0x00007ffb5affcd68) + 413 at threads.cpp:8850
    frame dotnet/runtime#3885: 0x00007ffff615bedb libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchMiddle(pCallState=0x00007ffb5affcd68) + 83 at threads.cpp:8901
    frame dotnet/runtime#3886: 0x00007ffff615be88 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_8::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::{lambda(Param*)#1}::operator()(Param*) const at threads.cpp:9139
    frame dotnet/runtime#3887: 0x00007ffff615be88 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at threads.cpp:9141
    frame dotnet/coreclr#75: 0x00007ffff615be88 libcoreclr.so`ManagedThreadBase_DispatchOuter(pCallState=<unavailable>) + 248 at threads.cpp:9178
    frame dotnet/runtime#3888: 0x00007ffff615c690 libcoreclr.so`ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) [inlined] ManagedThreadBase_FullTransitionWithAD(pAppDomain=<unavailable>, pTarget=<unavailable>, args=<unavailable>, filterType=ThreadPoolThread)(void*), void*, UnhandledExceptionLocation) + 43 at threads.cpp:9199
    frame dotnet/runtime#3889: 0x00007ffff615c665 libcoreclr.so`ManagedThreadBase::ThreadPool(pAppDomain=<unavailable>, pTarget=<unavailable>, args=<unavailable>)(void*), void*) + 21 at threads.cpp:9241
    frame dotnet/runtime#3890: 0x00007ffff62eea47 libcoreclr.so`ManagedPerAppDomainTPCount::DispatchWorkItem(this=0x00000000006952a0, foundWork=0x00007ffb5affcebf, wasNotRecalled=0x00007ffb5affcebe) + 295 at threadpoolrequest.cpp:754
    frame dotnet/runtime#3891: 0x00007ffff617ad23 libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(void*) [inlined] ThreadpoolMgr::ExecuteWorkRequest(bool*, bool*) + 1267 at win32threadpool.cpp:1560
    frame dotnet/coreclr#80: 0x00007ffff617ad02 libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(lpArgs=<unavailable>) + 1234 at win32threadpool.cpp:2024
    frame dotnet/coreclr#81: 0x00007ffff64f0f72 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x00007ffb74003b80) + 306 at thread.cpp:1682
    frame dotnet/coreclr#82: 0x00007ffff79bf494 libpthread.so.0`start_thread + 196
    frame dotnet/runtime#3892: 0x00007ffff6e64acf libc.so.6`clone + 63

(lldb) sos VerifyHeap
Object 00007ffb01dd4ff8 has an invalid method table.
Last good object: 00007FFB01DD4FD8.
Object 00007ffe12430570 has an invalid method table.
Last good object: 00007FFE12430358.
Object 00007fff4ffff000 has an invalid method table.Object 00007ffae914b000 has an invalid method table.Object 00007ffa9593b000 has an invalid method table.Object 00007ffe8ffff000 has an invalid method table.Object 00007fff5ffff000 has an invalid method table.Object 00007ffad9149000 has an invalid method table.Object 00007ffaa593d000 has an invalid method table.Failed to request SyncBlk at index 1.
VerifyHeap  failed

I don't have the coredump from this particular crash (because it got OOM killed after calling VerifyHeap!), but most times it crashes, the top of the stack is identical to this (see attached file 4 for one of the times it didn't look like this).

Unfortunately I can't post the code because it's proprietary. We're working on trying to produce a minimal test case, but that's a lot easier said than done. However if you have somewhere secure and private to upload core dumps to, I could do that.

I have also attached information from the debugger from other crashes (the one above is file 3):

dumpinfo1.txt
dumpinfo2.txt
dumpinfo3.txt
dumpinfo4.txt

The environment I set up to reproduce the issue is here: https://github.com/ist-ltd/dotnet-core-debug-helper-tools - but it relies on a private docker image.

[corruptmem]

If I execute the program with env var COMPlus_GCStress=0xC, it crashes at startup:

(lldb) run
Process 2073 launched: '/usr/bin/dotnet' (x86_64)
Process 2073 stopped
* thread dotnet/coreclr#11: tid = 2084, 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] Thread::GetPendingTypeLoad(this=0x0000000000000000) at threads.h:3195, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x128)
    frame #0: 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] Thread::GetPendingTypeLoad(this=0x0000000000000000) at threads.h:3195
(lldb) frame info
frame #0: 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] Thread::GetPendingTypeLoad(this=0x0000000000000000) at threads.h:3195
(lldb) up
frame dotnet/coreclr#1: 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] PendingTypeLoadHolder::PendingTypeLoadHolder(this=<unavailable>) + 33 at clsload.cpp:4115
(lldb) frame info
frame dotnet/coreclr#1: 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] PendingTypeLoadHolder::PendingTypeLoadHolder(this=<unavailable>) + 33 at clsload.cpp:4115
(lldb) up
frame dotnet/coreclr#2: 0x00007ffff62af3f9 libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(this=0x0000000000760c70, pTypeKey=0x00007fffec8020f0, typeHnd=<unavailable>, targetLevel=CLASS_LOAD_APPROXPARENTS) + 1161 at clsload.cpp:4319
(lldb) bt
* thread dotnet/coreclr#11: tid = 2084, 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] Thread::GetPendingTypeLoad(this=0x0000000000000000) at threads.h:3195, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x128)
    frame #0: 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] Thread::GetPendingTypeLoad(this=0x0000000000000000) at threads.h:3195
    frame dotnet/coreclr#1: 0x00007ffff62af41a libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(TypeKey*, TypeHandle, ClassLoadLevel) [inlined] PendingTypeLoadHolder::PendingTypeLoadHolder(this=<unavailable>) + 33 at clsload.cpp:4115
  * frame dotnet/coreclr#2: 0x00007ffff62af3f9 libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey_Body(this=0x0000000000760c70, pTypeKey=0x00007fffec8020f0, typeHnd=<unavailable>, targetLevel=CLASS_LOAD_APPROXPARENTS) + 1161 at clsload.cpp:4319
    frame dotnet/coreclr#3: 0x00007ffff62aaee6 libcoreclr.so`ClassLoader::LoadTypeHandleForTypeKey(this=0x0000000000760c70, pTypeKey=0x00007fffec8020f0, typeHnd=<unavailable>, targetLevel=CLASS_LOAD_APPROXPARENTS, pInstContext=0x00007fffec8021f0) + 214 at clsload.cpp:4044
    frame dotnet/coreclr#4: 0x00007ffff62aad89 libcoreclr.so`ClassLoader::LoadConstructedTypeThrowing(pKey=0x00007fffec8020f0, fLoadTypes=LoadTypes, level=CLASS_LOAD_APPROXPARENTS, pInstContext=<unavailable>) + 633 at clsload.cpp:1258
    frame dotnet/coreclr#5: 0x00007ffff62acb2e libcoreclr.so`ClassLoader::LoadGenericInstantiationThrowing(pModule=<unavailable>, typeDef=<unavailable>, inst=Instantiation @ 0x00007fffec8021e0, fLoadTypes=LoadTypes, level=CLASS_LOAD_APPROXPARENTS, pInstContext=0x00007fffec8021f0, fFromNativeImage=<unavailable>) + 366 at clsload.cpp:2315
    frame dotnet/coreclr#6: 0x00007ffff613d7fe libcoreclr.so`SigPointer::GetTypeHandleThrowing(this=<unavailable>, pModule=<unavailable>, pTypeContext=0x00007fffec802430, fLoadTypes=LoadTypes, level=CLASS_LOAD_APPROXPARENTS, dropGenericArgumentLevel=3, pSubst=<unavailable>, pZapSigContext=<unavailable>) const + 5070 at siginfo.cpp:1445
    frame dotnet/coreclr#7: 0x00007ffff61454bc libcoreclr.so`MetaSig::GetReturnTypeNormalized(TypeHandle*) const + 54 at siginfo.cpp:2409
    frame dotnet/coreclr#8: 0x00007ffff6145486 libcoreclr.so`MetaSig::GetReturnTypeNormalized(this=0x00007fffec802428, pthValueType=0x00007fffec8022b0) const + 70 at siginfo.cpp:5224
    frame dotnet/coreclr#9: 0x00007ffff60d861f libcoreclr.so`ArgIteratorTemplate<ArgIteratorBase>::ComputeReturnFlags() [inlined] ArgIteratorBase::GetReturnType(pthValueType=<unavailable>) + 47 at callingconvention.h:1632
    frame dotnet/coreclr#10: 0x00007ffff60d8613 libcoreclr.so`ArgIteratorTemplate<ArgIteratorBase>::ComputeReturnFlags(this=0x00007fffec802300) + 35 at callingconvention.h:1352
    frame dotnet/coreclr#11: 0x00007ffff60f118f libcoreclr.so`TransitionFrame::PromoteCallerStackHelper(void (*)(Object**, ScanContext*, unsigned int), ScanContext*, MethodDesc*, MetaSig*) [inlined] ArgIteratorTemplate<ArgIteratorBase>::HasRetBuffArg(this=<unavailable>) + 5 at callingconvention.h:330
    frame dotnet/coreclr#12: 0x00007ffff60f118a libcoreclr.so`TransitionFrame::PromoteCallerStackHelper(this=<unavailable>, fn=(libcoreclr.so`SVR::GCHeap::Promote(Object**, ScanContext*, unsigned int) at gc.cpp:33762), sc=0x00007fffec803cd8, pFunction=<unavailable>, pmsig=0x00007fffec802428)(Object**, ScanContext*, unsigned int), ScanContext*, MethodDesc*, MetaSig*) + 298 at frames.cpp:1291
    frame dotnet/coreclr#13: 0x00007ffff60f0a1f libcoreclr.so`TransitionFrame::PromoteCallerStack(this=0x00007ffbe712ef90, fn=<unavailable>, sc=<unavailable>)(Object**, ScanContext*, unsigned int), ScanContext*) + 543 at frames.cpp:1252
    frame dotnet/coreclr#14: 0x00007ffff61c0ca2 libcoreclr.so`GcStackCrawlCallBack(pCF=0x00007fffec802730, pData=<unavailable>) + 754 at gcenv.ee.common.cpp:291
    frame dotnet/coreclr#15: 0x00007ffff6146f70 libcoreclr.so`Thread::MakeStackwalkerCallback(this=<unavailable>, pCF=0x00007fffec802730, pCallback=(libcoreclr.so`GcStackCrawlCallBack(CrawlFrame*, void*) at gcenv.ee.common.cpp:186), pData=0x00007fffec803be0)(CrawlFrame*, void*), void*) + 160 at stackwalk.cpp:877
    frame dotnet/coreclr#16: 0x00007ffff61471b1 libcoreclr.so`Thread::StackWalkFramesEx(this=<unavailable>, pRD=<unavailable>, pCallback=<unavailable>, pData=<unavailable>, flags=34048, pStartFrame=0x0000000000000000)(CrawlFrame*, void*), void*, unsigned int, Frame*) + 449 at stackwalk.cpp:958
    frame dotnet/coreclr#17: 0x00007ffff614758c libcoreclr.so`Thread::StackWalkFrames(this=0x00007ffbd8009fa0, pCallback=(libcoreclr.so`GcStackCrawlCallBack(CrawlFrame*, void*) at gcenv.ee.common.cpp:186), pData=0x00007fffec803be0, flags=34048, pStartFrame=0x0000000000000000)(CrawlFrame*, void*), void*, unsigned int, Frame*) + 204 at stackwalk.cpp:1041
    frame dotnet/coreclr#18: 0x00007ffff63b30c9 libcoreclr.so`ScanStackRoots(pThread=0x00007ffbd8009fa0, fn=(libcoreclr.so`SVR::GCHeap::Promote(Object**, ScanContext*, unsigned int) at gc.cpp:33762), sc=<unavailable>)(Object**, ScanContext*, unsigned int), ScanContext*) + 345 at gcenv.ee.cpp:149
    frame dotnet/coreclr#19: 0x00007ffff63b2f05 libcoreclr.so`GCToEEInterface::GcScanRoots(fn=<unavailable>, condemned=<unavailable>, max_gen=<unavailable>, sc=<unavailable>)(Object**, ScanContext*, unsigned int), int, int, ScanContext*) + 261 at gcenv.ee.cpp:178
    frame dotnet/coreclr#20: 0x00007ffff63563bb libcoreclr.so`SVR::gc_heap::mark_phase(this=<unavailable>, condemned_gen_number=2, mark_only_p=NO) + 1163 at gc.cpp:19624
    frame dotnet/coreclr#21: 0x00007ffff6352eb2 libcoreclr.so`SVR::gc_heap::gc1(this=<unavailable>) + 466 at gc.cpp:15388
    frame dotnet/coreclr#22: 0x00007ffff6345a4d libcoreclr.so`SVR::gc_heap::garbage_collect(this=<unavailable>, n=2) + 4125 at gc.cpp:16964
    frame dotnet/coreclr#23: 0x00007ffff6344488 libcoreclr.so`SVR::gc_heap::gc_thread_function(this=<unavailable>) + 696 at gc.cpp:5397
    frame dotnet/coreclr#24: 0x00007ffff63441c2 libcoreclr.so`SVR::gc_heap::gc_thread_stub(arg=<unavailable>) + 82 at gc.cpp:24933
    frame dotnet/runtime#3858: 0x00007ffff63b4d38 libcoreclr.so`(anonymous namespace)::CreateNonSuspendableThread(void (*)(void*), void*, char const*)::$_1::__invoke(void*) [inlined] (anonymous namespace)::CreateNonSuspendableThread(void (*)(void*), void*, char const*)::$_1::operator()(void*) const + 168 at gcenv.ee.cpp:1324
    frame dotnet/runtime#3859: 0x00007ffff63b4c9d libcoreclr.so`(anonymous namespace)::CreateNonSuspendableThread(argument=<unavailable>)(void*), void*, char const*)::$_1::__invoke(void*) + 13 at gcenv.ee.cpp:1309
    frame dotnet/coreclr#27: 0x00007ffff64f0f72 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x0000000000733d60) + 306 at thread.cpp:1682
    frame dotnet/runtime#3860: 0x00007ffff79bf494 libpthread.so.0`start_thread + 196
    frame dotnet/runtime#3861: 0x00007ffff6e64acf libc.so.6`clone + 63
(lldb) frame variable
(ClassLoader *) this = 0x0000000000760c70
(TypeKey *) pTypeKey = 0x00007fffec8020f0
(TypeHandle) typeHnd = <no location, value may have been optimized out>

(ClassLoadLevel) targetLevel = CLASS_LOAD_APPROXPARENTS
(ClassLoadLevel) currentLevel = <variable not available>

(ReleaseHolder<PendingTypeLoadEntry>) pLoadingEntry = <no location, value may have been optimized out>

(CrstHolderWithState) unresolvedClassLockHolder = <no location, value may have been optimized out>

(CLRException::HandlerState) __state = <no location, value may have been optimized out>

(Exception *) __pExceptionRaw = <no location, value may have been optimized out>

(CAutoTryCleanup<CLRException::HandlerState>) __autoCleanupTry = <no location, value may have been optimized out>

(NativeExceptionHolderCatchAll) __exceptionHolder = <no location, value may have been optimized out>

(PendingTypeLoadHolder) ptlh = <no location, value may have been optimized out>
(lldb) p pTypeKey->u
(TypeKey::(anonymous union)) $8 = {
  asClass = {
    m_pModule = 0x00007fff7c550400
    m_typeDef = 33555331
    m_numGenericArgs = 1
    m_pGenericArgs = 0x00007fffec802180
  }
  asParamType = (m_paramType = 140735279334400, m_rank = 33555331, m_isTemplateMethodTable = YES)
  asFnPtr = {
    m_callConv = '\0'
    m_numArgs = 32767
    m_pRetAndArgTypes = 0x0000000102000383
  }
}

Note that the external dependencies we are using are GraphQL, AWSSDK, Dapper, Twilio, Nlog, Npgsql, Jetbrains.Annotations - the rest is Microsoft.

[jkotas]

Could you please try to collect this info:

• Find the method that this is trying to walk (CrawlFrame::pFunc in GcStackCrawlCallBack) and prints its name using sos DumpMD
• Find the thread that this is trying to walk (this in Thread::StackWalkFrames), switch to it in lldb, and print its stacktrace using sos clrstack

[corruptmem]

Does this help?

(lldb) f 14
frame dotnet/coreclr#14: 0x00007ffff61c0ca2 libcoreclr.so`GcStackCrawlCallBack(pCF=0x00007fffec904730, pData=<unavailable>) + 754 at gcenv.ee.common.cpp:291
(lldb) sos DumpMD pCF->pFunc
Method Name:          Npgsql.TypeHandling.NpgsqlTypeHandler`1[[System.String, System.Private.CoreLib]].Read[[System.String, System.Private.CoreLib]](Npgsql.NpgsqlReadBuffer, Int32, Boolean, Npgsql.BackendMessages.FieldDescription)
Class:                00007fff7ffc4f18
MethodTable:          00007fff8001ef10
mdToken:              00000000060007C8
Module:               00007fff7fb35798
IsJitted:             no
Current CodeAddr:     0000000000000000
Code Version History:
  CodeAddr:           0000000000000000  (Non-Tiered)
  NativeCodeVersion:  0000000000000000
(lldb) f 17
frame dotnet/coreclr#17: 0x00007ffff614758c libcoreclr.so`Thread::StackWalkFrames(this=0x00007ffbe0000a40, pCallback=(libcoreclr.so`GcStackCrawlCallBack(CrawlFrame*, void*) at gcenv.ee.common.cpp:186), pData=0x00007fffec905be0, flags=34048, pStartFrame=0x0000000000000000)(CrawlFrame*, void*), void*, unsigned int, Frame*) + 204 at stackwalk.cpp:1041
(lldb) print this->m_ThreadId
(DWORD) $11 = 4
(lldb) clrthreads
ThreadCount:      10
UnstartedThread:  0
BackgroundThread: 9
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                                                        Lock
       ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   1    1 4db1 000000000073CA10  2020020 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn
   2    2 4dbf 0000000000737430    21220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn (Finalizer)
  15    3 4dc0 000000000089C0F0    21220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn
  19    6 4dc7 0000000000985140    21220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn
  23    9 4dcd 00007FFBCC0009F0  1020220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn (Threadpool Worker)
  25   10 4dce 00007FFBD4003470    21220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn
  26    4 4dfb 00007FFBE0000A40  1021220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn (Threadpool Worker)
  24    5 4e11 00007FFBD4001270  1021220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn (Threadpool Worker)
  28    7 4e13 00007FFBD400AF30  1021220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn (Threadpool Worker)
  29    8 4e1a 00007FFBD400B580  1021220 Preemptive  0000000000000000:0000000000000000 000000000078E6B0 0     Ukn (Threadpool Worker)
(lldb) thread select 26
(lldb) * thread dotnet/runtime#3859: tid = 19963, 0x00007ffff79c515f libpthread.so.0`__pthread_cond_wait + 191, name = 'dotnet'
    frame #0: 0x00007ffff79c515f libpthread.so.0`__pthread_cond_wait + 191
libpthread.so.0`__pthread_cond_wait:
->  0x7ffff79c515f <+191>: movl   (%rsp), %edi
    0x7ffff79c5162 <+194>: callq  0x7ffff79c7eb0            ; __pthread_disable_asynccancel
    0x7ffff79c5167 <+199>: movq   0x8(%rsp), %rdi
    0x7ffff79c516c <+204>: movl   $0x1, %esi
(lldb) clrstack
OS Thread Id: 0x4dfb (26)
        Child SP               IP Call Site
00007FFBE77FC820 00007ffff79c515f [PrestubMethodFrame: 00007ffbe77fc820] Npgsql.TypeHandling.NpgsqlTypeHandler`1[[System.String, System.Private.CoreLib]].Read[[System.String, System.Private.CoreLib]](Npgsql.NpgsqlReadBuffer, Int32, Boolean, Npgsql.BackendMessages.FieldDescription)
00007FFBE77FC990 00007FFF8056C75C Npgsql.TypeHandling.NpgsqlTypeHandler`1[[System.__Canon, System.Private.CoreLib]].Read[[System.__Canon, System.Private.CoreLib]](Npgsql.NpgsqlReadBuffer, Int32, Npgsql.BackendMessages.FieldDescription)
00007FFBE77FC9F0 00007FFF8056C004 Npgsql.NpgsqlDefaultDataReader.GetFieldValue[[System.__Canon, System.Private.CoreLib]](Int32)
00007FFBE77FCA40 00007FFF8055D701 Npgsql.PostgresDatabaseInfo+<LoadBackendTypes>d__13.MoveNext()
00007FFBE77FCB50 00007FFF7CB2CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFBE77FCBB0 00007FFF8056B551 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Npgsql.PostgresDatabaseInfo+<LoadBackendTypes>d__13, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFBE77FCBE0 00007FFF7CBBA637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFBE77FCC20 00007FFF7CB7098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFBE77FCCC0 00007FFF7CBBFD46 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 421]
00007FFBE77FCCF0 00007FFF7CBFC60B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 639]
00007FFBE77FCD20 00007FFF8055FD2F Npgsql.NpgsqlCommand+<ExecuteDbDataReader>d__100.MoveNext()
00007FFBE77FCDE0 00007FFF7CB2CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFBE77FCE40 00007FFF8056B2A1 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Npgsql.NpgsqlCommand+<ExecuteDbDataReader>d__100, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFBE77FCE70 00007FFF7CBBA637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFBE77FCEB0 00007FFF7CB7098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFBE77FCF50 00007FFF7CC0FC1A System.Threading.Tasks.Task`1[[System.Boolean, System.Private.CoreLib]].TrySetResult(Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 424]
00007FFBE77FCF80 00007FFF7CBF7DBF System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Boolean, System.Private.CoreLib]].SetExistingTaskResult(Boolean) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 646]
00007FFBE77FCFA0 00007FFF80567AAB Npgsql.NpgsqlDataReader+<NextResult>d__46.MoveNext()
00007FFBE77FD1B0 00007FFF7CB2CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFBE77FD210 00007FFF80569940 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib],[Npgsql.NpgsqlDataReader+<NextResult>d__46, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFBE77FD240 00007FFF7CBBA637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFBE77FD280 00007FFF7CB7098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFBE77FD320 00007FFF7CBBFD46 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 421]
00007FFBE77FD350 00007FFF7CBFC60B System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetExistingTaskResult(System.__Canon) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 639]
00007FFBE77FD380 00007FFF80554751 Npgsql.NpgsqlConnector+<>c__DisplayClass161_0+<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()
00007FFBE77FD470 00007FFF7CB2CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFBE77FD4D0 00007FFF805690E1 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[Npgsql.NpgsqlConnector+<>c__DisplayClass161_0+<<ReadMessage>g__ReadMessageLong|0>d, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFBE77FD500 00007FFF7CBBA637 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/TaskContinuation.cs @ 798]
00007FFBE77FD540 00007FFF7CB7098E System.Threading.Tasks.Task.RunContinuations(System.Object) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs @ 3275]
00007FFBE77FD5E0 00007FFF7CBF641C System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Threading/Tasks/future.cs @ 424]
00007FFBE77FD610 00007FFF7CBC93BF System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.VoidTaskResult) [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 646]
00007FFBE77FD630 00007FFF805553B5 Npgsql.NpgsqlReadBuffer+<>c__DisplayClass31_0+<<Ensure>g__EnsureLong|0>d.MoveNext()
00007FFBE77FD6C0 00007FFF7CB2CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFBE77FD720 00007FFF80568C00 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Npgsql.NpgsqlReadBuffer+<>c__DisplayClass31_0+<<Ensure>g__EnsureLong|0>d, Npgsql]].MoveNext() [/root/coreclr/src/mscorlib/src/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 568]
00007FFBE77FD750 00007FFF7F6A6D73 System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs @ 219]
00007FFBE77FD770 00007FFF7F6A75F0 System.Net.Sockets.SocketAsyncEventArgs.ExecutionCallback(System.Object) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs @ 439]
00007FFBE77FD780 00007FFF7CB2CF9D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs @ 167]
00007FFBE77FD7E0 00007FFF7F6A8169 System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs @ 786]
00007FFBE77FD800 00007FFF7F6A95A4 System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs @ 413]
00007FFBE77FD820 00007FFF7F6A8686 System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, Byte[], Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs @ 122]
00007FFBE77FD830 00007FFF7F6B5202 System.Net.Sockets.SocketAsyncContext+BufferMemoryReceiveOperation.InvokeCallback(Boolean) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 464]
00007FFBE77FD880 00007FFF7F6B5D1B System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].ProcessAsyncOperation(System.__Canon) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 855]
00007FFBE77FD8B0 00007FFF7F6B6318 System.Net.Sockets.SocketAsyncContext+OperationQueue`1+<>c[[System.__Canon, System.Private.CoreLib]].<.cctor>b__18_0(System.Object) [/root/corefx-1818530/src/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs @ 697]
00007FFBE77FD8C0 00007FFF7CBBC6E8 System.Threading.ThreadPoolWorkQueue.Dispatch() [/root/coreclr/src/mscorlib/src/System/Threading/ThreadPool.cs @ 588]
00007FFBE77FDC90 00007ffff6269ecf [DebuggerU2MCatchHandlerFrame: 00007ffbe77fdc90]

This appears to be happening the first time my code calls into EFCore, and the exception is occurring as part of the initial npgsql database info loading procedure.

There are some unsafe parts to the NpgsqlReadBuffer class: https://github.com/npgsql/npgsql/blob/2ac23b7520aa53ab67e6e0192dbe3379a71bb8b9/src/Npgsql/NpgsqlReadBuffer.cs#L341 ....

[jkotas]

Yes, this helps. We are trying to find type for ValueTask<string> that is return type of https://github.com/npgsql/npgsql/blob/df65e0a42d2f5efa4f15df7d5da0c0dd07985a46/src/Npgsql/TypeHandling/NpgsqlTypeHandler.cs#L60

Are you able to tell why Process 2073 stopped ? Could you please take a look at the disassembly - is it trying to dererence invalid pointer?

[corruptmem]

Here is the disassembly. It looks like it's dereferencing a null pointer.

Note that it's from a different process - but with the same backtrace. LLDB crashed while I was investigating the last one.

[jkotas]

Here is the disassembly. It looks like it's dereferencing a null pointer.

It seems to accessing the current Thread* on this line PendingTypeLoadHolder ptlh(pLoadingEntry);, but we do not have one.

The problem is that we should have found the type inside ClassLoader::LoadGenericInstantiationThrowing, but we did not find it for some reason and ended up going into ClassLoader::LoadTypeHandleForTypeKey.

It is a bug, however it is unlikely to be the root cause for the crash at the top this issue.

[corruptmem]

I have narrowed down the original issue I reported. In my environment, this program will reliably eventually segfault if left running long enough. For example:

$ .\docker-run.ps1
root@53803362bd7c:/app/bin/Debug/netcoreapp2.1# lldb-3.9 dotnet Repro.dll
(lldb) target create "dotnet"
Current executable set to 'dotnet' (x86_64).
(lldb) settings set -- target.run-args  "Repro.dll"
(lldb) run
Process 15 launched: '/usr/bin/dotnet' (x86_64)
Process 15 stopped and restarted: thread 1 received signal: SIGCHLD
Process 15 stopped and restarted: thread 1 received signal: SIGCHLD
Starting stress loop
1412
2012
7866
8121
[...]
6980
2228
Process 15 stopped
* thread dotnet/coreclr#23: tid = 41, 0x00007fff7d0a8093, name = 'dotnet', stop reason = signal SIGSEGV: address access protected (fault address: 0x7fff58672180)
    frame #0: 0x00007fff7d0a8093
->  0x7fff7d0a8093: rep
    0x7fff7d0a8094: movsq  (%rsi), %es:(%rdi)
    0x7fff7d0a8096: incl   0x8(%rbx)
    0x7fff7d0a8099: movl   $0x1, %eax
(lldb)

I'm still trying to remove bits from the program until I can get to a minimal example, but you should hopefully be able to reproduce the issue with the provided repository, using the following commands:

.\docker-build.ps1
.\docker-run.ps1
lldb-3.9 dotnet Repro.dll
run

[jkotas]

Thanks for the repro @corruptmem . I was able to narrow down the root cause for the failure.

The problem is bad GC info generated by the JIT for assignments of large structs on Unix. The rsi and rdi registers are holding GC references during rep movsq, but they are not marked as such.

Repro:

using System;
using System.Threading;
using System.Runtime.CompilerServices;

struct S1
{
    public long a,b;
    public object str;
}

struct S2
{
    public long a,b,c,d,e;
}

struct S
{
    public S1 s1;
    public S2 s2;
}

class Test
{
    class E
    {
        public S a;
        public S b;

        [MethodImpl(MethodImplOptions.NoInlining)]
        public void Update(int c)
        {
            for (int i = 0; i < c; i++) {
                a = b; // GCInfo generated for this assignment is wrong
            }
        }
    }

    static void Main()
    {
        E e = new E();
        e.Update(1);
    }
}

Compile with /o+, run with the following environment settings and inspect the GCInfo generated by the JIT:

export COMPlus_JitDisasm=Update
export COMPlus_JitGCDump=Update
export COMPlus_TieredCompilation=0

Here are the JIT dumps. It is interesting that the problem does not repro on Windows:
Unix - Wrong.txt
Windows - Correct.txt

@dotnet/jit-contrib Could you please take a look?

[sandreenko]

I have found the issue, will push a fix soon.