Non-deterministic System.AccessViolationException when running tests

[FreddyDgh]

Starting a separate issue for conversation that began with:
dotnet/runtime#33899 (comment)

@lauxjpn @bubibubi Does anyone know if GitHub would allow us to move the comments from that PR into this issue instead, or do we have to manually copy/paste each one?

[lauxjpn]

Copy & paste is the way to go here. I am altering the title a bit, because this exception is thrown regardless of the environment they are being executed in.

[FreddyDgh]

Comments copied from the dotnet/runtime PR:

When running some of our EFCore.Jet.FunctionalTests with the latest nightly-build 5.0.0-preview.3.20171.3 of System.Data.OleDb, the test host often crashes in a non-deterministic fashion (it did with previous nightly-builds as well).

As it turns our, the underlying cause is the following System.AccessViolationException exception, so it makes sense that this results in a non-deterministic crash behavior:

The active test run was aborted. Reason: Test host process crashed : Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Data.Common.UnsafeNativeMethods+ICommandText.Execute(IntPtr, System.Guid ByRef, System.Data.OleDb.tagDBPARAMS, IntPtr ByRef, System.Object ByRef)
   at System.Data.Common.UnsafeNativeMethods+ICommandText.Execute(IntPtr, System.Guid ByRef, System.Data.OleDb.tagDBPARAMS, IntPtr ByRef, System.Object ByRef)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(System.Data.OleDb.tagDBPARAMS, System.Object ByRef)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandText(System.Object ByRef)
   at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(System.Data.CommandBehavior, System.String)
   at System.Data.Jet.JetCommand.InternalExecuteDbDataReader(System.String, System.Data.CommandBehavior)
   at System.Data.Jet.JetCommand.ExecuteDbDataReader(System.Data.CommandBehavior)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(Microsoft.EntityFrameworkCore.Storage.RelationalCommandParameterObject)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+Enumerator[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InitializeReader(Microsoft.EntityFrameworkCore.DbContext, Boolean)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`3<Microsoft.EntityFrameworkCore.DbContext,Boolean,Boolean>, System.Func`3<Microsoft.EntityFrameworkCore.DbContext,Boolean,Microsoft.EntityFrameworkCore.Storage.ExecutionResult`1<Boolean>>, Boolean)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Boolean, System.Func`3<Microsoft.EntityFrameworkCore.DbContext,Boolean,Boolean>, System.Func`3<Microsoft.EntityFrameworkCore.DbContext,Boolean,Microsoft.EntityFrameworkCore.Storage.ExecutionResult`1<Boolean>>)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+Enumerator[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Collections.Generic.LargeArrayBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AddRange(System.Collections.Generic.IEnumerable`1<System.__Canon>)
   at System.Collections.Generic.EnumerableHelpers.ToArray[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
   at Microsoft.EntityFrameworkCore.TestUtilities.QueryAsserter`1+<AssertQuery>d__7`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Microsoft.EntityFrameworkCore.TestUtilities.QueryAsserter`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AssertQuery[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Func`2<Microsoft.EntityFrameworkCore.TestUtilities.ISetSource,System.Linq.IQueryable`1<System.__Canon>>, System.Func`2<Microsoft.EntityFrameworkCore.TestUtilities.ISetSource,System.Linq.IQueryable`1<System.__Canon>>, System.Func`2<System.__Canon,System.Object>, System.Action`2<System.__Canon,System.__Canon>, Boolean, Int32, Boolean, System.String)
   at Microsoft.EntityFrameworkCore.Query.QueryTestBase`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AssertQuery[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Boolean, System.Func`2<Microsoft.EntityFrameworkCore.TestUtilities.ISetSource,System.Linq.IQueryable`1<System.__Canon>>, System.Func`2<Microsoft.EntityFrameworkCore.TestUtilities.ISetSource,System.Linq.IQueryable`1<System.__Canon>>, System.Func`2<System.__Canon,System.Object>, System.Action`2<System.__Canon,System.__Canon>, Boolean, Int32, System.String)
   at Microsoft.EntityFrameworkCore.Query.QueryTestBase`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AssertQuery[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Boolean, System.Func`2<Microsoft.EntityFrameworkCore.TestUtilities.ISetSource,System.Linq.IQueryable`1<System.__Canon>>, System.Func`2<System.__Canon,System.Object>, System.Action`2<System.__Canon,System.__Canon>, Boolean, Int32, System.String)
   at Microsoft.EntityFrameworkCore.Query.SimpleQueryTestBase`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Select_anonymous_empty(Boolean)

This is probably related to this issue here. I currently cannot say, whether this is a direct result of a wrong declaration of the System.Data.OleDb.tagDBPARAMS structure itself, or just a result of some other incorrect structure or behavior in OleDb, that will crash the app much later, but I would assume, it is the latter.

This mostly happens after at least 300 calls, but can happen after 700 calls or even never in our test runs.

Also note, that this behavior happens on an x64 test run, so there seem to be some issues with those as well and it might not be automatically the case, that the x64 structure definitions (or their marshaling) are correct.

Because of your 15K test suite, we are happy to test any fixes.

Originally posted by @lauxjpn in dotnet/runtime#33899 (comment)

@lauxjpn that sounds like a separate issue to me. You mention that it can happen on both x86 and x64, but do you know if it happens on both .NET Core and .NET Framework? I know that @bubibubi mentioned he encountered issues with large-scale test runs when using System.Data.Ole.Db even on .NET Framework.

Originally posted by @FreddyD-GH in dotnet/runtime#33899 (comment)

@lauxjpn it could be that we can't run all the test toghether (i.e. we can't include the tests in the CI pipeline). As far as I know it could also be an OleDb Provider (Jet/ACE) issue.

Originally posted by @bubibubi in dotnet/runtime#33899 (comment)

@FreddyD-GH @bubibubi You guys might be correct.
The stack trace of the dump file reveals that the invalid memory access happens in the ACE provider:

0000007b`b47f4370 00007ffc`1c48740f coreclr!EEPolicy::HandleFatalError+0x8b
0000007b`b47f4980 00007ffc`1c3a1046 coreclr!ProcessCLRException+0x149406
0000007b`b47f4b80 00007ffc`8b42121f ntdll!RtlpExecuteHandlerForException+0xf
0000007b`b47f4bb0 00007ffc`8b3ea289 ntdll!RtlDispatchException+0x219
0000007b`b47f52c0 00007ffc`8b41fe8e ntdll!KiUserExceptionDispatch+0x2e
0000007b`b47f59e0 00007ffc`103e6904 ACECORE+0x176904
0000007b`b47f5a70 00007ffc`103e6f88 ACECORE+0x176f88
0000007b`b47f5ac0 00007ffc`103f02d3 ACECORE+0x1802d3
0000007b`b47f5d50 00007ffc`103f7c55 ACECORE+0x187c55
0000007b`b47f5fc0 00007ffc`103f82f8 ACECORE+0x1882f8
0000007b`b47f6030 00007ffc`14443b14 ACEOLEDB!DllGetClassObject+0xdeb4
0000007b`b47f6080 00007ffc`14437300 ACEOLEDB!DllGetClassObject+0x16a0
0000007b`b47f6190 00007ffc`14436145 ACEOLEDB!DllGetClassObject+0x4e5
0000007b`b47f62c0 00007ffb`be62d26d unknown!unknown+0x0
0000007b`b47f63d0 00007ffb`bede8b88 unknown!unknown+0x0
0000007b`b47f6410 00007ffb`bede886c unknown!unknown+0x0
0000007b`b47f6480 00007ffb`bede7021 unknown!unknown+0x0
0000007b`b47f6500 00007ffb`bf2d8884 unknown!unknown+0x0
0000007b`b47f6620 00007ffb`bf2d5929 unknown!unknown+0x0
0000007b`b47f67c0 00007ffb`bf34e6a0 unknown!unknown+0x0
0000007b`b47f6c10 00007ffb`bf81afd7 unknown!unknown+0x0
0000007b`b47f6d50 00007ffb`bf34dad6 unknown!unknown+0x0
0000007b`b47f6f00 00007ffb`bf34d6b1 unknown!unknown+0x0
0000007b`b47f6f90 00007ffb`bf81a8bd unknown!unknown+0x0
0000007b`b47f7130 00007ffb`bf81a335 unknown!unknown+0x0
0000007b`b47f7190 00007ffb`bf81a2ae unknown!unknown+0x0
0000007b`b47f7220 00007ffb`bf8193ff unknown!unknown+0x0
0000007b`b47f7430 00007ffb`bf7ba4e7 unknown!unknown+0x0
0000007b`b47f7490 00007ffb`bf818fb0 unknown!unknown+0x0
0000007b`b47f7520 00007ffc`1c357116 coreclr!InstantiatingMethodStubWorker+0xd6
0000007b`b47f75e0 00007ffb`bf818e2e unknown!unknown+0x0
0000007b`b47f7640 00007ffb`bf818d58 unknown!unknown+0x0
0000007b`b47f76a0 00007ffb`bff9ab45 unknown!unknown+0x0
0000007b`b47f7740 00007ffb`bff9a9d4 unknown!unknown+0x0
0000007b`b47f7780 00007ffb`bff9a7fa unknown!unknown+0x0
0000007b`b47f7820 00007ffb`bf7ba4e7 unknown!unknown+0x0
0000007b`b47f7880 00007ffb`bff9a703 unknown!unknown+0x0
0000007b`b47f78f0 00007ffc`1c356693 coreclr!CallDescrWorkerInternal+0x83
0000007b`b47f7930 00007ffc`1c262146 coreclr!CallDescrWorkerReflectionWrapper+0x1a
0000007b`b47f7980 00007ffc`1c261818 coreclr!RuntimeMethodHandle::InvokeMethod+0x8f8
0000007b`b47f7f20 00007ffb`be5a052d unknown!unknown+0x0
0000007b`b47f7f90 00007ffb`bf86b20b unknown!unknown+0x0
0000007b`b47f7fd0 00007ffb`bf86aeb6 unknown!unknown+0x0
0000007b`b47f8060 00007ffb`bf86abcc unknown!unknown+0x0
0000007b`b47f80d0 00007ffb`bf86aab7 unknown!unknown+0x0
0000007b`b47f8140 00007ffb`bf86a7f0 unknown!unknown+0x0
0000007b`b47f81c0 00007ffb`bf86a694 unknown!unknown+0x0
0000007b`b47f8220 00007ffb`bf86a5dd unknown!unknown+0x0
0000007b`b47f8290 00007ffb`bf869e82 unknown!unknown+0x0
0000007b`b47f82f0 00007ffb`bf869074 unknown!unknown+0x0
0000007b`b47f8350 00007ffb`bf868fbd unknown!unknown+0x0
0000007b`b47f83b0 00007ffb`bf868904 unknown!unknown+0x0
0000007b`b47f84a0 00007ffb`bf86862c unknown!unknown+0x0
0000007b`b47f8510 00007ffb`bf86851c unknown!unknown+0x0
0000007b`b47f8590 00007ffb`bf84473d unknown!unknown+0x0
0000007b`b47f86a0 00007ffb`bf84400c unknown!unknown+0x0
0000007b`b47f8710 00007ffb`bf843517 unknown!unknown+0x0
0000007b`b47f8790 00007ffb`bf84325f unknown!unknown+0x0
0000007b`b47f8840 00007ffb`bf842b84 unknown!unknown+0x0
0000007b`b47f88a0 00007ffb`bf842acd unknown!unknown+0x0
0000007b`b47f8900 00007ffb`bef4a8eb unknown!unknown+0x0
0000007b`b47f8980 00007ffb`bf841d04 unknown!unknown+0x0
0000007b`b47f89e0 00007ffb`bf841c4d unknown!unknown+0x0
0000007b`b47f8a50 00007ffb`bf8419b7 unknown!unknown+0x0
0000007b`b47f8ac0 00007ffb`bf8418ac unknown!unknown+0x0
0000007b`b47f8b30 00007ffb`bf841779 unknown!unknown+0x0
0000007b`b47f8ba0 00007ffb`bf841174 unknown!unknown+0x0
0000007b`b47f8c90 00007ffb`bf840b3c unknown!unknown+0x0
0000007b`b47f8d00 00007ffb`bf840a17 unknown!unknown+0x0
0000007b`b47f8d80 00007ffb`bf83f6d9 unknown!unknown+0x0
0000007b`b47f8e40 00007ffb`bf83ef2c unknown!unknown+0x0
0000007b`b47f8eb0 00007ffb`bf83ee07 unknown!unknown+0x0
0000007b`b47f8f30 00007ffb`bef450d6 unknown!unknown+0x0
0000007b`b47f8f80 00007ffb`bef43c57 unknown!unknown+0x0
0000007b`b47f9090 00007ffb`bf7ba4e7 unknown!unknown+0x0
0000007b`b47f90f0 00007ffb`bef439a1 unknown!unknown+0x0
0000007b`b47f9160 00007ffb`bf83c75f unknown!unknown+0x0
0000007b`b47f91e0 00007ffb`bef43432 unknown!unknown+0x0
0000007b`b47f9250 00007ffb`bff0cc1c unknown!unknown+0x0
0000007b`b47f92c0 00007ffb`bff0bfa7 unknown!unknown+0x0
0000007b`b47f9340 00007ffb`bff0b711 unknown!unknown+0x0
0000007b`b47f93f0 00007ffb`bff0b0dc unknown!unknown+0x0
0000007b`b47f9460 00007ffb`bff0afb7 unknown!unknown+0x0
0000007b`b47f94d0 00007ffb`bef40286 unknown!unknown+0x0
0000007b`b47f9560 00007ffb`bef3fee9 unknown!unknown+0x0
0000007b`b47f9610 00007ffb`bef3fe05 unknown!unknown+0x0
0000007b`b47f9650 00007ffb`bef3fd70 unknown!unknown+0x0
0000007b`b47f96f0 00007ffb`be5e34bc unknown!unknown+0x0
0000007b`b47f9890 00007ffb`be5e2d79 unknown!unknown+0x0
0000007b`b47f9940 00007ffb`be5e2c95 unknown!noop+0x0
0000007b`b47f9980 00007ffb`be5e2bfd unknown!unknown+0x0
0000007b`b47f9a10 00007ffb`be5e26ca unknown!unknown+0x0
0000007b`b47f9ac0 00007ffb`be5e1f92 unknown!noop+0x0
0000007b`b47f9b40 00007ffb`be5e1c79 unknown!unknown+0x0
0000007b`b47f9bf0 00007ffb`be5e1785 unknown!unknown+0x0
0000007b`b47f9c30 00007ffb`be5e16f0 unknown!unknown+0x0
0000007b`b47f9cd0 00007ffb`be5dfe4d unknown!unknown+0x0
0000007b`b47f9e70 00007ffb`be5dfb09 unknown!unknown+0x0
0000007b`b47f9f20 00007ffb`be5dfa25 unknown!unknown+0x0
0000007b`b47f9f60 00007ffb`be5df98d unknown!unknown+0x0
0000007b`b47f9ff0 00007ffb`be5df0de unknown!unknown+0x0
0000007b`b47fa080 00007ffb`be5dd6a1 unknown!unknown+0x0
0000007b`b47fa0f0 00007ffb`bf734859 unknown!unknown+0x0
0000007b`b47fa150 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fa1d0 00007ffb`bf7345ee unknown!unknown+0x0
0000007b`b47fa280 00007ffb`bf734499 unknown!unknown+0x0
0000007b`b47fa2b0 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fa310 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fa3e0 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fa420 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fa470 00007ffc`1be27e31 System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[System.__Canon].SetResult+0x71
0000007b`b47fa4b0 00007ffb`be5e01c3 unknown!unknown+0x0
0000007b`b47fa650 00007ffb`bf733c19 unknown!unknown+0x0
0000007b`b47fa6b0 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fa730 00007ffb`bf7339be unknown!unknown+0x0
0000007b`b47fa7e0 00007ffb`bf733869 unknown!unknown+0x0
0000007b`b47fa810 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fa870 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fa940 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fa980 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fa9d0 00007ffb`be5e21ab unknown!unknown+0x0
0000007b`b47faa50 00007ffb`bf733829 unknown!unknown+0x0
0000007b`b47faab0 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fab30 00007ffb`bf7335ce unknown!unknown+0x0
0000007b`b47fabe0 00007ffb`bf733479 unknown!unknown+0x0
0000007b`b47fac10 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fac70 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fad40 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fad80 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fadd0 00007ffc`1be27e31 System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[System.__Canon].SetResult+0x71
0000007b`b47fae10 00007ffb`be5e3832 unknown!unknown+0x0
0000007b`b47fafb0 00007ffb`bf732519 unknown!unknown+0x0
0000007b`b47fb010 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fb090 00007ffb`bf7322be unknown!unknown+0x0
0000007b`b47fb140 00007ffb`bf732169 unknown!unknown+0x0
0000007b`b47fb170 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fb1d0 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fb2a0 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fb2e0 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fb330 00007ffb`bef404a4 unknown!unknown+0x0
0000007b`b47fb3c0 00007ffb`bf35d589 unknown!unknown+0x0
0000007b`b47fb420 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fb4a0 00007ffb`bf35d32e unknown!unknown+0x0
0000007b`b47fb550 00007ffb`bf35d1d9 unknown!unknown+0x0
0000007b`b47fb580 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fb5e0 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fb6b0 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fb6f0 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fb740 00007ffc`1be27e31 System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[System.__Canon].SetResult+0x71
0000007b`b47fb780 00007ffb`bef4287e unknown!unknown+0x0
0000007b`b47fb8e0 00007ffb`bf35d199 unknown!unknown+0x0
0000007b`b47fb940 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fb9c0 00007ffb`bf35cf3e unknown!unknown+0x0
0000007b`b47fba70 00007ffb`bf35cde9 unknown!unknown+0x0
0000007b`b47fbaa0 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fbb00 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fbbd0 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fbc10 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fbc60 00007ffb`bef43650 unknown!unknown+0x0
0000007b`b47fbcd0 00007ffb`bf35cda9 unknown!unknown+0x0
0000007b`b47fbd30 00007ffc`1bcd8a36 System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x76
0000007b`b47fbdb0 00007ffb`bf35cb4e unknown!unknown+0x0
0000007b`b47fbe60 00007ffb`bf35c9f9 unknown!unknown+0x0
0000007b`b47fbe90 00007ffc`1bcf6da6 System_Private_CoreLib!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction+0x56
0000007b`b47fbef0 00007ffc`1bcdc79d System_Private_CoreLib!System.Threading.Tasks.Task.RunContinuations+0x11d
0000007b`b47fbfc0 00007ffc`1bcdc67d System_Private_CoreLib!System.Threading.Tasks.Task.FinishContinuations+0x3d
0000007b`b47fc000 00007ffc`1bdcb82f System_Private_CoreLib!System.Threading.Tasks.Task`1[System.__Canon].TrySetResult+0x7f
0000007b`b47fc050 00007ffc`1be27e31 System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[System.__Canon].SetResult+0x71
0000007b`b47fc090 00007ffb`bef43da8 unknown!unknown+0x0
0000007b`b47fc1a0 00007ffc`1be281a8 System_Private_CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncS

I will open a separate issue for this and will track this bug there down further, because we need to be able to reliably run our tests in CI without non-deterministic exceptions.

Originally posted by @lauxjpn in dotnet/runtime#33899 (comment)

@lauxjpn , @FreddyD-GH the Jet/ACE provider problem could be related to a specific version.
As far as I remember it was so hard to reproduce it in a deterministic way that I could not understand.
I don't know Microsoft behaviour in these cases but probably we could try to understand if the issue is version specific then try to post the issue to Microsoft. Not sure they will listen us.

We could also evaluate another solution. There are some tests, about 80 tests, that are integration tests. In my mind this was the minimal set of tests that must run before a release. We could include these tests (and some other MUST run tests) in CI pipeline and run manually other tests.

Originally posted by @bubibubi in dotnet/runtime#33899 (comment)

@bubibubi Yes, we should narrow down the circumstances of this bug.
There is no alternative to the 15K EF Core tests in my opinion. The 80 historic integration tests don't even scratch the surface of what needs to be covered.

But it seems you are implying, that the 80 will always run without access violation exception. Is that correct, or do they don't trigger the exception only because they are just so few tests?

Originally posted by @lauxjpn in dotnet/runtime#33899 (comment)

@lauxjpn I think that the issues on ACE/Jet are related to some cirumnstances:

1. Transactions
2. Multithreading
3. SQL Errors but it's very hard to reproduce (for example select from a table that does not exists or select a field that does not exists). As far as I understood these are more frequent on Jet provider (Version 4.x) but I'm not sure. They just happen.

Also, if I run inside Visual Studio I had a behaviour, if I run with msbuild I had another behaviour. Outside Visual Studio the behaviour is usually better. This is related to GC but adding GC.Collect and GC.WaitForPendingFinalizers does not help.

The 80 tests just works always. It does not matter the order they run.

Originally posted by @bubibubi in dotnet/runtime#33899 (comment)

[FreddyDgh]

I'm wondering if Marshal.AreComObjectsAvailableForCleanup would be helpful in this situation.

I've never had to use this method before, but I think the documentation suggests that you'd use it like this:

while(Marshal.AreComObjectsAvailableForCleanup())
{
     GC.Collect();
     GC.WaitForPendingFinalizers();
}

I believe @bubibubi said he is already using GC.Collect(); and GC.WaitForPendingFinalizers(); between (at least some of) the test runs. @lauxjpn Maybe it's worth adding the code above between test runs to see if you notice any difference?

Even if it works, this seems like it's still not ideal, but might be good enough for now.

[lauxjpn]

I will dig deeper into this issue, now that the general upgrade is done.

First, I am going to build a scenario that is simpler and more flexible to debug than the xUnit test suit, and that is able to trigger the exception in a more reliable manner.

Then I will try the Marshal.AreComObjectsAvailableForCleanup() suggestion and see if that has an effect

Finally, I will take a look at the processor instructions to find out what the ACE database driver expects and what it actually gets.

[lauxjpn]

I created the smallest possible sample to reproduce the AccessViolationException.

• The exception will only be thrown when run as an x64 process.
• The exception should be thrown at least once in every 5 app executions.
• I tested with Microsoft.ACE.OLEDB.16.0 and Microsoft.ACE.OLEDB.12.0. Both will throw the exception (v16 usually very early, when still below 10 iterations; v12 usually after around 50 iterations).
• The exception will only be thrown, when a UNION query as been executed and later a SELECT query over the same table as the UNION query is executed.
• This second SELECT query must return only scalar values (no column reference).
• It doesn't matter, what the scalar values are (e.g. 0, 1, NULL).
• The count of scalar values of the second SELECT query must be at least as high as the count of returned fields in the previous UNION query.
• It does not matter, if another query is executed in between.
• In case there has been no AccessViolationException within the first 100 interations, it is very unlikely that they will ever appear within the lifetime of the process. (Though sometimes one is thrown near the 50K mark.)

A couple of examples that will throw:

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, NULL
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, NULL
FROM [Customers] AS [c];

A couple of examples that will not throw:

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT [c].[CustomerID] /* <-- not a scalar value */
FROM [Customers] AS [c];

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'; /* <-- not a UNION */

SELECT 1
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1 /* <-- fewer scalar values than fields in UNION */
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, [c].[CustomerID] /* <-- not just scalar values */
FROM [Customers] AS [c];

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT *
FROM [Orders] AS [o];  /* <-- table differs from the one used in the UNION */

The following code can be used to reproduce the issue. The code should be run with the latest nightly-build of System.Data.OleDb:

using System;
using System.Data.OleDb;

namespace JetProviderExceptionTests
{
    internal static class Program
    {
        private static void Main()
        {
            Console.WriteLine($"Running as {(Environment.Is64BitProcess ? "x64" : "x86")} process.");
            
            var tagDBPARAMBINDINFOName = "tagDBPARAMBINDINFO" + (Environment.Is64BitProcess
                ? string.Empty
                : "_x86");
            
            // Is 2 on x86 and 8 on x64.
            Console.WriteLine($"{tagDBPARAMBINDINFOName} field alignment is {Type.GetType($"System.Data.OleDb.{tagDBPARAMBINDINFOName}, System.Data.OleDb").StructLayoutAttribute.Pack}.");
            
            // Is 8 on both x86 and x64.
            Console.WriteLine($"tagDBPARAMS field alignment is {Type.GetType("System.Data.OleDb.tagDBPARAMS, System.Data.OleDb").StructLayoutAttribute.Pack}.");

            Console.WriteLine();
            Console.WriteLine("Press any key to start...");
            Console.ReadKey();

            RunNorthwindTest();
        }

        private static void RunNorthwindTest()
        {
            try
            {
                using var connection = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.16.0;Data Source=Northwind.accdb");
                connection.Open();

                for (var i = 0; i < 1000; i++)
                {
                    Console.WriteLine($"{i:000}");

                    //
                    // Select_Union:
                    //

                    using (var command1 = connection.CreateCommand())
                    {
                        command1.CommandText = @"SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London'";

                        using (var dataReader1 = command1.ExecuteReader())
                        {
                            while (dataReader1.Read())
                            {
                            }
                        }
                    }

                    /*
                    using (var command15 = connection.CreateCommand())
                    {
                        command15.CommandText = @"SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Madrid'";

                        using (var dataReader15 = command15.ExecuteReader())
                        {
                            while (dataReader15.Read())
                            {
                            }
                        }
                    }
                    */
                    
                    //
                    // Select_bool_closure:
                    //

                    using (var command2 = connection.CreateCommand())
                    {
                        command2.CommandText = @"SELECT 1
FROM [Customers] AS [c]";

                        using (var dataReader2 = command2.ExecuteReader())
                        {
                            while (dataReader2.Read())
                            {
                            }
                        }
                    }
                }
            }
            catch (AccessViolationException e)
            {
                Console.WriteLine(e);
            }
        }
    }
}

The code uses the version of the Northwind database we are also now using in our functional tests:
Northwind.zip

There are no transactions and no multi-threading involved to trigger the exception.

/cc @saurabh500

[FreddyDgh]

@lauxjpn Is this on .NET Core only? Or does it happen with .NET Framework as well?

[lauxjpn]

@FreddyD-GH The exception is being thrown in a .Net Framework 4.7.2 project using the .NET Framwork version of OLE DB and running in a 64 bit process as well:

Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Data.Common.UnsafeNativeMethods.ICommandText.Execute(IntPtr pUnkOuter, Guid& riid, tagDBPARAMS pDBParams, IntPtr& pcRowsAffected, Object& ppRowset)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
   at System.Data.OleDb.OleDbCommand.ExecuteReader(CommandBehavior behavior)
   at JetProviderExceptionTests.NorthwindTestOleDbCommand.Run() in E:\Sources\JetProviderExceptionTests\ConsoleApplication\NorthwindTestOleDbCommand.cs:line 67
   at JetProviderExceptionTests.Program.Main() in E:\Sources\JetProviderExceptionTests\ConsoleApplication\Program.cs:line 26

The same test works fine under .Net Framework 4.7.2 (x64) when using ODBC and the Microsoft Access Driver (*.mdb, *.accdb) (tested with version 16) by the way.

@saurabh500 With Access nowadays preferring DAO again over ADO, is OLE DB still the right choice for maximum compatibility or should we switch to ODBC?

We could also support both and let the user decide (with a reasonable default; probably ODBC as it looks right now).

[FreddyDgh]

@lauxjpn FYI, I tried the code you provided and was able to reproduce the behavior. I've seen a good number of cases like this where having resources that aren't cleaned up properly is the culprit, but that doesn't appear to apply in this situation (at least not on the managed side). I didn't get a chance to really dig into it, though.

As far as ODBC, I ended up trying that on a .NET Core project I had when I initially had the issues with OleDb x86. I can't remember the specific issues, but I remember not getting any further with ODBC than I did with OleDb (and OleDb x86 on .NET Core is broken AF). I think I remember ODBC was not as buggy as OleDb, but instead I think it gave me much nicer errors that it didn't support certain things. I didn't spend much time trying to find workarounds, though. Regardless, ODBC may be worth looking into, but my memory says not to get your hopes up.

[lauxjpn]

@FreddyD-GH I will test a parallel implementation, where people can choose which of the two technologies to choose as their base. Most of the code has already been abstracted away by @bubibubi, so the implementation cost should be relatively low.
Assuming we strife for SemVer compliance, the current major version upgrade would be the ideal point in time to change the default underlying technology, while offering backward compatibility for users who have heavily invested in OLE DB.

[bubibubi]

As far as I remember we can't avoid to use OleDb because we risk to loose code from DB features.
Microsoft Access does not have (sorry, has few) system tables so we need to retrieve data structures using OleDb or ODBC features (or ADOX but is not always installed and works only on 32 bits am I right?). Probably with ODBC is not possible at all to retrieve tables list.
About parameters, another issue, we could solve it in JetCommand. I think I already did something because also Jet/ACE "does not love" named parameters. Anyway, we need to keep in mind also that ODBC does not support named parameters.

[lauxjpn]

Microsoft Access does not have (sorry, has few) system tables so we need to retrieve data structures using OleDb or ODBC features (or ADOX but is not always installed and works only on 32 bits am I right?). Probably with ODBC is not possible at all.

I will take a closer look at that. Generally, the GetSchema method is available for ODBC as well, but I will check its implementation.

we need to keep in mind also that ODBC does not support named parameters.
That is a good point.

[bubibubi]

@lauxjpn I updated the comment sorry. I think that with ODBC we can't retrieve the tables list.

[lauxjpn]

@bubibubi No Problem. I will take a closer look at this, but I am very confident, that we will be able to get the table list from ODBC. If this should turn out to be not possible for some reason, we can always get it directly from the MSysObjects table.

ADOX but is not always installed and works only on 32 bits am I right?

ADOX is part of the Windows DAC, a modern version of the traditional MDAC, that ships as part of windows. You can find the x86 version of ADOX in C:\Program Files (x86)\Common Files\System\ado\msadox.dll and the x64 version of ADOX in C:\Program Files\Common Files\System\ado\msadox.dll.