CheckProjects test failing

[MichalStrehovsky]

Seen this in #89421.

    JIT\CheckProjects\CheckProjects\CheckProjects.cmd [FAIL]
      Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
         at CheckProjects!<BaseAddress>+0x440a4b
      
      Return code:      1
      Raw output file:      C:\h\w\ABA80982\w\A7770A09\uploads\Reports\JIT.CheckProjects\CheckProjects\CheckProjects.output.txt

I grabbed the binaries and dump with runfo get-helix-payload -j 454b9dd6-0743-4a6b-8af8-5fc3d19c70e7 -w JIT.1 -o c:\hell.

It's crashing super early in startup. The crash reproduces when I run the binary downloaded from runfo locally too.

00007FF752020A33 48 8D 0D 46 36 1C 00 lea         rcx,[CheckProjects__Module_::`vftable' (07FF7521E4080h)]  
00007FF752020A3A E8 A1 2A EA FF       call        S_P_CoreLib_Internal_Runtime_CompilerHelpers_LdTokenHelpers__GetRuntimeTypeHandle (07FF751EC34E0h)  
00007FF752020A3F 48 89 05 A2 BB 47 00 mov         qword ptr [S_P_CoreLib_Internal_Runtime_CompilerHelpers_StartupCodeHelpers::__NONGCSTATICS (07FF75249C5E8h)],rax  
00007FF752020A46 E8 F5 30 BC FF       call        __GetThreadStaticBase_S_P_CoreLib_System_Threading_Thread (07FF751BE3B40h)  

00007FF752020A4B 48 8B 48 68          mov         rcx,qword ptr [rax+68h]  
                                                                ^ ****** Nullref here because rax was 0 ******

00007FF752020A4F 48 85 C9             test        rcx,rcx  
00007FF752020A52 75 08                jne         CheckProjects__Module___StartupCodeMain+0BCh (07FF752020A5Ch)  
00007FF752020A54 E8 D7 FA E2 FF       call        S_P_CoreLib_System_Threading_Thread__InitializeCurrentThread (07FF751E50530h)  

I tried to trace through it but threadstatics are now super confusing so I'm leaving it to someone else to untangle.

One suspicious thing is that the threadstatic accessors look odd:

CheckProjects.exe!__GetThreadStaticBase_S_P_CoreLib_System_Runtime_InteropServices_PInvokeMarshal(void):
00007FF751BE3D63 8B 0D 07 12 8D 00    mov         ecx,dword ptr [_tls_index (07FF7524B4F70h)]  
00007FF751BE3D69 65 48 8B 04 25 58 00 00 00 mov         rax,qword ptr gs:[58h]  
00007FF751BE3D72 48 8B 04 C8          mov         rax,qword ptr [rax+rcx*8]  
00007FF751BE3D76 B9 10 00 00 00       mov         ecx,10h  
00007FF751BE3D7B 48 01 C1             add         rcx,rax  
00007FF751BE3D7E 48 8B 01             mov         rax,qword ptr [rcx]  
00007FF751BE3D81 48 85 C0             test        rax,rax  
00007FF751BE3D84 0F 84 A6 BC 2D 00    je          S_P_CoreLib_Internal_Runtime_ThreadStatics__GetInlinedThreadStaticBaseSlow (07FF751EBFA30h)  
00007FF751BE3D8A C3                   ret  
CheckProjects.exe!__GetThreadStaticBase_S_P_CoreLib_Internal_Runtime_ThreadStatics(void):
00007FF751BE3D8B 4C 8D 05 DE 2A 52 00 lea         r8,[S_P_CoreLib_Internal_Runtime_ThreadStatics::__THREADSTATICINDEX (07FF752106870h)]  
00007FF751BE3D92 49 8B 08             mov         rcx,qword ptr [r8]  
00007FF751BE3D95 41 8B 50 08          mov         edx,dword ptr [r8+8]  
00007FF751BE3D99 E9 72 BC 2D 00       jmp         S_P_CoreLib_Internal_Runtime_ThreadStatics__GetThreadStaticBaseForType (07FF751EBFA10h)  
CheckProjects.exe!__GetThreadStaticBase_S_P_CoreLib_System_Random_ThreadSafeRandom(void):
00007FF751BE3D9E 8B 0D CC 11 8D 00    mov         ecx,dword ptr [_tls_index (07FF7524B4F70h)]  
00007FF751BE3DA4 65 48 8B 04 25 58 00 00 00 mov         rax,qword ptr gs:[58h]  
00007FF751BE3DAD 48 8B 04 C8          mov         rax,qword ptr [rax+rcx*8]  
00007FF751BE3DB1 B9 10 00 00 00       mov         ecx,10h  
00007FF751BE3DB6 48 01 C1             add         rcx,rax  
00007FF751BE3DB9 48 8B 01             mov         rax,qword ptr [rcx]  
00007FF751BE3DBC 48 85 C0             test        rax,rax  
00007FF751BE3DBF 0F 84 6B BC 2D 00    je          S_P_CoreLib_Internal_Runtime_ThreadStatics__GetInlinedThreadStaticBaseSlow (07FF751EBFA30h)  
00007FF751BE3DC5 C3                   ret  

Notice the first and third are on inlined plan, the middle is on non-inlined plan. I don't know if that's related. But I would not expect it.

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Seen this in #89421.

    JIT\CheckProjects\CheckProjects\CheckProjects.cmd [FAIL]
      Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
         at CheckProjects!<BaseAddress>+0x440a4b
      
      Return code:      1
      Raw output file:      C:\h\w\ABA80982\w\A7770A09\uploads\Reports\JIT.CheckProjects\CheckProjects\CheckProjects.output.txt

I grabbed the binaries and dump with runfo get-helix-payload -j 454b9dd6-0743-4a6b-8af8-5fc3d19c70e7 -w JIT.1 -o c:\hell.

It's crashing super early in startup. The crash reproduces when I run the binary downloaded from runfo locally too.

00007FF752020A33 48 8D 0D 46 36 1C 00 lea         rcx,[CheckProjects__Module_::`vftable' (07FF7521E4080h)]  
00007FF752020A3A E8 A1 2A EA FF       call        S_P_CoreLib_Internal_Runtime_CompilerHelpers_LdTokenHelpers__GetRuntimeTypeHandle (07FF751EC34E0h)  
00007FF752020A3F 48 89 05 A2 BB 47 00 mov         qword ptr [S_P_CoreLib_Internal_Runtime_CompilerHelpers_StartupCodeHelpers::__NONGCSTATICS (07FF75249C5E8h)],rax  
00007FF752020A46 E8 F5 30 BC FF       call        __GetThreadStaticBase_S_P_CoreLib_System_Threading_Thread (07FF751BE3B40h)  

00007FF752020A4B 48 8B 48 68          mov         rcx,qword ptr [rax+68h]  
                                                                ^ ****** Nullref here because rax was 0 ******

00007FF752020A4F 48 85 C9             test        rcx,rcx  
00007FF752020A52 75 08                jne         CheckProjects__Module___StartupCodeMain+0BCh (07FF752020A5Ch)  
00007FF752020A54 E8 D7 FA E2 FF       call        S_P_CoreLib_System_Threading_Thread__InitializeCurrentThread (07FF751E50530h)  

I tried to trace through it but threadstatics are now super confusing so I'm leaving it to someone else to untangle.

One suspicious thing is that the threadstatic accessors look odd:

CheckProjects.exe!__GetThreadStaticBase_S_P_CoreLib_System_Runtime_InteropServices_PInvokeMarshal(void):
00007FF751BE3D63 8B 0D 07 12 8D 00    mov         ecx,dword ptr [_tls_index (07FF7524B4F70h)]  
00007FF751BE3D69 65 48 8B 04 25 58 00 00 00 mov         rax,qword ptr gs:[58h]  
00007FF751BE3D72 48 8B 04 C8          mov         rax,qword ptr [rax+rcx*8]  
00007FF751BE3D76 B9 10 00 00 00       mov         ecx,10h  
00007FF751BE3D7B 48 01 C1             add         rcx,rax  
00007FF751BE3D7E 48 8B 01             mov         rax,qword ptr [rcx]  
00007FF751BE3D81 48 85 C0             test        rax,rax  
00007FF751BE3D84 0F 84 A6 BC 2D 00    je          S_P_CoreLib_Internal_Runtime_ThreadStatics__GetInlinedThreadStaticBaseSlow (07FF751EBFA30h)  
00007FF751BE3D8A C3                   ret  
CheckProjects.exe!__GetThreadStaticBase_S_P_CoreLib_Internal_Runtime_ThreadStatics(void):
00007FF751BE3D8B 4C 8D 05 DE 2A 52 00 lea         r8,[S_P_CoreLib_Internal_Runtime_ThreadStatics::__THREADSTATICINDEX (07FF752106870h)]  
00007FF751BE3D92 49 8B 08             mov         rcx,qword ptr [r8]  
00007FF751BE3D95 41 8B 50 08          mov         edx,dword ptr [r8+8]  
00007FF751BE3D99 E9 72 BC 2D 00       jmp         S_P_CoreLib_Internal_Runtime_ThreadStatics__GetThreadStaticBaseForType (07FF751EBFA10h)  
CheckProjects.exe!__GetThreadStaticBase_S_P_CoreLib_System_Random_ThreadSafeRandom(void):
00007FF751BE3D9E 8B 0D CC 11 8D 00    mov         ecx,dword ptr [_tls_index (07FF7524B4F70h)]  
00007FF751BE3DA4 65 48 8B 04 25 58 00 00 00 mov         rax,qword ptr gs:[58h]  
00007FF751BE3DAD 48 8B 04 C8          mov         rax,qword ptr [rax+rcx*8]  
00007FF751BE3DB1 B9 10 00 00 00       mov         ecx,10h  
00007FF751BE3DB6 48 01 C1             add         rcx,rax  
00007FF751BE3DB9 48 8B 01             mov         rax,qword ptr [rcx]  
00007FF751BE3DBC 48 85 C0             test        rax,rax  
00007FF751BE3DBF 0F 84 6B BC 2D 00    je          S_P_CoreLib_Internal_Runtime_ThreadStatics__GetInlinedThreadStaticBaseSlow (07FF751EBFA30h)  
00007FF751BE3DC5 C3                   ret  

Notice the first and third are on inlined plan, the middle is on non-inlined plan. I don't know if that's related. But I would not expect it.

Author:	MichalStrehovsky
Assignees:	VSadov
Labels:	area-NativeAOT-coreclr
Milestone:	8.0.0

[VSadov]

Notice the first and third are on inlined plan, the middle is on non-inlined plan. I don't know if that's related. But I would not expect it.

Is this in checked build?

In checked build we force some random threadstatics to be on slow plan - just to exercise the code path. Forcing something to not be inlined should be ok.

That was correct assumption until ThreadStatics.t_inlinedThreadStaticBase started caching the inlined static base. Such caching benefits whoever calls GetThreadStaticBaseForType directly. Most commonly it is the CheckStaticClassConstructionReturnThreadStaticBase. It makes the code simpler too.

Such caching though indirectly assumes that t_inlinedThreadStaticBase itself always participates in inlining.
(since the cache entry is inlined, accessing the cache will guarantee initialization of inline storage and will initialize the cache)

In your case it looks like ThreadStatics.t_inlinedThreadStaticBase itself was unlucky to be bumped off the inlined plan. Accessing it would no longer initialize the cache. As a result - If the cache is accessed very early (before anyone else accessed inlined storage) the cache returns NULL

[VSadov]

The "forcing off inlining plan" makes much less sense now that we have a compiler switch that disables inlining and have entire platforms/configurations that do not inline. The slow path is certainly exercised.

I think the "forcing off inlining plan" hack should be removed.