Test failures: Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests

[BruceForstall]

Linux/arm64 JitStress

https://dev.azure.com/dnceng-public/public/_build/results?buildId=304266&view=ms.vss-test-web.build-test-results-tab&runId=6175518&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=126035

DOTNET_TieredCompilation=0
DOTNET_JitStress=101

  Starting:    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests (parallel test collections = on, max threads = 2)
...
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T_BinderOptions [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(294,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T_BinderOptions()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(340,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.TestCollectionsGen [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(694,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.TestCollectionsGen()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf_BinderOptions [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(386,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf_BinderOptions()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(202,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(248,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Configure [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(546,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Configure()
        --- End of stack trace from previous location ---
  Finished:    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests

@dotnet/jit-contrib

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Linux/arm64 JitStress

https://dev.azure.com/dnceng-public/public/_build/results?buildId=304266&view=ms.vss-test-web.build-test-results-tab&runId=6175518&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=126035

DOTNET_TieredCompilation=0
DOTNET_JitStress=101

  Starting:    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests (parallel test collections = on, max threads = 2)
...
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T_BinderOptions [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(294,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T_BinderOptions()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(340,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.TestCollectionsGen [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(694,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.TestCollectionsGen()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf_BinderOptions [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(386,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_TypeOf_BinderOptions()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(202,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(248,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Get_T()
        --- End of stack trace from previous location ---
    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Configure [FAIL]
      The collection was expected to contain a single element, but it was empty.
      Stack Trace:
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(724,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.VerifyAgainstBaselineUsingFile(String filename, String testSourceCode, LanguageVersion languageVersion, Action`1 assessDiagnostics)
        /_/src/libraries/Microsoft.Extensions.Configuration.Binder/tests/SourceGenerationTests/ConfigurationBindingGeneratorTests.Baselines.cs(546,0): at Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.ConfigurationBindingGeneratorTests.Configure()
        --- End of stack trace from previous location ---
  Finished:    Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests

@dotnet/jit-contrib

Author:	BruceForstall
Assignees:	-
Labels:	arch-arm64, os-linux, JitStress, area-CodeGen-coreclr
Milestone:	8.0.0

[BruceForstall]

Possibly related: #87506

[markples]

fyi - This same batch of tests (Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests) failed in #85632, which I just started trying to repro. However, that other one failed only once and in a different way, and this batch is a group of tests (old-style, not a newly merged group) so there might not be any connection at all.

[AndyAyersMS]

I can repro -- via binary search on DOTNET_JitStressRange the issue is in this method:

# 8508: JIT compiled Microsoft.Extensions.Configuration.Binder.SourceGeneration.ConfigurationBindingGenerator+Emitter:EmitHelperMethods():this [FullOpts, IL size=227, code size=964, hash=0x1359ac8f JitStress]

Isolating stress modes, this fails with

export DOTNET_JitStressModeNamesOnly=1
export DOTNET_JitStressModeNames="STRESS_BB_PROFILE STRESS_RANDOM_INLINE STRESS_NO_OLD_PROMOTION STRESS_PHYSICAL_PROMOTION"
export DOTNET_JitStressRange=1359ac8f

[AndyAyersMS]

@jakobbotsch PTAL

[jakobbotsch]

Oddly the above does not repro the issue for me... I am bisecting it anew now.

[jakobbotsch]

Looks like for me the failing one is:

JIT compiled Microsoft.Extensions.Configuration.Binder.SourceGeneration.SourceWriter:WriteBlock(System.String):this [FullOpts, IL size=84, code size=340, hash=0xd2ef78d7 JitStress]

From an initial glance the problem is around QMARKs. We need to be more careful around readbacks:

STMT00003 ( 0x007[E-] ... ??? )
               [000008] DAC--------                         *  STORE_LCL_VAR struct<System.ReadOnlySpan`1, 16> V04 loc2         
               [000006] --C-G------                         \--*  CALL      struct Microsoft.Extensions.Configuration.Binder.SourceGeneration.SourceWriter:GetNextLine(byref,byref):System.ReadOnlySpan`1[ushort]
               [000004] ----------- arg0                       +--*  LCL_ADDR  byref  V03 loc1         [+0]
               [000005] ----------- arg1                       \--*  LCL_ADDR  byref  V02 loc0         [+0]
Processing block operation [000008] that involves replacements
Marked store destination replacements to be read back (could not decompose this store)
STMT00004 ( 0x011[E-] ... 0x021 )
               [000011] -----------                         *  NOP       void  
STMT00005 ( 0x011[E-] ... ??? )
               [000026] DA-XG------                         *  STORE_LCL_VAR int    V06 tmp1         
               [000025] ---XG------                         \--*  QMARK     int   
               [000024] -----------    if                      +--*  EQ        int   
               [000014] -----------                            |  +--*  LCL_FLD   int    V04 loc2         [+8]
               [000015] -----------                            |  \--*  CNS_INT   int    1
               [000023] ---XG------    if                      \--*  COLON     int   
               [000022] ----------- else                          +--*  CNS_INT   int    0
               [000021] ---XG------ then                          \--*  EQ        int   
               [000019] ---XG------                                  +--*  IND       ushort
               [000018] -----------                                  |  \--*  ADD       byref 
               [000016] -----------                                  |     +--*  LCL_FLD   byref  V04 loc2         [+0]
               [000017] -----------                                  |     \--*  CNS_INT   long   0
               [000020] -----------                                  \--*  CNS_INT   int    123
Processing use [000014] of V04.[008..012)
  ..replaced with promoted lcl V16
Processing use [000016] of V04.[000..008)
  ..replaced with promoted lcl V15
New statement:
STMT00005 ( 0x011[E-] ... ??? )
               [000026] DA-XG------                         *  STORE_LCL_VAR int    V06 tmp1         
               [000025] -A-XG------                         \--*  QMARK     int   
               [000024] -A---------    if                      +--*  EQ        int   
               [000174] -A---------                            |  +--*  COMMA     int   
               [000173] DA---------                            |  |  +--*  STORE_LCL_VAR int    V16 tmp11        
               [000172] -----------                            |  |  |  \--*  LCL_FLD   int    V04 loc2         [+8]
               [000171] -----------                            |  |  \--*  LCL_VAR   int    V16 tmp11        
               [000015] -----------                            |  \--*  CNS_INT   int    1
               [000023] -A-XG------    if                      \--*  COLON     int   
               [000022] ----------- else                          +--*  CNS_INT   int    0
               [000021] -A-XG------ then                          \--*  EQ        int   
               [000019] -A-XG------                                  +--*  IND       ushort
               [000018] -A---------                                  |  \--*  ADD       byref 
               [000178] -A---------                                  |     +--*  COMMA     byref 
               [000177] DA---------                                  |     |  +--*  STORE_LCL_VAR byref  V15 tmp10        
               [000176] -----------                                  |     |  |  \--*  LCL_FLD   byref  V04 loc2         [+0]
               [000175] -----------                                  |     |  \--*  LCL_VAR   byref  V15 tmp10        
               [000017] -----------                                  |     \--*  CNS_INT   long   0
               [000020] -----------                                  \--*  CNS_INT   int    123

Reading V15 back in the arm of a QMARK should either not mark it as readback (easiest), or we should eagerly do these readbacks before the QMARK (harder, more efficient).

@AndyAyersMS Do you still expect to work on #86778?