Stack overflow in F# compiler. on net5.0

[KevinRansom]

With dotnet 5.0 we have seen a new stack overflow introduced in the F# compiler when running on the coreclr.

Repro:
unzip StackOverflow.zip
to your local zip.

The project assumes that a dotnet sdk with net5.0 is installed in c:\program files\dotnet

build this project and run it

dotnet build -t:Rebuild -c release
dotnet run -c release

observe the stack overflow.

If the project file is edited to netcoreapp3.1 and a netcoreapp3.1 runtime is installed, then executing the program will succeed without a stack overflow.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AndyAyersMS]

Thanks, this repros. Partial stack trace:

   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Microsoft.FSharp.Core.FSharpFunc`2<Results,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Results>>, Results, System.__Canon)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].exprNoInterceptF(System.__Canon, Expr)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Microsoft.FSharp.Core.FSharpFunc`2<Results,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Results>>, Results, System.__Canon)

6.0 seems to hang or is very slow? It is perhaps trying to serialize out this stack trace somewhere.

[AndyAyersMS]

Still trying to sort out what goes wrong... the repeating pattern on the stack is 36 frames long:

   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprNoInterceptF(System.__Canon, Expr)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   ** at FSharp.Compiler.Detuple+GlobalUsageAnalysis+targetIntercept@291.Invoke(FRE, Results, DecisionTreeTarget)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].targetF(System.__Canon, DecisionTreeTarget)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   ** at FSharp.Compiler.Detuple+GlobalUsageAnalysis+targetIntercept@291.Invoke(FRE, Results, DecisionTreeTarget)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].targetF(System.__Canon, DecisionTreeTarget)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   ** at FSharp.Compiler.Detuple+GlobalUsageAnalysis+targetIntercept@291.Invoke(FRE, Results, DecisionTreeTarget)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].targetF(System.__Canon, DecisionTreeTarget)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   ** at FSharp.Compiler.Detuple+GlobalUsageAnalysis+targetIntercept@291.Invoke(FRE, Results, DecisionTreeTarget)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].targetF(System.__Canon, DecisionTreeTarget)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   ** at FSharp.Compiler.Detuple+GlobalUsageAnalysis+targetIntercept@291.Invoke(FRE, Results, DecisionTreeTarget)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].targetF(System.__Canon, DecisionTreeTarget)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   ** at FSharp.Compiler.Detuple+GlobalUsageAnalysis+targetIntercept@291.Invoke(FRE, Results, DecisionTreeTarget)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].targetF(System.__Canon, DecisionTreeTarget)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)
   at FSharp.Compiler.TypedTreeOps+ExprFolders`1[[SPC]].exprF(System.__Canon, Expr)
   at FSharp.Compiler.Detuple+GlobalUsageAnalysis.foldUnderLambda[[SPC]](FRR, Results, System.__Canon)
   at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr, Void (IntPtr, IntPtr, IntPtr*), IntPtr)

where

SPC = corelib
FRR = Microsoft.FSharp.Core.FSharpFunc`2<Results,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Results>>
FRE = Microsoft.FSharp.Core.FSharpFunc`2<Results,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Results>>

My guess based on some of the comments over in #40864 is that the delegate invokes (** in above) could be a factor.

[AndyAyersMS]

cc @jkotas @jakobbotsch @janvorli

[KevinRansom]

Ouch!!! I had a similar issue trying to figure it out from the f# source. Everything looked idiomatic and normal.

[AndyAyersMS]

Delegate invoke codegen for 5.0.

; targetIntercept@291:Invoke

; Emitting BLENDED_CODE for X64 CPU with AVX - Windows
; Tier-0 compilation
; MinOpts code
; rbp based frame
; partially interruptible
; Final local variable assignments
;
;  V00 this         [V00    ] (  1,  1   )     ref  ->  [rbp+0x10]   this class-hnd
;  V01 arg1         [V01    ] (  1,  1   )     ref  ->  [rbp+0x18]   class-hnd
;  V02 arg2         [V02    ] (  1,  1   )     ref  ->  [rbp+0x20]   class-hnd
;  V03 arg3         [V03    ] (  1,  1   )     ref  ->  [rbp+0x28]   class-hnd
;  V04 loc0         [V04    ] (  1,  1   )     ref  ->  [rbp-0x08]   must-init class-hnd
;  V05 OutArgs      [V05    ] (  1,  1   )  lclBlk (32) [rsp+0x00]   "OutgoingArgSpace"
;  V06 tmp1         [V06    ] (  1,  1   )     ref  ->  [rbp-0x10]   must-init "argument with side effect"
;
; Lcl frame size = 48

G_M31457_IG01:              ;; offset=0000H
       55                   push     rbp
       4883EC30             sub      rsp, 48
       488D6C2430           lea      rbp, [rsp+30H]
       33C0                 xor      rax, rax
       488945F8             mov      qword ptr [rbp-08H], rax
       488945F0             mov      qword ptr [rbp-10H], rax
       48894D10             mov      gword ptr [rbp+10H], rcx
       48895518             mov      gword ptr [rbp+18H], rdx
       4C894520             mov      gword ptr [rbp+20H], r8
       4C894D28             mov      gword ptr [rbp+28H], r9
						;; bbWeight=1    PerfScore 8.00
G_M31457_IG02:              ;; offset=0024H
       4C8B4D28             mov      r9, gword ptr [rbp+28H]
       4D8B4910             mov      r9, gword ptr [r9+16]
       4C894DF8             mov      gword ptr [rbp-08H], r9
       4C8B4DF8             mov      r9, gword ptr [rbp-08H]
       488B5518             mov      rdx, gword ptr [rbp+18H]
       4C8B4520             mov      r8, gword ptr [rbp+20H]
       48B9A05FB88DFE7F0000 mov      rcx, 0x7FFE8DB85FA0
       E8B5E7FFFF           call     GlobalUsageAnalysis:foldUnderLambda(Microsoft.FSharp.Core.FSharpFunc`2[__Canon,__Canon],Results,System.__Canon):Results
       488945F0             mov      gword ptr [rbp-10H], rax
       488B55F0             mov      rdx, gword ptr [rbp-10H]
       48B99897B58DFE7F0000 mov      rcx, 0x7FFE8DB59798
       E84EF296FC           call     Microsoft.FSharp.Core.FSharpOption`1[__Canon][System.__Canon]:Some(System.__Canon):Microsoft.FSharp.Core.FSharpOption`1[__Canon]
       90                   nop      
						;; bbWeight=1    PerfScore 11.75
G_M31457_IG03:              ;; offset=0063H
       488D6500             lea      rsp, [rbp]
       5D                   pop      rbp
       C3                   ret      
						;; bbWeight=1    PerfScore 2.00

; Total bytes of code 105, prolog size 20, PerfScore 32.25, instruction count 26, allocated bytes for code 105 (MethodHash=cd8c851e) for method targetIntercept@291:Invoke

[AndyAyersMS]

3.1 has identical codegen here, so perhaps I'm off track.

5.0 also overfflows with TC disabled, so it's not some sort of timing thing where Tier0 frames are larger and 5.0 is taking longer to swap over to Tier1.

[jkotas]

Note that the repro stackoverflows on .NET Framework 4.8. I suspect that the regression is caused by fixed stack consumption: some frames got smaller from netfx4.8 to netcoreapp3.1 that made it work, and some frames got bigger from netcoreapp3.1 to net5.0 that broke it again.

[AndyAyersMS]

All the user methods in the above 36 frame chunk have identical frame sizes in 3.1 and 5.0. So could be other methods above or below this that are the culprit, or the dispatcher frames. Dispatcher frame is 96 bytes.

Iff it's other methods it suggests the 3.1 version should be close to overflowing. Running 3.1 under windb, !etb at process exit shows only 0x1B000 stack extent. Doesn't seem likely the dispatcher frames could add up to hundreds of KB. So still puzzled as to where the extra size is coming from.

There are also some odd debugger messages I don't recognize:

StackOverflow Information: 0 : Reactor: 44,176.134 enqueue Unknown.Compile (), length 0
StackOverflow Information: 0 : Reactor: 44,176.509 --> Unknown.Compile (), remaining 0
StackOverflow Information: 0 : Reactor: 44,284.273 BAD-OP: >10s <-- Unknown.Compile, took 107757.4633 ms

So maybe something in the process is trying to sense potential overflow and adapt?

With 5.0 the stack extent at overflow via !teb is 0x17F000 which jives with the stack reserve in the PE header

          180000 size of stack reserve

[AndyAyersMS]

StackOverflow Information ... looks like this first bit is coming from the app name, so not indicative of some sort internal stack size monitoring.

[AndyAyersMS]

Hmm, maybe in 3.1 this computation is not on the main thread.

[AndyAyersMS]

Not sure the quickest way to figure max stack extent on a random thread in a process that doesn't crash, but it looks like the worker thread in 3.1 has max stack is somewhere between 0x100000 and 0x140000 (measured by setting data breakpoints in windbg). That is compatible with the idea that the extra overhead from the new tail call support, or some slightly larger frames elsewhere, is the culprit.