[ARM] Potential codegen issues on Graviton3

[redknightlois]

Description

We have included Graviton3 chipsets on our testing matrix and found many different tests that error consistently. However, those errors are not triggering when run in the following platforms:

• Linux x64
• Windows x64
• Windows x86

Moreover, when run continuously errors only trigger after 2 conditions:

• It is run in arm64 (reproduced on Graviton3)
• A set amount of iterations happened (always beyond 128)
• It is run in release mode

At first we suspected that we were doing something wrong on our end [its possible we are corrupting the heap somehow] but the deterministic nature of the failure and repeatability after it triggers for the first time makes us suspect it is something more permanent than a heap corruption.

I include several different ways to reproduce the errors.

Reproduction Steps

Download any of the following reproductions:

https://github.com/redknightlois/ravendb/tree/numerical-difference
https://github.com/redknightlois/ravendb/tree/failed-datetime-comparison
https://github.com/redknightlois/ravendb/tree/failed-patching

Run them in order to ensure that optimizations will kick in:

cd test/Tryouts
dotnet run -c Release

Observe the errors after a set amount of loops.

Expected behavior

They should execute the 1000 loops without printing any exception.

Actual behavior

Trigger exception repeatedly after a set amount of loops.

Regression?

No response

Known Workarounds

None

Configuration

.NET SDK:
 Version:           8.0.103
 Commit:            6a90b4b4bc
 Workload version:  8.0.100-manifests.e99a2be4

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         ubuntu.22.04-arm64
 Base Path:   /usr/lib/dotnet/sdk/8.0.103/

.NET workloads installed:
 Workload version: 8.0.100-manifests.e99a2be4
There are no installed workloads to display.

Host:
  Version:      8.0.3
  Architecture: arm64
  Commit:       9f4b1f5d66

.NET SDKs installed:
  8.0.103 [/usr/lib/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.17 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.3 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.17 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.3 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

Other information

No response

[EgorBo]

Would be nice to have more details - does it fail on other ARM64 machines, which tests and what is the expected/actual inputs/outputs? Is it floating point related? E.g. we do not guarantee that all floating-point related operations e.g. casts to integers, edge cases in Math.* calls are always 100% the same for the same input on all architectures/platforms. Would be nice to confirm that the issue is indeed on dotnet/runtime side.

[redknightlois]

Some are binding errors at the runtime level which points into heap corruptions. One is showing 2 floating point numbers which is clearly not a floating point issue, more like getting the data from a different memory location.

I only have Graviton3 from AWS on ARM64, it is consistently failing on all of the machines provisioned. All the tests execute properly on Linux and Windows on 32-bits and 64-bit Intel.

That's why I provided 3 different ways to trigger the behavior with completely different error behaviors. But since, they all trigger after a number of passes happen (and the number is consistent across runs) and they all run successfully run under Debug build, it leads into suspect that after a certain optimization is performed, the errors trigger.

[ayende]

Okay, I have a much smaller scope for reproducing this.
Here is the relevant code (running as Tryout/Program.cs):

using System;
using System.Diagnostics;
using System.IO;
using System.Net.Http.Json;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;
using Amazon.S3.Model;
using Tests.Infrastructure;
using Raven.Server.Utils;
using SlowTests.Corax;
using SlowTests.Sharding.Cluster;
using Xunit;
using FastTests.Voron.Util;
using Jint;
using Jint.Native;
using Jint.Native.Function;
using Jint.Native.Object;
using Raven.Server.Config;
using Raven.Server.Documents;
using Raven.Server.Documents.Patch;
using Raven.Server.Documents.Queries.LuceneIntegration;
using Sparrow.Json;

namespace Tryouts;

public static class Program
{
    static Program()
    {
        XunitLogging.RedirectStreams = false;
    }

    public class MyKey : ScriptRunnerCache.Key
    {
        public override void GenerateScript(ScriptRunner runner)
        {
            runner.AddScript("function project(p) { return { Long: p.Long, Double: p.Double }; } ");
        }

        public override bool Equals(object obj)
        {
            return ReferenceEquals(this, obj);
        }

        public override int GetHashCode()
        {
            return RuntimeHelpers.GetHashCode(this);
        }
    }

    public static void Main(string[] args)
    {

        using var ctx = JsonOperationContext.ShortTermSingleUse();

        var ms = new MemoryStream("{'Long': 9007199254740990, 'Double': 1.7976931348623157E+308}"u8.ToArray());
        var doc = new Document { Data = ctx.ReadForMemoryAsync(ms, "test").Result };

        var func = Jint.Engine.PrepareScript("function project(p) { return { Long: p.Long, Double: p.Double }; } ", strict: false);
        var engine = new Jint.Engine();
        var jsonStr = (Function)engine.GetValue("JSON").AsObject().Get("stringify");
        engine.Execute(func);

        var method = engine.GetValue("project");
        const string expected = "{\"Long\":9007199254740990,\"Double\":1.7976931348623157E+308}";


        for (int i = 0; i < 100_000; i++)
        {
            var v = method.Call(new BlittableObjectInstance(engine, null, doc.Data, doc));

            var j = JsBlittableBridge.Translate(ctx, engine, (ObjectInstance)v).ToString();
            if (j != expected)
            {
                Console.WriteLine(i + " " + j);
                return;
            }
        }

        Console.WriteLine("Done");
    }
}

To make things easier, what this does is execute this funciton:

function project(p) { return { Long: p.Long, Double: p.Double }; }

Over this input:

{'Long': 9007199254740990, 'Double': 1.7976931348623157E+308}

Will sometimes return:

ubuntu@ip-172-31-47-129:~/ravendb/test/Tryouts$ dotnet run -c release
15862 {"Long":9007199254740990.0,"Double":9223372036854775807}
ubuntu@ip-172-31-47-129:~/ravendb/test/Tryouts$ dotnet run -c release
18067 {"Long":9007199254740990.0,"Double":9223372036854775807}
ubuntu@ip-172-31-47-129:~/ravendb/test/Tryouts$ dotnet run -c release
20129 {"Long":9007199254740990.0,"Double":9223372036854775807}

Note that the number of iterations is different.

I am assuming that this is tier JIT running and a bug happening there somewhere.

I wasn't able to simplify this further, I'm afraid.

When running this with debug mode, it completes successfully each time.

ubuntu@ip-172-31-47-129:~/ravendb/test/Tryouts$ dotnet run -c debug
Done
ubuntu@ip-172-31-47-129:~/ravendb/test/Tryouts$ dotnet run -c debug
Done

Once this happens, it is entirely reproducible.

[redknightlois]

We dont see the error immediately because of tiered compilation, but it fails immediately on the first iteration when disabled.

  <PropertyGroup>
    <TieredCompilation>false</TieredCompilation>
  </PropertyGroup>

This with a simplifier reproduction that barely needs anything confirms the suspicion that the error is related to code generation on ARM64 platform.

When we disable quick jit even if tiered compilation is on, we also fail on first iteration.

  <PropertyGroup>
    <TieredCompilation>true</TieredCompilation>
     <TieredCompilationQuickJit>false</TieredCompilationQuickJit>
  </PropertyGroup>

However, when we disable only quick jit for loops we revert to fail after a set amount of iterations again.

  <PropertyGroup>
    <TieredCompilation>true</TieredCompilation>
     <TieredCompilationQuickJit>true</TieredCompilationQuickJit>
     <TieredCompilationQuickJitForLoops>false</TieredCompilationQuickJitForLoops>
  </PropertyGroup>

[AndyAyersMS]

@EgorBo can you see if the repro above works for us? If not I can look later this week.

[ayende]

Some more information on this.

A user is reporting that running RavenDB or Mac M2 will crash with:

2024-04-16 13:28:58 assertion failed [block != nullptr]: BasicBlock requested for unrecognized address
2024-04-16 13:28:58 (BuilderBase.h:550 block_for_offset)

When running docker run ravendb/ravendb:5.4.116-ubuntu.22.04-arm64v8

When setting DOTNET_TC_QuickJitForLoops=0, it goes further and fails with:

ravendb exited with code 139

[ayende]

We tried with:

DOTNET_TieredCompilation=0
DOTNET_TC_QuickJit=0
DOTNET_TC_QuickJitForLoops=0

And the user is getting:

ravendb      | Running non-interactive.

ravendb      | assertion failed [block != nullptr]: BasicBlock requested for unrecognized address

ravendb      | (BuilderBase.h:550 block_for_offset)

ravendb      |

ravendb exited with code 133

Is there any way for us to extract additional information here?

[EgorBo]

@EgorBo can you see if the repro above works for us? If not I can look later this week.

I managed to repro it using dotnet run from SDK, but it never reproduces on locally build runtime (either net9 or net8, either Release or Checked) so I am a bit puzzled, the project is quite huge - does it spawn processes internally, does it try to build some managed code dynamically on run?

PS: it doesn't even repro when I copy my locally built net8.0 bits (from release branch) to the .net 8.0 sdk's folder