[Arm64] Optimize JIT_WriteBarriers*

[sdmaclea]

They appear to be on the critical path to writing object references to the heap.

AMD64 went to significant length to optimize these functions.

[sdmaclea]

@dotnet/arm64-contrib

The AMD64 optimization involves using a large set of WriteBarrier code instantiations. The JIT_WriteBarrier function is memcpy from one of many variants of the function based on what does or does not need to be executed at the moment. After memcpy, the code is patched to include the current value of key GC variables as constants to avoid memory reads. Whenever these values change the key values are rewritten if necessary and the Intruction Cache is flushed.

This is possible for arm64. Here are the complications

• 4 movz/movk 16-bit immediates would have to be patched
• Data Cache flush of Code Range to Point of Unification
• Often code pages on arm64 are expected to be not writable.

The amd64 implementation appears brittle with

• lots of copies of similar code. I think this can be avoided by using a macro and instantiating it a few times
• irregular naming conventions. Seems like a feature bitmask value might be simpler.

There is a simpler/less optimal and/or orthogonal versions...

• Use indirect branch based on current WriteBarrier mode.
• Use ldp for GC heap ranges
• Add JIT_ByRefWriteBarrier Pair or Range forms
• Eliminate redundant checks of heap ranges for large object copies
• Use conditional compare to avoid extra potentially mispredicted branches

For performance, with a benchmark writing lots of GC references, I can see why WriteBarrier code could get in the way.

With an Out Of Order speculatively executed machine, it seems that

• Using an indirect branch based on current mode should be very predictable. Implementation is significantly simpler than amd64 fast approach
• Using load pair and reducing branches should be a win
• As long as there is more than one way in the L1 Data Cache, reading the GC ranges should hit and should not have a large penalty
• Range versions could easily be a win. GC latency could be impacted if these were large. Maybe 64 references would be maximum supported...

I guess my default would be to use the simpler approach rather than trying to duplicate amd64.

Comments appreciated

[briansull]

Sounds like a good approach to me.

[BruceForstall]

ARM32 uses a set of macros to implement the barriers, which creates a bunch of labels, it appears, to give the address/offset of code to be updated with specific GC ranges. Presumably you could follow that model.

[sdmaclea]

@BruceForstall Thanks. Looks like the arm32 handles the mode switching like amd64. For each of three types of barriers, it supports 4 modes : single/multiple processor & pre/post grow of gen0 heap. The code looks like it is still loading GC ranges from the globals. So there may be an optimization opportunity if it is actually faster.

amd64 is supporting 8 modes : pre/post gen0 grow, server/desktop, GC write watch true/false

I think arm64 should use the amd64 modes or a subset of them.

I think I will take this in steps.

1. Optimize the code while using ldp for range checking.
2. Add range versions of helpers
3. Use modal approach using macros to implement modes and indirect branches to use the modal code.
4. Implement patching if necessary to replace loads with constants and eliminate indirect branch.

[BruceForstall]

The code looks like it is still loading GC ranges from the globals.

You're talking about things like g_lowest_address, etc.? These offset of these in the WriteBarriers appears to be stored in a table, and then vm\arm\stubs.cpp, UpdateGCWriteBarriers() is called to patch them when these values change. (The table is created in a .clrwb section in the clr image; see asmhelper.asm/S)

[sdmaclea]

@BruceForstall Thanks. I see and understand the code now. I am still surprised that neither arm32 nor amd64 has to make the pages writeable (like what might be done in PEImageLayout::ApplyBaseRelocations()). Are we able to get away with writeable executable pages because we are a VM and the client code can not see the machine executable pages?

[BruceForstall]

@sdmaclea We don't do a good job keeping pages Executable but not Writeable; we should do better.

[briansull]

These are stubs that are written into RWX memory that we allocate in process.
It is the same as with JITted code, the code gets written into memory and executed.
We don't go back and remove the Write permissions on the page, though we have talked about doing that.

[sdmaclea]

@BruceForstall arm32 JIT_CHECKEDWRITEBARRIER_MP & JIT_BYREFWRITEBARRIER mp are doing dmb even in case str is not to heap. I believe this is unnecessary and costly.

[sdmaclea]

+@Maoni0 @swgillespie

Has anyone given any thought to using the pointer tag bits for identifying whether a pointer is in gc heap & gen 0 heap.

It looks like it has been supported in the Linux kernel for a while. https://www.kernel.org/doc/Documentation/arm64/tagged-pointers.txt . Is it also supported in Windows Kernel?

enum
{
  gcNonHeap = 0
  gcGenHeap = 1ULL << 56
  gcGen0 = gcGenHeap | (1ULL << 57)
}

If we used the tag bits the heap check could be

    tbnz x12, dotnet/coreclr#56, JIT_WriteBarrier

The ephemeral check would be

    tbz x12, dotnet/runtime#3878, Exit

The card table, card bundle, write watch code would have to altered mask the high byte.

The GC would have to set/update the tag during allocation and relocation.
Calls into kernel would need to be sanitized.

[swgillespie]

I imagine that such a scheme would require a read barrier in order to mask off the tagged bits, or at least the JIT would need to mask them off before dereferencing it. Depending on the platform, though, that may not be all that expensive: this paper (2004) measured the overhead of a masking read barrier and found it to be around 8% on x86 and 0.85% on PowerPC (while the cost of the write barrier is around 2%). Perhaps ARM could reap the same benefits that PowerPC has in this regard.

That said I don't think our JIT has the ability to emit read barriers anyway and, even if it did, all of the unmanaged code in the runtime dealing with managed objects certainly aren't using read barriers. I'm not sure if there's any history around read barriers in the runtime but it sounds like it could be potentially difficult.

[sdmaclea]

@swgillespie I am not sure I understand why a read barrier would be necessary.

To make sure we are on the same page...

For aarch64, the MMU has a feature called Top Byte Ignore. It is designed to allow the top 8 bits of a Virtual Address to be used for any purpose software deems necessary. So the pointer should be usable as is for almost every purpose. The feature is enabled by default in Linux for User space pointers. There is theoretically no need to mask off the top byte (for most purposes).

I am assuming when objects are relocated the pointers all need to be updated anyway. So a promotion from gen0 to gen1, would update the tag bits as part of the object relocation.

[swgillespie]

Ah, I see! In that case, you're right - thanks for the clarification and sorry for the misunderstanding.