[RyuJIT] Update the emitter for VEX-encoded SSE4.1/4.2 instructions with containment

[fiigii]

Detected this issue from dotnet/coreclr#16287.

With VEX-encdoing, certain SSE4.1/SSE4.2 instructions (vpcmpeqq) have incorrect encoding on INS reg, reg, mem form.
For example, vpcmpeqq ymm7, ymm6, ymmword ptr[rax] (ECS:6, ACS:5) is compiled to

C4 E2 4D 39 38
         ^

but the correct encoding should be

C4 E2 4D 29 38
         ^

The opcode of vpcmpeqq (66 0F 38 29) seems accidently changed.

Subset of https://github.com/dotnet/coreclr/issues/15908
I will look into this issue after dotnet/coreclr#16249 gets merged.

cc @CarolEidt @tannergooding

[tannergooding]

@fiigii, could you provide a bit more context here? The encoding handling looks to be identical to some already supported code paths.

[fiigii]

The code vpcmpeqq ymm6, ymm6, ymmword ptr[rax] (ECS:6, ACS:5) is compiled to C4 E2 7D 39 30 that is actually vpcmpeqq ymm6, ymm0, ymmword ptr[rax] (ECS:6, ACS:5). I will look into it soon.

[tannergooding]

Looks like this is because 4 byte SSE4/AVX instructions are generally special cased so they don't currently call insEncodeReg3456 for reg2 in all the required places.

[tannergooding]

It works for the existing code paths since they are reliant on IsDstDstSrc and IsDstSrcSrc to handle it.

[fiigii]

I rebased top-on the latest master. Now, the problem becomes to that vpcmpeqq ymm7, ymm6, ymmword ptr[rax] (ECS:6, ACS:5) is compiled to

C4 E2 4D 39 38
         ^

but the correct encoding should be

C4 E2 4D 29 38
         ^

The opcode of vpcmpeqq (66 0F 38 29) seems accidently changed...

[tannergooding]

@fiigii, is this only happening in the _ro run for you?

[fiigii]

is this only happening in the _ro run for you?

@tannergooding Yes! Did you get this also?

[tannergooding]

@fiigii, yes and I see the problem.

Most of the instruction formats special case the 4 byte instructions before calling emitOutput (ex: https://github.com/dotnet/coreclr/blob/master/src/jit/emitxarch.cpp#L12161).

I had also fixed most of these up for the emitOutputAM case (https://github.com/dotnet/coreclr/blob/master/src/jit/emitxarch.cpp#L12030).

However, there are a couple places (https://github.com/dotnet/coreclr/blob/master/src/jit/emitxarch.cpp#L12053 and https://github.com/dotnet/coreclr/blob/master/src/jit/emitxarch.cpp#L12062, there may be more) where it isn't special cased and code is directly or'd with regcode.

[fiigii]

Thank you so much for the information. I will try to fix it after dotnet/coreclr#16287 merged and enable Avx2.CompareEqual.(U)Int64 again.

[tannergooding]

@fiigii, I've assigned to you since you indicated you wanted to try and fix it.

Let me know if you're focused on other things and I can take a look instead.