CIL fails to parse clang-20 xmmintrin.h

[mbyzhang]

CIL fails to parse C sources that include clang-20 xmmintrin.h.

Minimal C source xmm.c

#include <xmmintrin.h>

allocscc output from ALLOCSCC_CC=clang-20 CC_IS_CLANG=1 allocscc -c xmm.c

Warning: lexing _Float64 as an ident
Warning: lexing _Float64 as an ident
/usr/lib/llvm-20/lib/clang/20/include/xmmintrin.h[1696:31-124] : syntax error
Parsing errorFatal error: exception Frontc.ParseError("Parse error")

The offending lines in xmmintrin.h are

static __inline__ __m128 __DEFAULT_FN_ATTRS_SSE2
_mm_cvtpi32_ps(__m128 __a, __m64 __b)
{
  return (__m128)__builtin_shufflevector(
      (__v4sf)__a,
      __builtin_convertvector((__v4si)__zext128(__b), __v4sf), // <-- This is line 1696
      4, 5, 2, 3);
}

[stephenrkell]

Thanks Peter.

My first guess was simply that __built_convertvector is not a builtin known to CIL. Builtins needs special treatment by CIL, essentially because they cannot be prototyped. If you find the list of builtins in cil.ml and try adding __builtin_convertvector to it, that could be a quick fix or at least a way to test the hypothesis.

However, on second thoughts, since it is a parse error not an extraneous prototype, this may not be the problem. I would have to dig into a complete .i file -- feel free to send me one by e-mail or point me at one on our server.

(Other thoughts: for context, how essential is it that we make CIL parse this file? Is there a way to avoid the need?)

[mbyzhang]

Thanks for the reply. Should I have a try fixing that? I saw some commits that add builtins from Tom so maybe I can follow that. What's a bit more complicated might be that unlike most builtins, the second argument of __builtin_convertvector is a type name rather than an expression, but I think I can figure it out after spending some time. Or alternatively should I ask Tom about if he's fixing the same issue?

(Other thoughts: for context, how essential is it that we make CIL parse this file? Is there a way to avoid the need?)

openjpeg is using that, which is a dependency of the analyze-image challenge.

[mbyzhang]

We also have issue parsing _Float16 types in the xmmintrin.h. They are however implemented in Goblint CIL. I'm wondering if we should move on with our current CIL or should we move to Goblint CIL before making any changes?

[stephenrkell]

Thanks Peter. I think the way I would break this down is:

• understand exactly what is not supported -- it sounds like it's "another builtin taking a type" but worth double-checking
• assuming that's correct, then yes, go ahead and see if you can add support. You could possibly model what you do on __builtin_offsetof, which also takes a type and which is implemented.

16-bit floats are best seen as a separate issue. They "should not" be a problem -- I have added support for these in the past. However, I only tested it in GCC; it might be somehow gated out in Clang. So step 1 is to investigate why that is not being picked up. Maybe Clang uses different names for its built-in 16-bit float types than GCC does, and hides that with typedefs? In principle, this is stuff that the configure script and Machdep CIL module should take care of, i.e. if the configure-time compiler parses these idents as built-in typenames, CIL should do likewise (see frontc/clexer.mll). But perhaps that is not working correctly. I should add that I am preparing a patch to eliminate Machdep and instead work from the preprocessor clues left behind by -dD. This will allow CIL to grok the underlying compiler without probing it at configure time. However, some amount of Machdep will remain as a fallback/default.

[jryans]

Might not be relevant, but I'm reminded that Clang is doing a bit of a complex dance with 16-bit floating point types. Clang only supports parts of this feature on certain targets.

[stephenrkell]

Thanks Ryan... useful to know.

Another thing to check is whether openjpeg needs these intrinsics unconditionally, or if it has inferred e.g. in its configure script that the compiler should support them. If we can configure them out, it might be a quicker fix. See #129 for the underlying issue with how we are using CIL at present.

[stephenrkell]

As I recall, in my patch to support _Float16 / short float I did not add __bf16 or __fp16, which seem to be Clang-private names.

[mbyzhang]

I'll have a go at fixing __builtin_convertvector as discussed

[mbyzhang]

After __bf16 gets added (in my local changes), the next issue is about initializing vectors. Specifically, CIL can't process the following.

typedef long long __m64 __attribute__((__vector_size__(16), __aligned__(8)));


int main() {
    __m64 foo = { 0, 1 };
    return 0;
}

$ cilly --verbose -c foo.c
Preprocessing foo.c
Frontc is parsing /tmp/cil-1rZSPCQK.i
Converting CABS->CIL
b.c:5: Unimplemented: doInit: unexpected NEXT_INIT for long long __attribute__((__vector_size__(16),
__aligned__(8)))

Error in doStatement (Errormsg.Error)
Removing unused temporaries
Error: Cabs2cil had some errors
Fatal error: exception Errormsg.Error

The problem is with = { 0, 1 };. For arrays, CIL expends int foo[] = {0, 1}; to int foo; foo[0] = 0; foo[1] = 1;. But I think we can't do individual elements assignments with vectors.

[stephenrkell]

Thanks Peter. I understand the problem.

My first instinct is to check the vector builtin functions that are offered by GCC and Clang, in case one of them provides a way (perhaps non-obvious / convoluted) to do this elementwise assignment.

Another option is to do a global rewrite such that any variable or field declared with a type having a vector_size attribute is instead rendered as a union of the original type and an array of the vector element. That is a bigger hammer and also requires rewriting most reads and writes.

The third option is to add support for local variable initializers directly to CIL. It already has a notion of global variable initializer. I don't know how well this would transfer. It's a breaking change in that a lot of CIL passes will assume that local variables don't have initializers... though, e.g. if the pass is simply using the visitor, then a change to the visitor will probably do the right thing most of the time (visiting the initializer expression, or whatever).

I am still wondering whether the need for this can be configured out somehow, e.g. building openjpeg without vectorisation, although that has obvious downsides.

[stephenrkell]

Here is what I thinks is a proof-of-concept way to do it. I'm not saying this is the right way. It does require pregenerating various global vectors. What it does achieve is to simulate elementwise assignment to local vectors, i.e. avoiding the need to introduce local initializers to CIL. We rely on the global initializers and __builtin_shuffle. Of course it still does not parse without an understanding of vector_size, array-style global initializers and array subscript operations on types carrying that attribute.

#include <stdio.h>
#include <strings.h>

typedef int v4si __attribute__ ((vector_size (16)));

v4si v4si_zero;
v4si mask_v4si_assign_first =  {4, 1, 2, 3 };
v4si mask_v4si_assign_second = {0, 5, 2, 3 };
v4si mask_v4si_assign_third =  {0, 1, 6, 3 };
v4si mask_v4si_assign_fourth = {0, 1, 2, 7 };

int main(void)
{
	v4si a; // = {4,3,2,1};
	a = __builtin_shuffle(a, v4si_zero + 4, mask_v4si_assign_first);
	a = __builtin_shuffle(a, v4si_zero + 3, mask_v4si_assign_second);
	a = __builtin_shuffle(a, v4si_zero + 2, mask_v4si_assign_third);
	a = __builtin_shuffle(a, v4si_zero + 1, mask_v4si_assign_fourth);

	printf("%d %d %d %d\n", a[0], a[1], a[2], a[3]);
	return 0;
}

[stephenrkell]

Actually, are you sure we can't just do array-style assignment to individual vector elements? GCC seems happy to do this for me.

[mbyzhang]

Ah sorry I was wrong. Both clang and GCC does support array-style assignment. So maybe let's do that instead similar to array handling.

[mbyzhang]

Fixed in mbyzhangAlt/cil@9d69f97