Merge testsuite and relevant compiler fixes up 2.076.1

[ibuclaw]

This synchronizes the front-end with GDC.

@joakim-noah - Care to have a look at mars.c and the rest of the back-end?

[dlang-bot]

Thanks for your pull request, @ibuclaw!

Bugzilla references

Auto-close	Bugzilla	Description
✓	13116	Should not be able to return ref to 'this'
✓	14411	switch statement: docs/behavior differ
✓	15384	assignment is sometimes still accepted as a condition
✓	15989	Initializing manifest constants with CTFE allocated data

[joakim-noah]

OK, I'll take a look.

[ibuclaw]

I might need to revert some parts, some pieces of gdc specific fixes may have slipped in (such as #7065).

[joakim-noah]

Sure, just update this pull and I'll grab the latest to test.

[joakim-noah]

With this simple patch applied on top of this pull, I get your new C++ dmd to build on linux/x64.

diff --git a/src/dmodule.c b/src/dmodule.c
index 9e9ba8087..b4aa87c26 100644
--- a/src/dmodule.c
+++ b/src/dmodule.c
@@ -73,6 +73,9 @@ Module::Module(const char *filename, Identifier *ident, int doDocComment, int do
     searchCacheSymbol = NULL;
     searchCacheFlags = 0;
     decldefs = NULL;
+    massert = NULL;
+    munittest = NULL;
+    marray = NULL;
     sictor = NULL;
     sctor = NULL;
     sdtor = NULL;
diff --git a/src/glue.c b/src/glue.c
index 51fb20489..d22f90b45 100644
--- a/src/glue.c
+++ b/src/glue.c
@@ -63,6 +63,7 @@ Symbol *toModuleUnittest(Module *m);
 Symbol *toModuleArray(Module *m);
 Symbol *toSymbolX(Dsymbol *ds, const char *prefix, int sclass, type *t, const char *suffix);
 static void genhelpers(Module *m);
+int blockExit(Statement *s, FuncDeclaration *func, bool mustNotThrow);
 
 elem *eictor;
 symbol *ictorlocalgot;
@@ -1197,7 +1198,7 @@ void FuncDeclaration_toObjFile(FuncDeclaration *fd, bool multiobj)
             Statement *sf = ExpStatement::create(fd->loc, e);
 
             Statement *stf;
-            if (sbody->blockExit(fd, false) == BEfallthru)
+            if (blockExit(sbody, fd, false) == BEfallthru)
                 stf = CompoundStatement::create(Loc(), sbody, sf);
             else
                 stf = TryFinallyStatement::create(Loc(), sbody, sf);
diff --git a/src/iasm.c b/src/iasm.c
index b97fca275..408169c1b 100644
--- a/src/iasm.c
+++ b/src/iasm.c
@@ -120,6 +120,9 @@ const char *asm_opstr(OP *pop);
 OP *asm_op_lookup(const char *s);
 void init_optab();
 
+// From expressionsem.c
+Expression *semantic(Expression *e, Scope *sc);
+
 static unsigned char asm_TKlbra_seen = 0;
 
 struct REG
@@ -3583,7 +3586,7 @@ static code *asm_db_parse(OP *pop)
             {
                 Expression *e = IdentifierExp::create(asmstate.loc, asmtok->ident);
                 Scope *sc = asmstate.sc->startCTFE();
-                e = e->semantic(sc);
+                e = semantic(e, sc);
                 sc->endCTFE();
                 e = e->ctfeInterpret();
                 if (e->op == TOKint64)
@@ -3669,7 +3672,7 @@ int asm_getnum()
         {
             Expression *e = IdentifierExp::create(asmstate.loc, asmtok->ident);
             Scope *sc = asmstate.sc->startCTFE();
-            e = e->semantic(sc);
+            e = semantic(e, sc);
             sc->endCTFE();
             e = e->ctfeInterpret();
             i = e->toInteger();
@@ -4372,7 +4375,7 @@ static OPND *asm_primary_exp()
                         }
                     }
                     Scope *sc = asmstate.sc->startCTFE();
-                    e = e->semantic(sc);
+                    e = semantic(e, sc);
                     sc->endCTFE();
                     e = e->ctfeInterpret();
                     if (e->isConst())
diff --git a/src/module.h b/src/module.h
index 76dffdfd8..a2b8289b4 100644
--- a/src/module.h
+++ b/src/module.h
@@ -166,6 +166,10 @@ public:
 
     Symbol *sfilename;          // symbol for filename
 
+    Symbol *massert;            // module assert function
+    Symbol *munittest;          // module unittest failure function
+    Symbol *marray;             // module array bounds function
+
     Module *isModule() { return this; }
     void accept(Visitor *v) { v->visit(this); }
 };
diff --git a/src/objc_glue_stubs.c b/src/objc_glue_stubs.c
index 7f72b47c1..433b74c81 100644
--- a/src/objc_glue_stubs.c
+++ b/src/objc_glue_stubs.c
@@ -17,6 +17,11 @@ class Type;
 class TypeFunction;
 struct elem;
 
+void objc_initSymbols()
+{
+    // noop
+}
+
 void objc_callfunc_setupEp(elem *esel, elem **ep, int reverse)
 {
     // noop

Checking out the v2.076.1 tag for druntime and phobos, setting DMD to use the older path ../dmd/src/dmd, and removing the -dip1000 flag from druntime's UDFLAGS, I got druntime and phobos to build. The tests for druntime don't build however, because of a type mismatch in the new SIMD core.internal.arrayop.

I suggest you apply this small patch also and let's see how far the various CIs get. I'll see if I can get the tests running for the 2.076.1 stdlib.

[ibuclaw]

• massert = NULL;
• munittest = NULL;
• marray = NULL;

This was removed from upstream, and so the codegen needs to be updated to reflect instead. (See these PRs #6530, #6536, and #6758).

The tests for druntime don't build however, because of a type mismatch in the new SIMD core.internal.arrayop.

I suspect this because of my referenced vector patch which I've applied locally to gdc (seriously dmd, allowing implicity void16<->simd conversion was a dumb idea).

[ibuclaw]

+// From expressionsem.c
+Expression *semantic(Expression *e, Scope *sc);

By the way, would it be useful putting this in a header? i.e: semantic.h.

Likewise there could be a frontend.h for a similar list of functions that are exported, but never declared in headers.

Currently, there's a ridiculous list of declarations at the top of sources, such as:

https://github.com/ibuclaw/dmd/blob/5ac1f3a19db9fec31012c61ecc0bd50ef5e24689/src/mtype.c#L50-L62

or:

https://github.com/ibuclaw/dmd/blob/5ac1f3a19db9fec31012c61ecc0bd50ef5e24689/src/expressionsem.c#L41-L81

[joakim-noah]

This was removed from upstream, and so the codegen needs to be updated to reflect instead. (See these PRs #6530, #6536, and #6758).

Yeah, I found and initally ported those pulls to C++, then realized the current 2.071 backend on this branch didn't support that. I'd like to hold off on updating the backend for now, as I started down that path by jumping to the 2.076 backend, but hit more and more of the dmd glue layer that needed updating too. I'd rather get this pull somewhat working with this older backend, then submit another pull later updating the backend. You can add these module reversions as a separate commit, and I will revert it later.

By the way, would it be useful putting this in a header? i.e: semantic.h.

Yes, I just followed what's already done, but headers would be good. OTOH, I don't see myself refactoring or doing much with this codebase other than getting it to work, so no need to put much effort into that.

[ibuclaw]

Unpicked vector conversion fixes in gdc (from here and here) because dmd's core.simd support depends on a broken type system.

[joakim-noah]

Tried your updated pull without the vector conversion fixes, but got a different error in core.internal.arrayop now:

src/core/internal/arrayop.d-mixin-44(44): Error: integer constant expression expected instead of _param_2

So I reverted that file to the commit where it was first introduced, dlang/druntime@184435f24, to get it to compile again and I was able to build the tests for druntime and phobos. With the following patches to get rid of the -dip1000 flag and add an -fPIC, I got almost all the stdlib tests to pass. I only had to disable two unittest blocks and an assert, plus the additional druntime tests likely only because of different text formatting, everything else worked on linux/x64:

diff --git a/posix.mak b/posix.mak
index 3ec4db77..4aae3cfc 100644
--- a/posix.mak
+++ b/posix.mak
@@ -23,7 +23,7 @@ ifneq ($(BUILD),release)
     endif
 endif
 
-DMD=$(DMD_DIR)/generated/$(OS)/release/$(MODEL)/dmd
+DMD=$(DMD_DIR)/src/dmd
 INSTALL_DIR=../install
 
 DOCDIR=doc
@@ -60,7 +60,7 @@ ifeq (solaris,$(OS))
 endif
 
 # Set DFLAGS
-UDFLAGS:=-conf= -Isrc -Iimport -w -dip1000 $(MODEL_FLAG) $(PIC_FLAG) $(OPTIONAL_COVERAGE)
+UDFLAGS:=-conf= -Isrc -Iimport -w $(MODEL_FLAG) $(PIC_FLAG) $(OPTIONAL_COVERAGE)
 ifeq ($(BUILD),debug)
 	UDFLAGS += -g -debug
 	DFLAGS:=$(UDFLAGS)
@@ -203,7 +203,7 @@ $(DRUNTIME): $(OBJS) $(SRCS)
 
 UT_MODULES:=$(patsubst src/%.d,$(ROOT)/unittest/%,$(SRCS))
 HAS_ADDITIONAL_TESTS:=$(shell test -d test && echo 1)
-ifeq ($(HAS_ADDITIONAL_TESTS),1)
+ifeq ($(HAS_ADDITIONAL_TESTS),0)
 	ADDITIONAL_TESTS:=test/init_fini test/exceptions test/coverage test/profile test/cycles test/allocations test/typeinfo test/thread
 	ADDITIONAL_TESTS+=$(if $(SHARED),test/shared,)
 endif
diff --git a/src/core/internal/arrayop.d b/src/core/internal/arrayop.d
index b11ffe6c..1f2b1b04 100644
--- a/src/core/internal/arrayop.d
+++ b/src/core/internal/arrayop.d
@@ -346,7 +346,7 @@ version (unittest) template _arrayOp(Args...)
     alias _arrayOp = arrayOp!Args;
 }
 
-unittest
+version(none) unittest
 {
     static void check(string op, TA, TB, T, size_t N)(TA a, TB b, in ref T[N] exp)
     {
diff --git a/src/core/memory.d b/src/core/memory.d
index af0fee1a..6f4b6faa 100644
--- a/src/core/memory.d
+++ b/src/core/memory.d
@@ -903,7 +903,7 @@ void pureFree(void* ptr) @system pure @nogc nothrow
     //  the size to the Windows API
     void* z = pureMalloc(size_t.max - 2); // won't affect `errno`
     assert(errno == fakePureGetErrno()); // errno shouldn't change
-    assert(z is null);
+    //assert(z is null);
 }
 
 // locally purified for internal use here only
diff --git a/test/common.mak b/test/common.mak
index 84aa328b..94129588 100644
--- a/test/common.mak
+++ b/test/common.mak
@@ -25,7 +25,7 @@ ifneq (default,$(MODEL))
 	MODEL_FLAG:=-m$(MODEL)
 endif
 CFLAGS:=$(MODEL_FLAG) -Wall
-DFLAGS:=$(MODEL_FLAG) -w -I../../src -I../../import -I$(SRC) -defaultlib= -debuglib= -dip1000 $(PIC_FLAG)
+DFLAGS:=$(MODEL_FLAG) -w -I../../src -I../../import -I$(SRC) -defaultlib= -debuglib= $(PIC_FLAG)
 # LINK_SHARED may be set by importing makefile
 DFLAGS+=$(if $(LINK_SHARED),-L$(DRUNTIMESO),-L$(DRUNTIME))
 ifeq ($(BUILD),debug)

Phobos tweaks:

diff --git a/posix.mak b/posix.mak
index 122fe84bc..8212e4053 100644
--- a/posix.mak
+++ b/posix.mak
@@ -97,7 +97,7 @@ ifeq ($(OS),win32wine)
 	DMD = wine dmd.exe
 	RUN = wine
 else
-	DMD = $(DMD_DIR)/generated/$(OS)/release/$(MODEL)/dmd
+	DMD = $(DMD_DIR)/src/dmd
 	ifeq ($(OS),win32)
 		CC = dmc
 	else
@@ -363,7 +363,7 @@ $(UT_LIBSO): $(UT_D_OBJS) $(OBJS) $(DRUNTIMESO)
 	$(DMD) $(DFLAGS) -shared -unittest -of$@ $(UT_D_OBJS) $(OBJS) $(DRUNTIMESO) $(LINKDL) -defaultlib= -debuglib=
 
 $(ROOT)/unittest/test_runner: $(DRUNTIME_PATH)/src/test_runner.d $(UT_LIBSO)
-	$(DMD) $(DFLAGS) -of$@ $< -L$(UT_LIBSO) -defaultlib= -debuglib=
+	$(DMD) $(DFLAGS) -fPIC -of$@ $< -L$(UT_LIBSO) -defaultlib= -debuglib=
 
 endif
 
diff --git a/std/outbuffer.d b/std/outbuffer.d
index 1d594982c..f128e95b8 100644
--- a/std/outbuffer.d
+++ b/std/outbuffer.d
@@ -375,7 +375,7 @@ class OutBuffer
 }
 
 ///
-@safe unittest
+version(none) @safe unittest
 {
     import std.string : cmp;
 

Nice work, Ian, this means porters can use this dmd-cxx branch with a relatively recent stdlib, after a couple more screws are tightened for these remaining issues.

[wilzbach]

Awesome guys! So let's up update dmd-cxx at druntime and Phobos? @joakim-noah your changes look pretty unproblematic (I am really surprised that so few changes are needed). I would recommend doing them in a separate PR, so that merging the merge commit to 2.076 is a no-brainer.

[ibuclaw]

I am really surprised that so few changes are needed

Well, gdc uses a more or less vanilla 2.076 druntime and phobos library, and passes the testsuite that ships with 2.076 as well (excluding static foreach and some dmd-specific tests).

[joakim-noah]

I guess someone can go ahead and merge this then and I will submit separate pulls to get this branch building.

[wilzbach]

I guess someone can go ahead and merge this then

What's up with Travis? I thought that one was the only CI working so far?

glue.c: In function ‘void obj_write_deferred(Library*)’:
glue.c:145:17: error: ‘class Module’ has no member named ‘massert’
             md->massert = m->massert;
                 ^
glue.c:145:30: error: ‘class Module’ has no member named ‘massert’
             md->massert = m->massert;
                              ^
glue.c:146:17: error: ‘class Module’ has no member named ‘munittest’
             md->munittest = m->munittest;
                 ^
glue.c:146:32: error: ‘class Module’ has no member named ‘munittest’
             md->munittest = m->munittest;
                                ^
glue.c:147:17: error: ‘class Module’ has no member named ‘marray’
             md->marray = m->marray;
                 ^
glue.c:147:29: error: ‘class Module’ has no member named ‘marray’
             md->marray = m->marray;
                             ^
glue.c: In function ‘void FuncDeclaration_toObjFile(FuncDeclaration*, bool)’:
glue.c:1200:24: error: ‘class Statement’ has no member named ‘blockExit’
             if (sbody->blockExit(fd, false) == BEfallthru)
                        ^
make: *** [glue.o] Error 1

[ibuclaw]

+    //assert(z is null);

Note, this is a glibc bug in 2.26.

[ibuclaw]

What's up with Travis? I thought that one was the only CI working so far?

I explicitly did not make any backend changes. So backend code won't compile unless its been updated to match the new frontend api/interface

[wilzbach]

OK. Let's move dmd-cxx forward then.