[ltsmaster] backport betterC and OS-specific assert functions

[joakim-noah]

This is useful in ltsmaster when porting to new platforms, certainly would like to have it for the Android/AArch64 port. I simply ran git log master --grep=betterC and found 11 commits that referenced it. This pull contains 8 of them, the first couple coming from @kinke's betterC pull #2365, but leaving out subsequent commits 97514dd, 1b8a3b9, and 04cd029, as they didn't seem necessary. Let me know if any of them should be in; I also had to add 612c8a7 to get the refactored getCAssertFunctionParamTypes().

A basic betterC test file worked fine, time to run this pull through the CI and I'll backport the other betterC tests in the dmd testsuite when this gets in.

[dnadlinger]

What is it actually useful for, though?

[joakim-noah]

When checking codegen for new platforms, you can compile small betterC test files without needing to port even druntime.

[joakim-noah]

CI looks good, not sure why two stdlib tests fail on Circle, different environment? Travis is good, only the one multilib build times out as always. I'll get those DMD testsuite files in, update this pull to use them, and we can get this pull in.

[joakim-noah]

The testsuite turned up some more frontend commits from LDC merges that check for betterC, will get those into this pull too and once I get the testsuite passing, this patch should be ready.

[joakim-noah]

Btw, given the recent change to bump the host compiler version for dmd's CI, are we going to stop putting out ltsmaster releases and tell people to cross-compile instead? If so, not much value in this pull.

[kinke]

I'd strive for (cross-)compiles, but the non-runtime CMake script needs work to support that properly (i.e., not just from Linux to another arch's Linux). I don't know if that (having a D compiler) is an option for bootstrapping for distro maintainers etc. though.

[joakim-noah]

I don't know if that (having a D compiler) is an option for bootstrapping for distro maintainers etc. though.

We discussed it in the NG (linking you to the mailing list archive as the forum seems down now), appears to be the norm.

[kalev]

With my distro package maintainer hat on, I'd say it's way easier to bootstrap natively from ltsmaster than to use cross-compiling.

[joakim-noah]

Why would that be? Once you have a cross-compiled compiler binary, like those ghc packages linked from my NG post, it's quicker to build the latest compiler using that cross-compiled compiler binary.

For porters, it's easier to just port the ldc and stdlib master branch once, rather than porting both the ltsmaster and master branches.

For the ldc devs, they just have to keep one branch working, master.

It seems easier for everyone involved. In any case, with dmd master switching to building with dmd 2.079, it's inevitable that newer features from that D version will creep in to the D frontend, just as some use of druntime and phobos has crept into ldc master. So that means ldc will have to either maintain two branches for porters, ltsmaster and ldc 1.9, or just tell people to cross-compile.

[kalev]

Why would that be?

It's just more manual steps involved. Right now when I'm updating Fedora ldc packages, I have to upload two source tarballs (ltsmaster and the actual ldc being compiled), and Fedora build servers do the rest.

If it needs bootstrapping through cross compilation, then I need to have a cross compilation environment set up, with llvm and all other necessary libraries for each of the target architectures, then cross compile for each of the target arches to get a binary boostrap compiler, then upload that to the Fedora build servers together with the source tarball, do a build using the binary cross-compiled compiler to build ldc from source on the build servers, then disable boostrap, do one more build to be sure that it's self hosting and fully built from source.

Just more steps involved with cross compilation :) I am not even saying that I'm opposed to that, just pointing out that it's more work like this for Linux distro maintainers. The tradeoff (having to only maintain one code base) is probably still worth it.

Also note that this above only applies to bootstrapping; if we can avoid re-bootstrapping all the time and can just use already-boostrapped previous ldc version then this becomes pretty much one time effort that's only necessary when bringing up a new arch or distro.

[joakim-noah]

if we can avoid re-bootstrapping all the time and can just use already-boostrapped previous ldc version then this becomes pretty much one time effort that's only necessary when bringing up a new arch or distro.

Yep, that's all that's needed. You wouldn't need to cross-compile all the time, only for the initial port.

[joakim-noah]

Closing, since I will not be finishing this pull nor do I think ltsmaster should be used anymore, since I was able to do the Android/x86-x64 ports without using it.

Also, I have decided to leave the D community and stop using D, with the last straw being the recent censorship of criticism on the D forums. Collaborating a little with you guys on ldc is one of the few things I will miss.

Of course, I will not be volunteering to put out Alpine or native Android builds anymore. If anyone wants to pick up the Android builds, it should be pretty easy to bump the ldc build script in the Termux package repository, as most of my recent Termux pulls for ldc were just bumping commit hashes.

For Alpine, I would simply spin up a VPS every couple months, install all the listed Alpine packages in the release notes, plus some dev packages like g++, cmake, and ninja, and manually build and pack it up. Should be pretty easy for someone else to do or script if wanted.

Finally, my only remaining in-progress work was getting the Android port to work with other linkers than ld.bfd. I got it working for building executables with this patch for ldc:

diff --git a/driver/codegenerator.cpp b/driver/codegenerator.cpp
index 6d867c02..efef6137 100644
--- a/driver/codegenerator.cpp
+++ b/driver/codegenerator.cpp
@@ -321,15 +321,8 @@ void CodeGenerator::emit(Module *m) {
           ir_->module, llvm::Type::getInt32Ty(ir_->module.getContext()), false,
           llvm::GlobalValue::ExternalLinkage,
           llvm::ConstantInt::get(ir_->module.getContext(), APInt(32, 0)),
-          "_tlsstart", &*(ir_->module.global_begin()));
+          "_tlsfiller", &*(ir_->module.global_begin()));
       startSymbol->setSection(".tdata");
-
-      auto endSymbol = new llvm::GlobalVariable(
-          ir_->module, llvm::Type::getInt32Ty(ir_->module.getContext()), false,
-          llvm::GlobalValue::ExternalLinkage,
-          llvm::ConstantInt::get(ir_->module.getContext(), APInt(32, 0)),
-          "_tlsend");
-      endSymbol->setSection(".tcommon");
     } else if (global.params.targetTriple->isOSLinux()) {
       // On Linux, strongly define the excecutabe BSS bracketing symbols in
       // the main module for druntime use (see rt.sections_elf_shared).

It keeps around at least one TLS symbol _tlsfiller in .tdata, which I didn't think was needed at first, but several small executables in the dmd testsuite would fail with this associated druntime patch if I removed both these bracketing symbols:

diff --git a/src/rt/backtrace/elf.d b/src/rt/backtrace/elf.d
index e7c9420d..954d362f 100644
--- a/src/rt/backtrace/elf.d
+++ b/src/rt/backtrace/elf.d
@@ -28,7 +28,7 @@ struct Image
 {
     private ElfFile file;
 
-    static Image openSelf()
+    static Image openSelf() @nogc nothrow
     {
         Image image;
 
@@ -38,11 +38,27 @@ struct Image
         return image;
     }
 
-    @property bool isValid()
+    @property bool isValid() @nogc nothrow
     {
         return file != ElfFile.init;
     }
 
+    long getSectionAddressAndSize(const(char)[] sectionName, out size_t sectionAddress) @nogc nothrow
+    {
+        auto stringSectionHeader = ElfSectionHeader(&file, file.ehdr.e_shstrndx);
+        auto stringSection = ElfSection(&file, &stringSectionHeader);
+
+        auto sectionIndex = findSectionByName(&file, &stringSection, sectionName);
+        if (sectionIndex != -1)
+        {
+            auto sectionHeader = ElfSectionHeader(&file, sectionIndex);
+            sectionAddress = sectionHeader.sh_addr;
+            return sectionHeader.sh_size;
+        }
+	else
+            return -1;
+    }
+
     const(ubyte)[] getDebugLineSectionData()
     {
         auto stringSectionHeader = ElfSectionHeader(&file, file.ehdr.e_shstrndx);
diff --git a/src/rt/sections_android.d b/src/rt/sections_android.d
index 33644151..8d57e596 100644
--- a/src/rt/sections_android.d
+++ b/src/rt/sections_android.d
@@ -14,8 +14,8 @@ version (CRuntime_Bionic):
 
 // debug = PRINTF;
 debug(PRINTF) import core.stdc.stdio;
-import core.stdc.stdlib : malloc, free;
-import rt.deh, rt.minfo;
+import core.stdc.stdlib : free;
+import rt.deh, rt.minfo, rt.util.container.array;
 import core.sys.posix.pthread;
 import core.stdc.stdlib : calloc;
 import core.stdc.string : memcpy;
@@ -56,7 +56,7 @@ struct SectionGroup
 
 private:
     ModuleGroup _moduleGroup;
-    void[][1] _gcRanges;
+    Array!(void[]) _gcRanges;
 }
 
 void initSections() nothrow @nogc
@@ -75,11 +75,32 @@ void initSections() nothrow @nogc
     }
     _sections.moduleGroup = ModuleGroup(mbeg[0 .. mend - mbeg]);
 
-    auto pbeg = cast(void*)&_tlsend;
-    auto pend = cast(void*)&__bss_end__;
-    // _tlsend is a 32-bit int and may not be 64-bit void*-aligned, so align pbeg.
-    version (D_LP64) pbeg = cast(void*)(cast(size_t)(pbeg + 7) & ~cast(size_t)7);
-    _sections._gcRanges[0] = pbeg[0 .. pend - pbeg];
+    auto delim = cast(void*)&_tlsfiller;
+	import rt.backtrace.elf;
+	import core.internal.abort;
+    auto image = Image.openSelf();
+    if (image.isValid)
+    {
+	size_t addr, size;
+	size =  image.getSectionAddressAndSize("__minfo", addr);
+   auto base = cast(void*)mbeg;
+   base -= addr;
+	tdataSize =  image.getSectionAddressAndSize(".tdata", addr);
+	tdatabegin = base + addr;
+	tbssSize =  image.getSectionAddressAndSize(".tbss", addr);
+	tbssbegin = base + addr;
+	delim < tbssbegin || abort("D's Android support assumes .tdata comes before .tbss.");
+
+	size =  image.getSectionAddressAndSize(".data", addr);
+	void* sectionStart = base + addr;
+        _sections._gcRanges.insertBack(sectionStart[0 .. size]);
+	size =  image.getSectionAddressAndSize(".bss", addr);
+	if(size >= 0)
+	{sectionStart = base + addr;
+	auto foo = sectionStart[0 .. size];
+        _sections._gcRanges.insertBack(foo);
+	}else {abort("couldn't find .bss section");}
+    }
 }
 
 void finiSections() nothrow @nogc
@@ -118,8 +139,10 @@ void scanTLSRanges(void[]* rng, scope void delegate(void* pbeg, void* pend) noth
 extern(C) void* __tls_get_addr( void* p ) nothrow @nogc
 {
     debug(PRINTF) printf("  __tls_get_addr input - %p\n", p);
-    immutable offset = cast(size_t)(p - cast(void*)&_tlsstart);
+    size_t offset = cast(size_t)(p - tdatabegin);
     auto tls = getTLSBlockAlloc();
+    if (p >= tbssbegin)
+        offset -= tbssbegin - tdatabegin - tdataSize;
     assert(offset < tls.length);
     return tls.ptr + offset;
 }
@@ -149,17 +172,19 @@ ref void[] getTLSBlockAlloc() nothrow @nogc
     auto pary = &getTLSBlock();
     if (!pary.length)
     {
-        auto pbeg = cast(void*)&_tlsstart;
-        auto pend = cast(void*)&_tlsend;
-        auto p = .malloc(pend - pbeg);
-        memcpy(p, pbeg, pend - pbeg);
-        *pary = p[0 .. pend - pbeg];
+        auto p = .calloc(tdataSize + tbssSize, 1);
+        memcpy(p, tdatabegin, tdataSize);
+        *pary = p[0 .. tdataSize + tbssSize];
     }
     return *pary;
 }
 
 __gshared SectionGroup _sections;
 
+        __gshared void* tdatabegin;
+	__gshared size_t tdataSize;
+        __gshared void* tbssbegin;
+	__gshared size_t tbssSize;
 extern(C)
 {
     /* Symbols created by the compiler/linker and inserted into the
@@ -180,9 +205,6 @@ extern(C)
             void* __stop_minfo;
         }
 
-        size_t __bss_end__;
-
-        int _tlsstart;
-        int _tlsend;
+        int _tlsfiller;
     }
 }

The problem was that this scheme tries to index starting from the .tdata section, so it cannot be empty: hence the need for _tlsfiller.

Anyway, this scheme uses the existing check of /proc/self/exe in rt.backtrace.elf to get the path to the ELF binary, so it only works for executables and not for D shared libraries, which is what an Android GUI app is.

I tried using the new version in dlang/druntime#2330, but there's no program_invocation_name in Bionic and it asserts in the constructor for ElfSectionHeader on Android, even though that assert is there now and worked before that pull. I never got around to looking into why.

Alright, that's everything, I'm out.

[kinke]

Also, I have decided to leave the D community and stop using D, with the last straw being the recent censorship of criticism on the D forums. Collaborating a little with you guys on ldc is one of the few things I will miss.

Ouch, very sorry to hear that. I must have missed the censorship thing, but then again I don't use the forum that much or find it that useful.
Anyway, all the best, and I hope you reconsider sometime in the future.

[JohanEngelen]

I have decided to leave the D community and stop using D

That is sad news. You did great work for D. Best wishes and I also hope you'll reconsider.

[joakim-noah]

Thanks guys, I appreciate it. I think the work all you ldc devs have done has done a lot to legitimize D.

Regarding my decision to leave, I had some pointed criticism for DConf and its decision-making process in this forum thread, but many of my posts there were deleted. You can still find them on the email archive if you're interested.

I took the matter to the head of the moderation team and was told that I was "characterizing people" with negative words, which wasn't "professional demeanor." I pointed out that I never criticized people in those posts, only arguments they used and processes, and asked that the posts be reinstated.

No response to that and nothing was done after a week, so I wrote an email with my decision to some of the core team, which included the following rationale:

"I have a hard and fast rule from participating in online communities for a couple decades now: once criticism is being censored, I leave. I've invoked this rule more than a half-dozen times at various blogs/forums/communities over the years and I've never once regretted the decision.

The reasoning is simple: the D community cannot receive the critical feedback it needs to succeed if criticism is being deleted like this.

To me, D is just a tool, not a religion like many devs treat their programming languages. If that tool's development is being hamstrung by a bad process, I have no problem with abandoning it.

You guys and the D community have built some nice tech, but bad non-technical decisions like this will ensure it never gets very far."

I suspect that someone on the core team got fed up with my criticism, and knowing that I have been very vocal, both publicly and privately when another one of my posts was deleted before, that I would not abide by such censorship, deleted my posts on purpose to push me out. That makes a lot more sense to me than the stated reasons for deletion, but I can't prove that was their intent.

Regardless, as I alluded to in that very forum thread, I have long felt that non-technical decisions generally on this project are handled badly. In my experience, such non-technical issues actually matter more to the success of tech projects/companies than technical quality.

Also, I don't have a large codebase written in D, so there is little cost to me personally. I can chalk this up as a learning experience, and I have learned a fair amount over the years contributing to and using D, from you guys, the core technical team, and the community.

I know that everybody puts different weight on censorship and non-technical issues, so I'm not trying to convince you, just explaining my decision.

[andralex]

First off, let me repeat what we mentioned in private correspondence: we are grateful to @joakim-noah for his contributions and we're sorry to let him go. He'd be welcome to return should he change his mind.

What is described as censorship we see as a distinction in definition of terms, which sadly turned out to be irreducible. However insufficiently we act on criticism, we do have a long record of accepting it with an open mind. All we did in this instance was to kindly ask, several times, to keep the discussion professional and within appropriate bounds, especially - in fact, only - when unkind retorts to other members of the community were involved. What to some might have seemed a passionate discussion of opinions, to us it appeared unequivocally and repeatedly as transgression into the inappropriate. Clearly people can and will disagree in such matters, but we can't simultaneously agree with every opinion about what's appropriate and at the same time do what we think is right.

Again, thanks for the contributions and we are sorry to see @joakim-noah go.

[joakim-noah]

unkind retorts to other members of the community were involved

This is the crux of the matter. If you read all my posts in that thread, I only ever criticized arguments that were used, the process, or a decision once, never people as was claimed. I take pains not to personalize my criticism or insult people, yet that is the stated reason for the deletions, ie being "unprofessional" or "inappropriate."

All we did in this instance was to kindly ask, several times, to keep the discussion professional and within appropriate bounds,

This never happened, I never received any feedback on my behavior during that discussion. There was only a single public post by Walter after it was all over telling me not to insult prople, to which I responded that I hadn't. Then, several of my posts were deleted, including strangely that last post by Walter and my response.

And I even noted in private email that I was fine with some of the posts being deleted, ie my off-topic slapfight with Russel at the end. People can judge for themselves from the email archive whether the rest is worthwhile moderation of "unkind retorts" or censorship of criticism by wrongly accusing me of being unprofessional, like I claim.

[thewilsonator]

There was only a single public post by Walter after it was all over telling me not to insult prople, to which I responded that I hadn't. Then, several of my posts were deleted, including strangely that last post by Walter and my response.

Those were the ones that were deleted? I thought you were overreacting, but yeah, if I was moderating, those would have been at the bottom of my list to remove. That is really not the kind of moderation I want to see, even with the questionable signal-to-noise of the forum in general. We tend to have a lot of "robust discussion" and that was certainly not any worse than usual.

I'm sorry it has come to this. All the best for wherever life takes you, and know you are welcome back anytime.

[joakim-noah]

We tend to have a lot of "robust discussion" and that was certainly not any worse than usual.

Thanks, I appreciate your backing up what I'm saying, that I didn't go over any line. I understand that it may have felt that way to some, since I was obviously angry and frustated in a few posts and criticized others' arguments, but I try not to personalize matters and was careful not to call anyone a negative word, only arguments and processes. It's not my temperament to do otherwise, and I understand it's counter-productive.

My only request to the moderation team was that my seven or so posts that dealt with DConf be reinstated, as I don't care about the last OT posts responding to Walter or bickering with Russel. My suggested remedy would be for someone from the moderation team to put up one last forum post in that DConf thread saying that several posts were deleted but that some were mistakenly removed, then either reposting those seven posts or linking to each of them in the mail archive, which would take 5-10 mins to do.

While my own trust in the moderation team has been damaged and I likely wouldn't post in the forum much anymore, this would at least signal to me and the community that the spirit of "accepting [criticism] with an open mind" generally continues.

This might seem a trivial issue to some, but I didn't come to D because it was compiled, had great meta-programming, or some other feature checklist. No, I joined this community after years of lurking in the newsgroup, after seeing a community of open debate where Walter made many good technical points. I believe such a debate process led by good technical minds is much more important than any single decision, as that critical discussion is the best way we know to gather information.

I'm fine with bad decisions being made, particularly for non-technical matters like DConf, or I wouldn't have stuck around this long. What I won't stick around for is when the process of open debate is compromised, as that will damage everything else. That's why I leave online communities that are censored.

It was never my intent to fight with the community about DConf, or to force "my way" as some accused, but to debate the matter and then get a rationale on the decision. Once Walter gave one, I was done, which is why I didn't respond to Nicholas's last response to me.

If the remedy above is put in, I'm willing to let this go and stick with D. On the other hand, if it's decided that criticism like that won't be allowed anymore, I'll just head out. To be honest, chucking all the remaining D items off my todo list was a bit of a relief, as contributing to the language itself has become a hobby that's gotten out of control. That's what my family tells me too, a sentiment I'm sure you guys are familiar with. 😉

Despite my misgivings on some non-technical matters, I've stuck around this long because I think the core team and community has done great work on the language tech itself, as I said in my quote above from my email to the core team.

[thewilsonator]

Thanks, I appreciate your backing up what I'm saying, that I didn't go over any line. I understand that it may have felt that way to some, since I was obviously angry and frustated in a few posts and criticized others' arguments, but I try not to personalize matters and was careful not to call anyone a negative word, only arguments and processes. It's not my temperament to do otherwise, and I understand it's counter-productive.

No problem, don't worry I'm used to talking with Manu, thats just as exciting.

My only request to the moderation team was that my seven or so posts that dealt with DConf be reinstated, as I don't care about the last OT posts responding to Walter or bickering with Russel. My suggested remedy would be for someone from the moderation team to put up one last forum post in that DConf thread saying that several posts were deleted but that some were mistakenly removed, then either reposting those seven posts or linking to each of them in the mail archive, which would take 5-10 mins to do.

That seems a reasonable request.

While my own trust in the moderation team has been damaged

Fair enough

and I likely wouldn't post in the forum much anymore, this would at least signal to me and the community that the spirit of "accepting [criticism] with an open mind" generally continues.

Tbh, you're probably not missing out on a whole lot ;) at least not until after dconf when I've had a decent shot at restructuring D foundation processes e.g. splitting General into Strictly on topic and whatever is left over, and a whole host of other things.

I'm fine with bad decisions being made, particularly for non-technical matters like DConf, or I wouldn't have stuck around this long. What I won't stick around for is when the process of open debate is compromised, as that will damage everything else. That's why I leave online communities that are censored.

Indeed.

If the remedy above is put in, I'm willing to let this go and stick with D. On the other hand, if it's decided that criticism like that won't be allowed anymore, I'll just head out.

Great, I hope so too, 'cause I've got a whole lot more criticism bundled up in my list-of-things-that-need-doing-at-dconf.

To be honest, chucking all the remaining D items off my todo list was a bit of a relief, as contributing to the language itself has become a hobby that's gotten out of control. That's what my family tells me too, a sentiment I'm sure you guys are familiar with. 😉

Yep, been there. Go on, take a holiday and come back later and hopefully we'll have vision and process and all those other things we've been missing over the years.

[joakim-noah]

That seems a reasonable request... take a holiday and come back later

Alright, I guess that means my request is being considered, I will take some time away from the D community until a decision is made.

Thanks for intermediating, Nicholas. As I mentioned in one of those now-deleted posts, I feel the D community would be better off if they focused their limited resources on funding good work like you've been doing instead.

[thewilsonator]

Alright, I guess that means my request is being considered, I will take some time away from the D community until a decision is made.

I was speaking for myself there, but I hope the moderators do take notice. I'll try to remember to bring it up at the Foundation meeting if ti hasn't already been done by then.

Thanks for intermediating, Nicholas. As I mentioned in one of those now-deleted posts, I feel the D community would be better off if they focused their limited resources on funding good work like you've been doing instead.

That organisation is precisely what I plan to use dconf for ;)

[joakim-noah]

I tried raising this issue of forum censorship again over email with other members of the D team, but I'm told nothing will be done. As a result, I'm leaving the D community for good, as I feel strongly that such deletions of criticism are a big mistake.

I forgot to mention before that any unmerged patches of mine are available under the same license as the project they're for, so the LLVM Android patch is under the University of Illinois/NCSA license or whatever Apache license LLVM switches to and any dlang patches under the Boost license.

I will be writing up a postmortem of my time in the D community and posting it somewhere. As I said before, I've enjoyed collaborating with the ldc devs, goodbye.