Fix Issue 17581 - Document behavior of -betterC

[wilzbach]

I couldn't find much about -betterC, the best resource was Ilya's guide:
https://github.com/libmir/mir-algorithm/wiki/BetterC-for-Mir

CC @WalterBright @andralex: would be great if we could document the current behavior.

[dlang-bot]

Thanks for your pull request, @wilzbach!

Bugzilla references

Auto-close	Bugzilla	Description
✓	17581	Document behavior of -betterC

[adamdruppe]

I think it is important to make sure the distinction between dynamic arrays and slices is clear. Slices still work (aside from some typeinfo bugs in some cases, but we can fix those).

[CyberShadow]

Thanks for taking the initiative!

Also, you're spoiling the compiler guys. Soon they'll be expecting you to document all their changes :P

[PetarKirov]

That's wrong. shared has as many runtime/link-time dependencies as const - none!
Currently it's purely a type-system feature - it disappears when you compile your code, except for symbol mangling.

[PetarKirov]

Only regular D classes don't work. I think there are no obstacle to using extern (C++) and extern (Windows) (i.e. COM) classes.

[adamdruppe]

They still emit references to Typeinfo_Class, even if not D. Again, this might change in the future, but not right now.

[PetarKirov]

They still emit references to Typeinfo_Class

But that's good! How else would you initialize instances of them? At minimum, the vtbl ptr can't be initialized in the constructor by user code (easily). Until we move all of TypeInfo* to traits / templates that lazily pull only what's needed, we still need Typeinfo.

I tried the following and it seems to work well.

betterC is a spectrum - yes if you're doing bare-metal development you really don't want those stuff, but if you're writing a plugin and you don't want to carry druntime, some of the TypeInfo stuff can still be useful. So when talking about betterC, I think we need to differentiate between regular user-land apps / libraries that still rely on libc, apps / libraries that don't rely on libc (doing system calls themselves) and real bare-metal development.

---

P.S.:

After playing a bit, I saw what you meant. I (wrongly) expected that the emitted type info into the binary would be self-contained. Unfortunately that's not the case. The following code:

extern (C++) class C { }
extern (C) void main() { }

Produces a undefined reference to _D14TypeInfo_Class6__vtblZ:

$ dmd -c -betterC extern_cpp_class.d && nm extern_cpp_class.o
0000000000000000 t
                 U _D14TypeInfo_Class6__vtblZ
0000000000000000 V _D16extern_cpp_class1C6__initZ
0000000000000000 V _D16extern_cpp_class1C6__vtblZ
0000000000000000 V _D16extern_cpp_class1C7__ClassZ
0000000000000000 W main

Also while experimenting with that I ran into this bug.

Also extern (Windows) is much worse than extern (C++) classes:

extern (Windows) class C { }
extern (C) void main() { }

$ dmd -c -betterC test.d && nm test.o
0000000000000000 t
                 U _D14TypeInfo_Class6__vtblZ
0000000000000000 V _D16extern_cpp_class1C6__initZ
0000000000000000 V _D16extern_cpp_class1C6__vtblZ
0000000000000000 V _D16extern_cpp_class1C7__ClassZ
                 U _D6object6Object5opCmpMFC6ObjectZi
                 U _D6object6Object6toHashMFNbNeZm
                 U _D6object6Object8opEqualsMFC6ObjectZb
                 U _D6object6Object8toStringMFZAya
                 U _D6Object7__ClassZ
                 U _d_dso_registry
0000000000000000 W main
                 U __start_minfo
                 U __stop_minfo

---

So scratch everything I said, extern (C++), or worse extern (Windows) classes are really not usable without linking druntime with your object files.

[adamdruppe]

Yeah, TypeInfo is useful but this emits a reference to the base class which is found in druntime.

What I think we should do is to massively simplify the typeinfo so the runtime library defines it. The compile only outputs the bare minimum for classes. So when it sees class Foo {}, the compiler basically does

class TypeInfo_Foo : TypeInfo_Class {
   override TypeInfo_Class base() const { return whatever_symbol; }
   override immutable(Interface[]) interfaces const() { return whatever_generated_static_data; }
}

Something along those lines - the minimum needed for dynamic casting. Let the rest be defined with rtInfo and the library-defined base class. I didn't even put the initializer in here since that is static data the compiler generates and can be added via rtInfo template to typeinfo if desired, but you can just use something similar to __traits(classInstanceSize) (the actual data doesn't exist via traits now, but it could).

Anyway, much -betterC probably wouldn't want to use classes since they do require a runtime module and you don't want to provide your own out of fear of ABI incompatibilities. But with this scheme, it would be pretty easy to opt in to as little or as much info as you want.

[PetarKirov]

I completely agree.

[PetarKirov]

Which reminds me that I need to move the core of the implementation of core.sync.mutex into a struct so it is usable in betterC for regular user-land programs that link libc.

[adamdruppe]

Note that struct destructors do NOT work right now but I have a PR that hacky fixes them: dlang/dmd#6923 (walter wants a test case but idk how to do that) so it is possible to have

[PetarKirov]

That's a bummer, though since you can't define a parameter-less constructor on a structs, you can't mimic C++11's std::mutex anyway. I'm leaning towards defining empty (default) ctor and dtor and two methods create() and release() , which is obviously not very safe, but it's meant to be a low-level implementation wrapped by a higher-level one like core.sync.mutex.Mutex or a factory of some sort (with good enough API reusing mutexes can be both safer and faster).

[PetarKirov]

I think that's also wrong. I guess you took it out of Ilya's design guide for mir too literally.

[wilzbach]

I actually tried this quickly with -betterC:

void main() {
    import core.stdc.stdio : printf;
    import core.math : cos;
    printf("Hello world: %.2f", cos(2));
}

and got a segfault. However, I simply forgot extern(C) for the main method.

[PetarKirov]

No, extern(C) has nothing to do with that. extern(C) comes into play only when you want to use libraries with C interface, or you want to expose a C interface of your code. -betterC doesn't require any of that - your whole code can use the extern (D) (the default) function calling and name mangling convention for the whole library / application without any problems. The compiler automatically generates an extern (C) main() which calls your D main function.
The problem is that core.math functions use real (which may be 64 or 80-bit in size) while printf 's %f format assumes that its argument is 32-bit. You either need to cast the result to float:

void main()
{
    import core.stdc.stdio : printf;
    import core.math : cos;
    printf("Hello world: %.2f\n", cast(float)cos(0));
}

$ dmd -betterC use_core_math.d && ./use_core_math
Hello world: 1.00

Or you need to use a different conversion format specifier like %Lf:

void main() {
    import core.stdc.stdio : printf;
    import core.math : cos;
    printf("Hello world: %.2Lf\n", cos(0));
}

$ dmd -betterC use_core_math.d && ./use_core_math
Hello world: 1.00

Though, since it's not clear what the size of real would be on the target libc, it's better to not rely on that. I think Ilya also avoids using real, for performance reasons (SSE is faster than x87) and for precision (avoiding unnecessary rounding when going from FP register to memory and back or conversion to double), etc.

[adamdruppe]

Indeed, core.math did work for me too.

[PetarKirov]

I think @WalterBright is working on making this work with betterC. (At least shared static constructors, by using C++'s static constructors.)

[PetarKirov]

generated -> generates

[PetarKirov]

"barebone, runtime-free" -> Druntime free
(The libc runtime is still linked, as far as I know.)

[jpf91]

The libc runtime is still linked, as far as I know.

I'm not sure if we emit any calls to C runtime functions directly from the compiler though. memcpy and memset are likely candidates.

[PetarKirov]

I'm not sure either, but one simple litmus test is that the compiler still calls D applications' main from extern (C) main , as opposed to _start (on Linux). I.e. it expects the C runtime to perform its initialization.

[adamdruppe]

Yes, Unwind_Resume and __assert are both emitted by the compiler almost always. It assumes a C lib, but it is fairly easy to pull that out too for special uses. Read my post here: http://forum.dlang.org/post/cwzmbpttbaqqzdetwkkf@forum.dlang.org

[PetarKirov]

`printf("Hello betterC\n");

[PetarKirov]

This should be at the top, and the list should be ordered like this:

1. GC
2. Thread local storage (not sure about that)
3. TypeInfo and ModuleInfo
4. extern(D) classes, because of 3) and in part because of 1) (the new expression, though there are various ways around that).
5. No built-in threading (a consequence of 4), i.e. no core.thread, std.concurrency, std.parallelism, vibe.d, etc.
6. Dynamic arrays (but not slices) and associative arrays
7. Exceptions
8. switch with strings
9. final switch and other assert-like features like bounds checking (@WalterBright please correct me, if I'm wrong)
10. synchronized
11. For reference on all (I think) the runtime hooks, see: https://github.com/dlang/dmd/blob/master/src/ddmd/backend/rtlsym.h

[adamdruppe]

Assert features are supported (imo even better than they are without -betterC! LOL) See: http://arsdnet.net/this-week-in-d/2017-jul-02.html for range checks specifically: dlang/dmd#6927

Exceptions are no right now, but might change. I actually wouldn't strictly document this since much of it is subject to change.

[wilzbach]

@ZombineDev @adamdruppe @jpf91: thanks a lot for your help. As you might have noticed, it's not very well know what -betterC actually implies for the end user - hence the need for this page.

No static module constructors or deconstructors
I think @WalterBright is working on making this work with betterC. (At least shared static constructors, by using C++'s static constructors.)
I actually wouldn't strictly document this since much of it is subject to change.
Note that struct destructors do NOT work right now but I have a PR that hacky fixes them

@adamdruppe The idea is to document the status quo. Future improvement can always update this page.

[wilzbach]

extern(C) void main() {}
unittest {
    import core.stdc.stdio : printf;
    printf("Hello world: \n");
}

> dmddev -betterC -unittest -run foo.d

(no output)

[PetarKirov]

Since you're not linking druntime there would be no test runner to run the unittests. In theory you could write your own, though I don't know if that's feasible. Instead what I think we should explain is that you can use unittests to test the functions you're using in -betterC mode by building without -betterC your unittest builds.
Edit:
Of course you won't be testing that the functions would behave properly in a -betterC environment, but assuming there are no linking errors, it's probably good enough.

[PetarKirov]

Turns out that it's not hard to get them running:

test.d:

extern(C) void main()
{
    foreach (test; __traits(getUnitTests, mixin(__MODULE__)))
        test();
}

// Thanks for the tip, Adam!
version(linux) extern(C) __gshared void _d_dso_registry() {}

unittest
{
    import core.stdc.stdio : printf;
    printf("Hello unit tests!\n");
}

$ dmd -c -betterC -unittest test.d && gcc test.o && ./test
Hello unit tests!

[PetarKirov]

My investigation showed that indeed currently classes are not usable (at least without hacks) without linking druntime, so unfortunately you need to remove this line.

[PetarKirov]

@wilzbach of course, thanks for doing this! I also think that documenting the current status quo is very important.

[PetarKirov]

Sorry nitpick again about this paragraph, but I had a bit more time to think about it and I came up with this:

-betterC is a command-line flag for dmd, which restricts the compiler's support of certain runtime features. Notably, D programs or libraries would not be linked with Druntime. The use of compile-time features is not restricted in any way. Limiting a program or library to this subset of runtime features is useful when targeting constrained environments where the use of such features is not practical or possible.

Additionally, this is also makes embedding D libraries in larger projects easier by:

• Simplifying the process of integration at the build-system level
• Removing the need to ensure that Druntime is properly initialized on calls to the library, when an initialization step is not performed before the library is used.
• Mixing memory management strategies (GC + manual memory management) can sometimes be tricky, hence removing D's GC from the equation may be good solution

Most probably this can be made more clear or informative, so feel free to improve it.

[wilzbach]

Most probably this can be made more clear or informative, so feel free to improve it.

Adapted it a bit, but I think it's an excellent motivation for betterC. Thanks!

[WalterBright]

-betterC is not code, it should be wrapped $(CONSOLE -betterC)

[WalterBright]

extern (C++) classes will work

[PetarKirov]

I would rather advertise they work when they actually do so. See: #1796 (comment)

After we get them working, changing this would be trivial.

[WalterBright]

this will work

[WalterBright]

Approved with requested changes

[jacob-carlborg]

Should work if the platform natively supports it?

[PetarKirov]

You need Druntime's rt/sections* for TLS, AFAIK. Since -betterC means not linking Druntime, it follows that TLS is not supported with -betterC.

[jacob-carlborg]

The platform is doing the setup/initialization of the TLS data. What D needs to do is extract the TLS ranges to add them as roots of the GC. Not sure if it's possible to use some D specific type that requires additional setup by the D runtime. But all types that exists in C and can be used as TLS variables in C should work.

[joakim-noah]

Yeah, Jacob is right, all rt.sections does with native TLS is register it with the GC. Take a look at the much simpler one for Solaris to confirm.

[MartinNowak]

Yes, Android should be the last platform where we still use emulated TLS.

[joakim-noah]

That shouldn't matter though, as it's only a couple functions, so you could just take that emulation out of druntime and ship it easily with your -betterC code.

[WalterBright]

I'm going to merge this, because it's needed despite its flaws.

[PetarKirov]

@WalterBright please try to hesitate merging pull requests that fail to build in the future. The ddoc build of pull requests is automatically tested with Vladimir's DAutoTest, and you can see the result at the bottom of each pull request, including this one:

As you can see this PR fails to build indicated by the red cross. You can see the result of the build by clicking on the Details link to the right. Normally, if the documentation for a PR built successfully, you would see:

If a pull request introduces changes to the website content, you should see them under the "Changes" section, where for each of the affected pages there will be "Old" and "New" links pointing to the current version and to the one proposed by the pull request, respectively.

For pull requests that fail to build, you can try to find why by opening the "Build log". Most often errors are at the bottom of the build log. In this particular case:

make[1]: Leaving directory '/dev/shm/dtest/work/repo/dlang.org/.generated/phobos-release'
../dub-1.1.0/bin/dub upgrade --missing-only --root=dpl-docs
Upgrading project in /dev/shm/dtest/work/repo/dlang.org/dpl-docs
DFLAGS="-conf=/dev/shm/dtest/work/repo/dlang.org/.generated/stable_dmd-2.072.2/dmd2/linux/bin64/dmd.conf -L--no-as-needed" ../dub-1.1.0/bin/dub build --root=dpl-docs \
	--compiler=.generated/stable_dmd-2.072.2/dmd2/linux/bin64/dmd
Performing "$DFLAGS" build using .generated/stable_dmd-2.072.2/dmd2/linux/bin64/dmd for x86_64.
hyphenate 1.1.1: target for configuration "library" is up to date.
libdparse 0.7.0: target for configuration "library" is up to date.
vibe-d:utils 0.7.31: target for configuration "library" is up to date.
vibe-d:data 0.7.31: target for configuration "library" is up to date.
vibe-d:core 0.7.31: target for configuration "libevent" is up to date.
vibe-d:http 0.7.31: target for configuration "library" is up to date.
vibe-d:diet 0.7.31: target for configuration "library" is up to date.
vibe-d:web 0.7.31: target for configuration "library" is up to date.
ddox 0.15.18: target for configuration "library" is up to date.
dpl-docs ~master: building configuration "application"...
To force a rebuild of up-to-date targets, run again with --force.
Searching for trailing whitespace
Searching for undefined macros
/dev/shm/dtest/work/repo/dlang.org/web/spec/betterc.html:105:<style>    body { counter-reset: h1 <!--UNDEFINED MACRO: "CHAPTER"-->; counter-increment: h1 -1; }
posix.mak:646: recipe for target 'test' failed
make: *** [test] Error 1
Test failed: object.Exception@ae/sys/d/manager.d(801): Command ["timeout", "1800", "make", "-f", "posix.mak", "DIFFABLE=1", "test"] failed with status 2
----------------
/usr/local/src/phobos/std/exception.d:420 pure @safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x7ec406]
/usr/local/src/phobos/std/exception.d:388 pure @safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x7ec476]
ae/sys/d/manager.d:801 void ae.sys.d.manager.DManager.Component.run(const(immutable(char)[])[], const(immutable(char)[][immutable(char)[]]), immutable(char)[]) [0x7d70c8]
ae/sys/d/manager.d:1667 void ae.sys.d.manager.DManager.Website.make(const(immutable(char)[][])) [0x7dd43b]
ae/sys/d/manager.d:1698 void ae.sys.d.manager.DManager.Website.performTest() [0x7dd752]
ae/sys/d/manager.d:648 void ae.sys.d.manager.DManager.Component.test() [0x7d659d]
ae/sys/d/manager.d:2020 void ae.sys.d.manager.DManager.test() [0x7df6af]
autotest.d:244 autotest.main().Result autotest.main().runBuild(immutable(char)[], int, immutable(char)[], immutable(char)[]) [0x743fec]
autotest.d:346 _Dmain [0x742b66]
[2017-08-18 01:24:30.160863] dautotest: Error: Test failed
[2017-08-18 01:24:30.168329] dautotest: Setting status for dlang.org pull 1796 (commit c7b3c834273118264bc79c7784c6703196b42696) to failure (Test failed): http://dtest.dlang.io/results/2e94dab2be1338bd05d362ce62f1f0ad31fdeaa4/c7b3c834273118264bc79c7784c6703196b42696/
[2017-08-18 01:24:30.542626] dautotest: ----------------------------- Log end -----------------------------

[joakim-noah]

I just experimented a bit with -betterC on the hello world from the testsuite, with extern (C) prepended before main, dmd -betterC hello.d. I guess varargs also doesn't work with the arguments passed into it, as it prints out junk data from the environment variables instead, worth mentioning no varargs in the spec?

[adamdruppe]

That hello world does not have the correct signature for an extern(C) main. It should be main(int argc, char** argv,) (with const if you want) and perhaps with char** envp at the end.

char[][] is a D type and not directly compatible with C's array arguments.