Add -revert=noautoimports to begin transitioning to a pay-as-you-go opt-in langauge

[JinShil]

Abstract

This PR hopes to further transition D to a pay-as-you-go opt-in language as described by @andralex here. It adds a compiler switch to revert no longer automatically importing object.d (or any other module). The double negative there can mess with your mind, but it seems to me to be the right way to do this. That is, in a future PR, I will disable automatically importing object.d, and this compiler switch will revert that.

The Plan

1. This PR just adds the -revert=noautoimports compiler switch, but it does nothing for now.
2. After this PR is merged, we need to merge Add -revert=noautoimports to dmd.conf and sc.ini files installer#390 to add this switch to the compilers' default dmd.conf and sc.ini files. Due to this PR, adding the compiler switch to those files will not cause any "unrecognized switch" error.
3. After the next compiler release, all users who update to the latest compiler with have that switch in their dmd.conf file.
4. After the compiler release mentioned in 3., I will submit another PR to activate the compiler switch and disable automatically importing object.d. Since users will already have the -revert=noautoimports switch in the dmd.conf and sc.ini file there will be no breaking changes.

Background

Currently, users trying to use D in an opt-in pay-as-you-go way have to create an empty object.d file just to get a build. This is because the compiler automatically imports object.d into every file. Below is a minimal x86_64 Hello World program illustrating the problem. (See it in action at https://run.dlang.io/is/khbFfl)

module main;

alias string = immutable(char)[];

private extern(C) void __d_sys_exit(long arg1)
{
    asm
    {
        mov RAX, 60;
        mov RDI, arg1;
        syscall;
    }
}

private long __d_sys_write(long arg1, in void* arg2, long arg3)
{
    long result;

    asm
    {
        mov RAX, 1;
        mov RDI, arg1;
        mov RSI, arg2;
        mov RDX, arg3;
        syscall;
    }

    return result;
}

void write(string text)
{
    __d_sys_write(2, text.ptr, text.length);
}

private extern(C) void _start()
{
    main();
    __d_sys_exit(0);
}

void main()
{
    write("Hello, World!\n");
}

Users doing this kind of programming will compile with dmd -conf= to prevent automatically importing their system's default druntime and phobos. However, without an empty object.d the compiler will emit the following error:

Error: cannot find source code for runtime library file 'object.d'
       dmd might not be correctly installed. Run 'dmd -man' for installation instructions.
       config file: not found
Specify path to file 'object.d' with -I switch

(See it in action at https://run.dlang.io/is/oRtPbo)

Adding an empty object.d file resolves the error.

(At https://run.dlang.io adding an object.d file can be done by adding the --- object.d and --- main.d lines. See https://run.dlang.io/is/khbFfl See https://github.com/marler8997/har for more information about that syntax)

It's a bit silly to require users doing this kind of programming to create an empty object.d, and definitely not in the spirit of creating a pay-as-you-go opt-in continuum. This PR hopes to correct that and put D on a better trajectory without causing any breaking changes.

A previous attempt at this can be found at #7825

This is basically a continuation of several other PRs that have already been merged over the past 2 years:

• Don't generate ModuleInfo if it's not declared in druntime #7395 and Optional ModuleInfo (Part 2): Backend Optimizations #7768 for opt-in ModuleInfo
• Don't generate exception handling code if Throwable is not declared #7786 for opt-in Throwable
• Don't require TypeInfo in a minimal runtime if it's not needed #7799 for opt-in TypeInfo

[dlang-bot]

Thanks for your pull request and interest in making D better, @JinShil! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the annotated coverage diff directly on GitHub with CodeCov's browser extension
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + dmd#9814"

[jacob-carlborg]

How about just importing object.d when it's available and skip it when it's not? That would not require a flag.

[JinShil]

How about just importing object.d when it's available and skip it when it's not?

That was what I first proposed in #7825. It was objected to because the user would not receive a specific error indicating that there is a problem with their installation or runtime configuration. I'd be perfectly happy only importing object.d when it exists if that is preferred and is the consensus.

[jacob-carlborg]

Hmm, I see.

[JinShil]

However, we should consider what we want the opt-in continuum to look like. With this PR, it gives us a blank slate, forcing the user to explicitly import what they need. If we still allow object.d to be imported if it exists, it constrains us to a specific convention; one which currently doesn't make much sense. Most of what is in object.d has very little to do with Object. This PR wipes the slate clean and gives us a fresh start.

[WalterBright]

I have misgivings about this, as dmd should be able to compile a file without needing additional arguments beyond the filename. Not needing object.d would apply only for a very small percentage of users.

[JinShil]

dmd should be able to compile a file without needing additional arguments beyond the filename.

This PR would not change that. The user could explicilty import object; and then dmd would only require the filename to compile.

Not needing object.d would apply only for a very small percentage of users.

That is true today, but I don't think it would necessarily be true in the future.

This PR forces us to consider what we want @andralex's opt-in continuum to look like. Let's consider a couple of options:

Option 1: druntime/phobos are automatically imported, but contain ONLY TEMPLATES. This option essentially makes druntime/phobos a standard template library and a header-only library. This is one way to achieve an opt-in contiuum as you only pay for the templates that you instantiate. But modules are implicitly imported.

Option 2 (this PR): Users are required to explicitly import what they want from the language. For example, if users want support for classes, they would need to import std.classes which would include class Object, the garbage collector, and all runtime lowerings (e.g. __equals, __cmp, TypeInfo, etc.). But if the user had no intention of using classes, they could simply not import std.classes, and any attempt to use classes in their code would result in a compiler error because the support for classes wasn't imported. This option does not require everything to be a template, but does require the user to explicitly import what it is they want. To maintain the status quo, users would either need to use the -revert=noautoimports compiler flag proposed in this PR, or explicitly import object;, with that latter being preferred, IMO.

Perhaps there's another way to achieve the opt-in continuum. Now is the time to have that discussion.

[andralex]

I'm not sure I have a good understanding of this PR. My current understanding is "Currently D looks to import object.d in every compiled module, and that's not good. Providing an empty object.d is too awkward, hence this PR."

I think automatically importing something into every compiled module is not only necessary, it's essential for a good "minimize the runtime" and "pay as you go" strategy. Not having a place to provide the absolutely simplest services such as comparing/assigning two arrays would force us to provide them in the language (which runs against the notion "use the power of the language to move functionality from language magic into library space"), or require the vast majority of users and libraries to import object.d as first order of business, just because a few can't be bothered to define an empty object.d (which seems to be the wrong default, as Walter noted).

A pay-as-you-go object.d would contain everything but traditional functions (function templates are fine). That way there's no need to link anything - template instantiation takes care of it, and only on a need basis. The cost of parsing the unused templates is negligible (as measured) so it's a win in most dimensions. The disadvantage is repeated instantiation of frequently used templates (such as array comparison etc) but that scales well and we have the chance to improve it (unless e.g. the linker etc).

So I suggest we leave the automatic importing of object.d alone. The best first step would be to improve the compiler so various constructs lower to templates instead of traditional C-style functions.

[JinShil]

My current understanding is "Currently D looks to import object.d in every compiled module, and that's not good. Providing an empty object.d is too awkward, hence this PR."

No, that's not it. It's more "Those using D without importing object.d are the pioneers of D using the language in a pay-as-you-go fashion, so understanding their use case will help reveal the best way forward for D".

That way there's no need to link anything - template instantiation takes care of it, and only on a need basis.

So it appears you favor my "Option 1" above. Option 1 requires the compiler distribution to place Phobos and druntime in it's import path. The compiler then implicily imports object.d (or something equivalent) that makes all essential features of the language available for use. However, all of those essential features are templated, so the pay-as-you-go, opt-in continuum is achieved by only paying for templates that the user instantiates. Essentially druntime becomes a header-only, template library. Is this what you, @andralex, are envisioning at the moment? If not, please elaborate futher.

Not having a place to provide the absolutely simplest services such as comparing/assigning two arrays would force us to provide them in the language

That's not true. What I'm proposing with this PR is that if users want a specific feature of the language they need to explicitly import it. With the current state of things, that means they would need to explicitly import object.d or use this PR's new compiler flag to automatically import it.

This removes the technical debt in D, and wipes the slate clean allowing us to move forward implementing a new pay-as-you-go, opt-in, united Phobos/druntime library. That new library, let's call it Phobos2, would implement something like a language feature stack allowing users to import only what they need. I'm not entirely sure what that looks like at this time, but to help convey the vision I'll propose a hypothetical below:

• types.d -- contains the most basic types of the language (e.g. string, size_t, etc.)
• structs.d -- publicly imports types.d, but also contains any implementations and compiler lowerings needed to support D's structs.
• staticArrays.d -- publicly imports types.d and also contains any implementations and compiler lowerings needed to support D's static arrays.
• gc.d -- publicly imports types.d and also contains any implementations and compiler lowerings needed to support D's garbage collector
• dynamicArrays.d -- publicly imports gc.d and also contains any implementations and compiler lowerings needed to support D's dynamic arrays
• classes.d -- publicly imports and gc.d and also contains any implementions and compiler lowerings to support D's classes
• exceptions.d -- publicly imports classes.d and also contains any implementations and compiler lowerings to support D's exception handling

Reality would probably look different, but that's the idea. Users would then explicitly opt-in to whatever point in that stack they're willing to pay for, and that's how the opt-in continuum is achieved. Attempting to use a language feature without importing the necessary module providing the requisiste implementations would result in a compiler error.

---

This PR disables automatically importing object.d (or any module/package for that matter). It adds a new compiler switch simply to avoid breaking code. With this simple change, D becomes a pay-as-you-go language, starting with an empty, blank slate. An empty main() does not import or link in anything, and that starts the opt-in continuum. To move forward from there, users need to import the features they intend to use.

Here are a few advantages of Option 2 over Option 1:

• It doesn't requires us to make everything a template
• There would likely be compile-time benefits as the compiler would not be required to import and parse the entire house when the user only wanted a closet.
• The library distribution can include compiled code so it doesn't need to be re-compiled every time the user does a build.
• Phobos/druntime/Phobos2 become just like any other library in the D ecosystem. It can even be hosted on DUB or any other library distribution system. Users just add it to their project as a dependency just like any other library
• Users can create and import Phobos/druntime alternatives hosted on DUB/GitHub/Other to support new platforms (e.g. bare-metal, microcontrollers, RISV, Arduino, ARM, Raspberry Pi, etc.) or new problem domains (e.g. no GC, C-like features only -- no more need for a -betterC switch). They just add the platform they want as a dependency, and import what they intend to use.
• It would allow us to move forward without breaking changes, because everything is opt-in by importing. That is if users want to use the old Phobos/druntime they explicitly import it. If users want to use the std.v2, Phobos2, or whatever else we decide on, they explicitly import that instead.
• Distributing Phobos/druntime/Phobos2 on DUB would have a number of different benefits. Instead of relying on @MartinNowak to maintain build and release all supported platforms, we could assign a maintainer to each platform, reducing our reliance on one single person, and leverage the platform expertise of a maintainer that specializes in that platform.

I get the feeling I'm trying to do something revolutionary, though I didn't really think that when I submitted this PR. Do I need to write a manifesto to help articluate the vision?

[andralex]

I get the feeling I'm trying to do something revolutionary, though I didn't really think that when I submitted this PR. Do I need to write a manifesto to help articluate the vision?

I am sorry, to me it seems that eliminating the import of object.d is not a meaningful step toward any vision. Would it be appropriate if you made a salient improvement using an empty object.d as a workaround? It seems like a small effort that wouldn't impede showcasing the revolution.

[andralex]

Upon a bit more thought, a good one-liner is: "Pay as you go does not entail import as you go." The two should not be conflated. Importing a module with templates at the ready is essentially cost-free, and that's an important realization in designing with D.

[WalterBright]

Another consideration is compile speed. Importing files is gated by the time spent doing the file lookup. Splitting object.d into numerous smaller imports will negatively impact compile time.