fix Issue 10378 - Local imports hide local symbols

[WalterBright]

This is a fix of https://issues.dlang.org/show_bug.cgi?id=10378 and a reboot of #4915

It requires dlang/phobos#3989

This version is based on a comment Andrei made to me a long time ago about it. Symbol lookup is now done in two passes (formerly one). The first pass goes up through the scopes, checking everything but imported modules. If that fails, the scopes are gone through again, checking only imported modules.

This algorithm:

1. retains the scoped nature of nested imports, i.e. names in nested imports will hide outer imported names
2. works the same for function locals as it does for struct members, which was my complaint about Issue 10378 - Local imports hide local symbols #4915
3. does not issue errors for shadowing - it simply finds the shadowed symbol first.
4. breaks existing code. Turns out that people did rely on it. See fix problems that prevent implementing import shadowing protection phobos#3989
5. is pretty simple. (the large number of lines of code changed is mostly due to debugging and printing code which I'll remove once it passes the autotester
6. exposed a bug in the way the compiler is implemented, see fix problems that prevent implementing import shadowing protection phobos#3989 for that, too. I'm sure there'll be other instances in the wild of this.
7. it should only be a hair slower, as it never looks in the same hash table twice

[dlang-bot]

Fix	Bugzilla	Description
✓	10378	Prevent local imports from hiding local symbols

[WalterBright]

Looks like @deadalnix deadalnix proposed this algorithm here:

https://issues.dlang.org/show_bug.cgi?id=10378#c9

[ibuclaw]

Remove version none and move initialising sx into version LOGSEARCH?

[WalterBright]

that code will be removed when it passes the autotester

[deadalnix]

Yes, that's how it needs to be !

[dnadlinger]

We should make fixing this and issues 313/314 the main point of this release.

[WalterBright]

Needs this Phobos fix:
dlang/phobos#3990

Folks, this language change is clearly going to break existing code.

[andralex]

nice work!

[andralex]

Auto-merge toggled on

[schveiguy]

When this is merged, I think we need a doc change too.

[9rnsr]

Auto-merge toggled off

[9rnsr]

I want to review this.

[9rnsr]

A module bar; is imported by import imports.bar10378;. It's a mismatch.

I'm not sure why the error is not reported, but at least, this line should be: module imports.bar10378;.

[WalterBright]

That's actually a feature of D. The file name doesn't have to match the module name.

[9rnsr]

I know the feature, but this is not its case. This file is imported as the module name imports.bar10378, but the module declaration has a different fully qualified name bar which does not match.

[MartinNowak]

That's actually a feature of D. The file name doesn't have to match the module name.

It's actually the result of a horribly broken implementation. The module is inserted w/ both names into the global symbol table.

I'm not sure why the error is not reported

The error is only reported if the module is know under the module declaration name before importing it, this happens when it's parsed before the module importing it.

[9rnsr]

@WalterBright : Also I give you a chance to remove debugging code from this PR.

[Geod24]

@WalterBright : Is there any chance we get a deprecation for the old behaviour ?

I agree that is the right direction to go, and that old code relied on a bug. However, that code was not necessarily broken / buggy, which is why I don't think breaking it without a proper deprecation is okay.

[9rnsr]

It's better to move to compilable directory, rather than deleting test case for issue 12598.

Of course adding a comment about that in commit log is even better.

[WalterBright]

It doesn't test for anything novel anymore and serves no purpose. Hence, deleting it.

[9rnsr]

You should record into the git history what happened in this PR. Moving the test case to compiable would be the best clarify way that the issue 12598 case is now just accepted.
If you don't like it, you should add a comment into the commit message, like "issue 12598 case is now properly accepted, so its test case fail_compilation/diag12598.d is deleted.

[9rnsr]

The most of PR summary needs to be copied into commit log, no?

[9rnsr]

Excepting my comments, the implementation itself looks good to me.

[WalterBright]

add another switch

Please, no:

1. complexity
2. documentation
3. it is not as easy as you suggest - note that every override of search() changed the flags.
4. if a user sees new "undeclared symbol" messages, he can simply add the -transition=import switch and if the error goes away, he knows that's what the problem is.

BTW, doing a fork has more problems. This change touches a lot of files, and I've had to rebase it twice in 4 days just so it will merge. I'm not keen on the time investment in this.

[schveiguy]

I don't have much to offer here, except support for this change. As much as this will break code, people have been clamoring for import issues to be fixed for almost a decade. I think folks will understand this is going to help them even though their code breaks.

Regarding build issues, I think those are somewhat secondary to the ability to have code that compiles for both older and newer versions of the compiler. Simply because the two alternative compilers lag behind DMD. So as long as there is a way to fix broken code so it compiles for the new scheme and the old scheme, I'd say passing the right flags is a secondary concern.

[MartinNowak]

Your arguments are correct, the very few issues this caused ¹ relied on rather obscure behavior, and the new lookup rules are a huge improvement.
The code is also clearer now, and we have a nice explanation in the changelog.

[MartinNowak]

Auto-merge toggled on

[JackStouffer]

Auto-merge toggled on

Thanks Walter for doing this!

[schveiguy]

I have a question on this. If I want to have an inner scope where I add an overload via an import, how would I do it? Currently, you have to re-import the original so they are on the same level. From my reading of the new rules, this would continue to be the case.

An example of what I'm talking about is in the bug report this PR is fixing (at the end of the comment): https://issues.dlang.org/show_bug.cgi?id=10378#c16

[WalterBright]

Currently, you have to re-import the original so they are on the same level.

That's right (and this PR does not change that behavior).

Edit: Functions that exist in two imports do not actually overload against each other. They form separate "overload sets". The anti-hijacking rules keep them distinct. The only way you'd want to overload against both is if the scope wants to actually call both, in which case it makes sense to require re-importing just for clarity.

[WalterBright]

So as long as there is a way to fix broken code so it compiles for the new scheme and the old scheme,

I had no problems doing that for the Phobos library breakage.

[schveiguy]

Right. The only confusing part is to require importing the current module you are in (because another module hijacked a module symbol you were using).

Interestingly, this doesn't happen when you import at module level. The current module trumps any imports. But when you import at a scope, then the rules change. As a user of library code, I can see value in importing a module locally when it only pertains to that particular function. What I'm concerned with (and am still, even though I like the PR as merged), is that a movement of an import from the module level to a scoped level changes what happens.

example:

module a;

void foo() {import std.stdio; writeln("a");}

void bar() {}

module b;
import a;

void foo() {import std.stdio; writeln("b");}

void main()
{
    bar();
    foo();
}

This outputs "b"

Hey, look, we can just import a locally in main, because we only need bar in there!

module b;

void foo() {import std.stdio; writeln("b");}

void main()
{
    import a;
    bar();
    foo();
}

This now outputs "a".

I'm unsure of the "fix" for this, but it may be good practice to recommend always importing within an inner scope using renamed imports (to avoid hijacking like this).

[WalterBright]

@schveiguy Your second example now outputs b with this PR. This fix is the whole point of this PR.

[schveiguy]

our second example now outputs b with this PR

Oh right! I wasn't thinking about the fact that local module functions aren't actually imported :)

So no need to reimport the current module if the current module's function is what you want. This is great! Thanks.

[CyberShadow]

Could someone please tell me if this breakage is intentional or should I file a regression?

int memcmp(in ubyte[] a, in ubyte[] b)
{
    import core.stdc.string;
    assert(a.length == b.length);
    return memcmp(a.ptr, b.ptr, a.length);
}

This no longer works because it tries to call itself - not what I would expect TBH, seems like a bug to me.

[schveiguy]

I think it's intentional, as imports now are considered after local module functions.

I disagree that it's a bug. If you imported core.stdc.string in the module, the imported memcmp wouldn't be considered.

[CyberShadow]

If you imported core.stdc.string in the module, the imported memcmp wouldn't be considered.

That would not be surprising. The behavior in the above snippet is surprising, though, because the import is "closer" (in scope terms) to the use of the memcmp symbol than the function declaration.

Changing the import to a selective import makes it compile again, which is weird, as I always thought that selective imports are simply selective versions of imports, and do not grant additional visibility to the selected symbols.

I kind of see why this change was necessary, though, but the additional complexity and code breakage isn't nice.

[schveiguy]

Yes, selective import actually aliases the symbol (and only the one you selected) into your local scope, which would override any other imported symbols and local symbols.

It is going to be very confusing, and code-breaky, because it's been this way for so long. I don't think there's a way we could do this without breaking code.

[schveiguy]

Note: I always looked at it the opposite way -- that importing in a local scope should not change how the import was done, just what scopes could see the import. Moving an import inside a function shouldn't change the behavior of the function, you are just trying to avoid polluting namespaces (you can see my example above).

[ibuclaw]

@schveiguy are you sure? core.stdc.string.memcmp has a different signature to the function that imports it's module. I can see no reason why that snippet would even consider recursively calling itself.

[WalterBright]

Could someone please tell me if this breakage is intentional or should I file a regression?

It's behaving exactly as intended.

has a different signature

Signatures are considered for overload resolution, not for name lookup. D has always worked this way, and so has C++.

[MartinNowak]

Turn on -transition=checkimports, it should give you a proper deprecation warning about the changed lookup.

[CyberShadow]

Turn on -transition=checkimports, it should give you a proper deprecation warning about the changed lookup.

I am seeing new deprecation warnings even without this flag (using dmd master). Is the flag meant for stable and the breaking changes for master, or something else?

[CyberShadow]

It's behaving exactly as intended.

OK. Another question: does a selective import not imply a static one? E.g.:

import std.stdio : readln;
import std.conv;

void main()
{
    std.stdio.writeln("Hello, world!");
}

This now gives:

test.d(6): Deprecation: module std.stdio is not accessible here, perhaps add 'static import std.stdio;'

but that strikes me as redundant considering the selective import.

[schveiguy]

does a selective import not imply a static one?

No it was not supposed to. But this is due to a different pull:

#5426