Skip to content

Fix Issue 11997 - rdmd should search it's binary path for the compiler#250

Merged
dlang-bot merged 1 commit intodlang:masterfrom
joakim-noah:rdmd
Jul 17, 2017
Merged

Fix Issue 11997 - rdmd should search it's binary path for the compiler#250
dlang-bot merged 1 commit intodlang:masterfrom
joakim-noah:rdmd

Conversation

@joakim-noah
Copy link
Copy Markdown
Contributor

@joakim-noah joakim-noah commented Jul 12, 2017

rdmd should work out of the box now, without having to add it to your PATH, tested with dmd and ldc, which just started shipping with rdmd. Would be nice to get this in before the final 2.075 release of dmd, pinging @MartinNowak.

Only drawback I can see is that if someone takes rdmd and puts it in some arbitrary directory with a file that's happened to be named dmd or ldmd2, rdmd will now try to run it. Seems far-fetched, but the real solution is to just integrate rdmd with dmd and ldc, so this external script doesn't always have to go find them.

@dlang-bot
Copy link
Copy Markdown
Contributor

dlang-bot commented Jul 12, 2017

Thanks for your pull request, @joakim-noah! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.

Some tips to help speed things up:

  • smaller, focused PRs are easier to review than big ones

  • try not to mix up refactoring or style changes with bug fixes or feature enhancements

  • provide helpful commit messages explaining the rationale behind each change

Bear in mind that large or tricky changes may require multiple rounds of review and revision.

Please see CONTRIBUTING.md for more information.

Bugzilla references

Auto-close Bugzilla Description
11997 rdmd should search its binary path for the compiler

@dlang-bot dlang-bot added the Type.Bug Things don't work as they were intended to, or the way they were intended to work doesn't make sense label Jul 12, 2017
rdmd.d Outdated
//writeln("Invoked with: ", args);
string binPath = thisExePath();
if (binPath.exists && binPath.isFile && filenameCmp(baseName(binPath),
baseName(args[0])) == 0)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these sanity checks necessary? Even if not, may be worth leaving in for now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imho they aren't necessary, just confuse the reader and prevent symlinks, so I would suggest to remove them.
What do you think @CyberShadow?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is the right approach. Regardless, the last check may fail on Windows or other systems with case-insensitive filesystems.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you mean about the approach, but filenameCmp claims to be able to handle case-sensitivity.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that's right. Still the matter of the extension.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what you mean about the extension, currently compiler does nothing with extensions. It simply passes dmd, ldmd2, or gdmd to spawnProcess.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C:\Temp\2017-07-12>cat test.d
import std.stdio;
import std.file;

void main(string[] args)
{
        writefln!"thisExePath = %s"(thisExePath);
        writefln!"args[0] = %s"(args[0]);
}

C:\Temp\2017-07-12>dmd test.d

C:\Temp\2017-07-12>test
thisExePath = C:\Temp\2017-07-12\test.exe
args[0] = test

Copy link
Copy Markdown
Contributor Author

@joakim-noah joakim-noah Jul 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok, you didn't say what the problem was, so I thought you were referencing something like the wrapper script you mentioned below. I guess this means the last check will fail on Windows, and it'll fall back to the dmd from PATH instead. I'll just remove this third check, doesn't do much even when it works.

Copy link
Copy Markdown
Contributor

@wilzbach wilzbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to get this in before the final 2.075 release of dmd

You would need to rebase onto stable.

This feature makes sense to me in general, but it would be nice to leave a comment about the motivation of tring to find the defaultCompiler executable in the directory of the rdmd executable.

rdmd.d Outdated
if (binPath.exists && binPath.isFile && filenameCmp(baseName(binPath),
baseName(args[0])) == 0)
{
string compilerPath = buildPath(dirName(binPath), defaultCompiler);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: binPath.dirName.buildPath(defaultCompiler) would be more phobosque ;-)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but I prefer the old-fashioned style for this, reads better to me.

Copy link
Copy Markdown
Member

@CyberShadow CyberShadow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change should take effect only if rdmd is unable to otherwise find a compiler binary - i.e. if --compiler is unspecified or does not contain any path components, AND the compiler binary is not found in PATH.

On Windows, you will need to mind the PathExt environment variable, as users may be expecting rdmd to run a compiler wrapper script they have placed in PATH.

This is also lacking a test.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

I think this change should take effect only if rdmd is unable to otherwise find a compiler binary - i.e. if --compiler is unspecified or does not contain any path components, AND the compiler binary is not found in PATH.

--compiler overrides this and the point of setting PATH is almost always to use the compiler binary in the directory with rdmd. Right now, if the compiler is somewhere else, say if you download the tarfile and uncompress it but also have dmd in your system path, it will invoke the one from the system path, not the local one, which is very confusing. That's why I tried to use the compiler right next to rdmd first but put in all kinds of checks, so it can fall back to the PATH if anything isn't right.

On Windows, you will need to mind the PathExt environment variable, as users may be expecting rdmd to run a compiler wrapper script they have placed in PATH.

I don't mess with the one in PATH at all, so I'm not sure how that env variable is relevant. Yes, if the user has some weird setup where the first hit in their PATH is some compiler wrapper that's in a completely different location, this will override that. I think that's worth it, as rdmd is meant for beginners, and the people doing that should be able to figure out that they should be using --compiler for that.

This is also lacking a test.

I can add one once we figure out the appropriate behavior.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

Added the doc comment Seb wanted.

@CyberShadow
Copy link
Copy Markdown
Member

now, if the compiler is somewhere else, say if you download the tarfile and uncompress it but also have dmd in your system path, it will invoke the one from the system path, not the local one, which is very confusing.

Ah, I see. I agree, that would be useful, though it could still potentially break users' setups. For example, they might install rdmd with their package manager (probably as part of a dmd/dtools package), then add to their PATH a directory containing a version of dmd built from source. With the new behaviour, rdmd will use the system dmd instead of the one in PATH.

The change is still probably worth it, as currently dmd will use the sc.ini / dmd.conf from its own directory to find the Phobos includes and libraries adjacent to its own directory, so updating rdmd to behave similarly will improve consistency. However, this needs a changelog entry, and is certainly something that is better left for the next major version (2.076.0).

@joakim-noah
Copy link
Copy Markdown
Contributor Author

joakim-noah commented Jul 12, 2017

Whatever we do is going to break some user's setup, but we need to make sure we get the beginners' experience right, because if some noob downloads the compiler tarfile, uncompresses it, and runs rdmd on a sample file, this is what they see:

[joakim@localhost ~]$ ./dmd2/linux/bin64/rdmd dmd2/samples/d/sieve.d
std.process.ProcessException@std/process.d(369): Executable file not found: dmd
----------------
??:? @trusted std.process.Pid std.process.spawnProcessImpl(const(char[][]), std.stdio.File, std.stdio.File, std.stdio.File, const(immutable(char)[][immutable(char)[]]), std.process.Config, const(char[])) [0x591c1f]
??:? @trusted std.process.Pid std.process.spawnProcess(const(char[][]), std.stdio.File, std.stdio.File, std.stdio.File, const(immutable(char)[][immutable(char)[]]), std.process.Config, const(char[])) [0x59196d]
??:? int rdmd.run(immutable(char)[][], immutable(char)[], bool) [0x52ad56]
??:? immutable(char)[][immutable(char)[]] rdmd.getDependencies(immutable(char)[], immutable(char)[], immutable(char)[], immutable(char)[][]) [0x52b119]
??:? _Dmain [0x528c23]
??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFNlZv [0x57868b]
??:? scope void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5785bb]
??:? scope void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x578634]
??:? scope void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x5785bb]
??:? _d_run_main [0x57852b]
??:? main [0x573365]
??:? __libc_start_main [0xd07c3439]

They thought rdmd was for them, and it's broken.

For more advanced users who know rdmd picks up the dmd from the path- though I'd argue that almost nobody knows about and relies upon that behavior, most will be surprised that that is how it works- we could now have rdmd always put out a message about what compiler path it's using, or tell them to run --chatty to check it now.

I'm fine with not rushing to get this into 2.075. It's been broken for years, we can get it right for the next release.

@CyberShadow
Copy link
Copy Markdown
Member

because if some noob downloads the compiler tarfile, uncompresses it, and runs rdmd on a sample file, this is what they see:

It's no argument that fixing this would be a good change.

we could now have rdmd always put out a message about what compiler path it's using, or tell them to run --chatty to check it now.

I think that would be excessive (and annoying).

I think the current approach proposed here (always picking up the dmd binary from rdmd's executable's directory if there is one) is fine, just needs a test and changelog entry (plus review nits).

BTW, what do you think of activating the new behaviour only if there is also a dmd.conf / sc.ini file in rdmd's directory as well? That would avoid breaking the situation I described above, because these files will generally be in the /etc directory or such.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

just needs a test and changelog entry (plus review nits).

I'll get on that later today.

BTW, what do you think of activating the new behaviour only if there is also a dmd.conf / sc.ini file in rdmd's directory as well? That would avoid breaking the situation I described above, because these files will generally be in the /etc directory or such.

In your scenario above, both the system dmd and the locally-built dmd are most likely in the path. If you prepend your local dmd folder to the path, this PR will change what rdmd does. If you append your local dmd folder to the path, the more likely scenario, the system dmd is still being used and this PR won't change anything. ldc doesn't put its ldc2.conf in the same directory- it's in an adjacent directory called etc/- I don't know what config file gdc uses and where it keeps it.

Seems like a lot of work for an unlikely scenario, better to just add a note to the changelog telling them to use --compiler instead of relying on weird path setups.

@CyberShadow
Copy link
Copy Markdown
Member

Seems like a lot of work for an unlikely scenario,

How so? Just check for the configuration file when you check for the compiler binary.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

How so? Just check for the configuration file when you check for the compiler binary.

I just told you, each compiler has differently named and placed config files. Why add all that logic for some weirdo intentionally using rdmd with some strange path setup? Tell him to state his intentions clearly, with --compiler.

@CyberShadow
Copy link
Copy Markdown
Member

ldc doesn't put its ldc2.conf in the same directory- it's in an adjacent directory called etc/- I don't know what config file gdc uses and where it keeps it.

I think we should look into that then before going forward with this. I was thinking mainly about DMD, I don't know how the other compilers deal with this (running them straight out of their tarball), if at all.

Why add all that logic for some weirdo intentionally using rdmd with some strange path setup? Tell him to state his intentions clearly, with --compiler.

That's not a constructive way to look at things. The situation doesn't seem otherworldy to me, and at least in case of DMD, supporting it should be very little effort.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

joakim-noah commented Jul 12, 2017

I think we should look into that then before going forward with this.

No, checking all the config files is a dead end.

I was thinking mainly about DMD, I don't know how the other compilers deal with this (running them straight out of their tarball), if at all.

As I said initially, rdmd now ships with ldc and is how I first ran into this bug, as I certainly didn't have ldmd2 in my path.

If you're so worried about not breaking all such niche uses of PATH, I could do a variation of what you initially suggested: stick spawnProcess in a try block and if it can't find dmd in its path, check for that exception/error message, swallow it, and then run it again with this dmd next to rdmd.

I think we're more likely to be allowing in broken rather than non-broken uses of PATH with such an approach, ie somebody doesn't realize that some random dmd in their path is being pulled in over the one next to rdmd, but it won't break the intentional esoteric PATH scenarios you mention.

@CyberShadow
Copy link
Copy Markdown
Member

No, checking all the config files is a dead end.

If you don't want to look into how other compilers behave, then please restrict the new behaviour to compiler == "dmd". Otherwise, there is no point in enabling this feature for a compiler which doesn't support this execution mode anyway. Either way, I don't understand why you're against the idea or doing some research on the subject.

If you're so worried about not breaking all such niche uses of PATH, I could do a variation of what you initially suggested: stick spawnProcess in a try block and if it can't find dmd in its path, check for that exception/error message, swallow it, and then run it again with this dmd next to rdmd.

I think that would be fine too. Are you proposing this because it is simpler than checking PATH?

FWIW, there is std.process.searchPathFor, however it is private at the moment.

I think we're more likely to be allowing in broken rather than non-broken uses of PATH with such an approach, ie somebody doesn't realize that some random dmd in their path is being pulled in over the one next to rdmd, but it won't break the intentional esoteric PATH scenarios you mention.

I am not against changing the behaviour, but I think it would be best if we were to avoid doing so when we can detect with reasonable effort that it would be pointless in the first place, hence the configuration file idea.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

This is ridiculous, you have way too much time on your hands. I have updated this PR to get rid of the first three checks. This commit enforces that the adjacent compiler is used, which is what 99% of users expect, and only falls back to the path otherwise. It fixes a long-standing bug in rdmd for the types of noobs this tool was written for, but apparently never used by.

Whether you merge it or throw it away, I don't care at this point. I tried to help the noobs out- I don't use rdmd, only noticed this because I finally tried it after the ldc inclusion- but if you'd rather worry about how some idiot set up his path instead of not handing new users a broken experience, no wonder people always complain about the first 5 mins. still.

I think this should be rushed into 2.075, but considering this issue sat untouched for 3.5 years now, I wouldn't be surprised if that continued.

@CyberShadow
Copy link
Copy Markdown
Member

This is ridiculous, you have way too much time on your hands.

I think we are at the point where it's more worthwhile to research issues thoroughly rather than rushing in fixes, even if it means that contributors' limited time can limit what contributions can be accepted.

but if you'd rather worry about how some idiot set up his path

Degrading potential users who have an uncommon setup is not only not constructive, but actively prevents having a clear picture of the situation and fallout of the change. So, please refrain from doing that.

instead of not handing new users a broken experience,

Exaggerating your position is not helpful either. The expected way to use binary release archives is to prepend the bin directory to the PATH. Yes, there is an opportunity to improve thing here, but calling things broken is pushing it.

I think this should be rushed into 2.075

Please get acquainted with our release process. Once the beta period starts, new development continues on the master branch, while the stable branch is used for regression and other urgent fixes. Anything that has an obvious risk of breakage needs to be on the master branch.

In any case, if you don't have time to continue working on this, that's OK; I'll have a look at how the other compilers behave and what would best make sense for rdmd to do.

@CyberShadow
Copy link
Copy Markdown
Member

CyberShadow commented Jul 14, 2017

@joakim-noah I had a look at what GDC and LDC do, and it looks like all three compilers support being ran by absolute path from their zip/tarball. The way they achieve this is different between compilers:

  • DMD has a dmd.conf / sc.ini file in the bin directory, which it reads if found. That has a line such as:
     DFLAGS=-I%@P%/../src/phobos -I%@P%/../src/druntime/import -L-L%@P%/../lib
    
    %@P% means the compiler executable's directory, so that causes it to look in ../src/....
  • LDC also has a configuration file, but it is located in ../etc/. The relevant code is here.
  • GDC does not have a configuration file. Instead the path is calculated as it passes through its many layers, but I understood from Iain that GCC_EXEC_PREFIX and -iprefix are involved. In any case, the compiler executable directory serves as the default base for these path calculations.

So, you're right that checking for configuration files is pointless; I had a wrong assumption that at least the DMD-like drivers (gdmd/ldmd) would have an implementation or equivalent of DMD's sc.ini/dmd.conf.

So, I think this change is good as it is; just missing a changelog entry which includes a description of the situation where this change could break a user's setup, and a workaround (e.g. --compiler=dmd to again use dmd from PATH). Let me know if you want to continue working on this.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

I've looked at the config file lookup code before: if you scroll up from the block you linked from ldc, it checks for ldc2.conf in three other locations first, the working directory, next to the compiler, and then ~/.ldc (a fourth location on Windows, the home directory). dmd has a similar lookup process. I shut down this config file suggestion because I didn't want to navigate that mess, all for a niche user who's using his PATH with rdmd in an unusual way.

Regarding the changelog, that would require a separate PR in dmd's changelog/? I can submit another PR for that, once this is in. Should I submit a changelog PR to dmd master or stable?

@CyberShadow
Copy link
Copy Markdown
Member

CyberShadow commented Jul 15, 2017

if you scroll up from the block you linked from ldc, it checks for ldc2.conf in three other locations first, the working directory, next to the compiler, and then ~/.ldc (a fourth location on Windows, the home directory). dmd has a similar lookup process.

I saw that too, and I think that's fine, because the only way a config file can end up there is if it were placed there by the user, which comes associated with the expectation that compilers will use that config file no matter where they're run from.

Regarding the changelog, that would require a separate PR in dmd's changelog/? I can submit another PR for that, once this is in. Should I submit a changelog PR to dmd master or stable?

Oh, I thought the tools repo had a changelog directory like the others. @wilzbach What do you think, should we add a changelog directory for tools too?

@CyberShadow
Copy link
Copy Markdown
Member

Almost forgot, this also still needs a test...

@joakim-noah
Copy link
Copy Markdown
Contributor Author

I will add the test and update.

@wilzbach
Copy link
Copy Markdown
Contributor

Oh, I thought the tools repo had a changelog directory like the others. @wilzbach What do you think, should we add a changelog directory for tools too?

It's already checked by the script, so adding a file to changelog like on the other repos will work.

FYI: the commit list is parsed as well (like on the other repos), see e.g. https://dlang.org/changelog/2.074.0.html (last entry at the bottom)

first, whereas before, it would default to invoking a compiler
from the user's path (which it still falls back to if there's
no D compiler next to rdmd). If you want rdmd to use a specific
compiler, use the --compiler flag to force it.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First time writing a changelog entry, let me know if everything's right.

Copy link
Copy Markdown
Member

@CyberShadow CyberShadow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM sans nits, thanks!

rdmd_test.d Outdated
assert(res.status == -SIGSEGV, format("%s", res));
}

/* https://issues.dlang.org/show_bug.cgi?id=11997- rdmd should search it's
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, "its" not "it's" (which is always short for "it is")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, :) but you'll have to take that up with Martin, as it's a copy-paste from his bug title. I can still change it here if you prefer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyone can edit Bugzilla issues, including their titles! Fixed it there now :)

@@ -0,0 +1,7 @@
Check for a D compiler next to rdmd first
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the heading list, it won't be clear that this applies to rdmd, so how about:

"rdmd now prefers using a compiler binary from its directory, if one exists"

@@ -0,0 +1,7 @@
Check for a D compiler next to rdmd first

Rdmd will now try to use the D compiler it's installed with
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backticks around rdmd (plus lowercase) and, --compiler?

first, whereas before, it would default to invoking a compiler
from the user's path (which it still falls back to if there's
no D compiler next to rdmd). If you want rdmd to use a specific
compiler, use the --compiler flag to force it.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plus:

To restore the old behaviour of always using the compiler from `PATH`, use `--compiler=dmd`.


Rdmd will now try to use the D compiler it's installed with
first, whereas before, it would default to invoking a compiler
from the user's path (which it still falls back to if there's
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"path" should be in uppercase and backticks, so that it's clear the the word refers to the name of an environment variable, and not some nonspecific user path.

scope(exit) std.file.remove(localDMD);
res = execute(rdmdApp ~ [modelSwitch, "--force", "--chatty", "--eval=writeln(`Compiler found.`);"]);
assert(res.status == 1, res.output);
assert(res.output.canFind(`spawn ["` ~ localDMD ~ `",`));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could do a "positive" test by putting a fake compiler binary in the same directory as rdmd, but it's entirely up to you. Another way would be looking at the output of --dry-run, and simply checking that it would have executed the correct compiler.


res = execute(rdmdApp ~ [modelSwitch, "--force", "--chatty", "--eval=writeln(`Compiler found.`);"]);
assert(res.status == 1, res.output);
assert(res.output.canFind(`spawn ["` ~ localDMD ~ `",`));
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the test: it would be nice if we didn't expect it to fail, but that would mean copying a whole dmd install over, which would be too much of a PITA. I'm not sure if the --force flag is necessary, didn't need it myself, but all other invocations of --eval in this test file use it, so I stuck it here too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could do a "positive" test by putting a fake compiler binary in the same directory as rdmd, but it's entirely up to you.

You mean positive in that rdmd would run and exit successfully? Wouldn't dmd need druntime and phobos and a config file too? Becomes too much of a PITA, especially once you consider other compilers, like ldc.

Another way would be looking at the output of --dry-run, and simply checking that it would have executed the correct compiler.

Yeah, that might work, but you're not even executing the eval=writeln code then, so not much of a difference.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't dmd need druntime and phobos and a config file too? Becomes too much of a PITA, especially once you consider other compilers, like ldc.

Not if the dmd binary in question is just a small stub program that prints something and returns success OSLT.

Copy link
Copy Markdown
Contributor

@wilzbach wilzbach Jul 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @CyberShadow means sth. along these lines:

mkdir -p $(ROOT)/dmd_stub
echo 'void main() { import std.stdio; "Hello World".writeln; }' > $(ROOT)/dmd_stub/dmd.d
dmd -of/$(ROOT)/dmd_stub/dmd $(ROOT)/dmd_stub/dmd.d

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, heh, seems pointless. I meant it would be nice if we could actually compile with the adjacent dmd. If we're not going that far, I actually like that it fails, because the first assert will trigger if rdmd pulls a dmd from PATH, because it will unexpectedly compile! :) No need to even check the path in the spawn ["dmd"] invocation then.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems nuts that your tempDir is the one where all your source packages are unpacked, surely there's a better way.

It's not in my hand to change that. That's how the package manager Nix works for building packages. You are only allowed to write inside your build directory because you are not allowed to change anything else on the system this way. The package has only access to that very directory, this is important to prevent any side effects while building and maintain a reproducible build.

Not sure what you mean, these tests simply assume that the dmd compiler is installed somewhere in the system path and that there's no random file/directory named dmd lying around your tempDir, where it runs the tests.

Currently dmd's source directory needs to be placed in the parent directory of the source directory of the tools. I suggest to expect the dmd source inside the tools directory. This is a much cleaner way because you don't know your environment and ../dmd might be an old dmd needed by the user or whatever which might lead to strange errors.
This would also fix my issue with Nix and prevent my current workaround.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently dmd's source directory needs to be placed in the parent directory of the source directory of the tools. I suggest to expect the dmd source inside the tools directory.

Not sure how that's related to this discussion, but potentially having a copy of every component inside the directory of every component doesn't sound better than each component expecting its siblings to be in its parent directory.

@ThomasMader Feel free to submit a PR which changes the rdmdApp and localDMD paths to a more unique subdirectory of tempDir().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which means that dtools is in /tmp/nix-build-dtools-2.078.0.drv-5/dtools and I also need to place the dmd source at /tmp/nix-build-dtools-2.078.0.drv-5/dmd which is the same path localDMD wants to be written to. :-)

I don't see how you reached that conclusion. I don't see how the location of the dmd source code is in any way relevant here.

The problem here is that both rdmd_test and Nix do not do a sufficiently good job at namespacing their temporary files in $TMP. The collision could be resolved from etiher side, and you seem to have chosen the D side, but I would suggest to also raise an issue in Nix's issue tracker.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I think I understand what you meant now regarding the DMD dir:

$(shell [ ! -d $(DMD_DIR) ] && git clone --depth=1 https://github.com/dlang/dmd $(DMD_DIR))

It's cloned automatically by the Makefile. I don't know much about Nix packaging, but if you care about the package being buildable into the future or reproducible builds, you will probably want to provide the contents of the dmd repository separately, and perhaps specify the path to it using the DMD_DIR makefile parameter.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know much about Nix packaging, but if you care about the package being buildable into the future or reproducible builds, you will probably want to provide the contents of the dmd repository separately, and perhaps specify the path to it using the DMD_DIR makefile parameter.

While I was preparing #287 I realized that I already can set DMD_DIR and move it inside the tools directory myself.
So it's no real issue for me anymore.
The remaining question is if this should be the default behavior but should probably better be discussed in the PR.

@joakim-noah
Copy link
Copy Markdown
Contributor Author

Updated to fix your nits and describe exactly the situation that motivated this change.

@@ -0,0 +1,14 @@
rdmd now checks for a D compiler binary in the directory
it's in first
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First changelog line has to be all on its own line (not wrapped)... CI / changelog generation will fail otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@CyberShadow
Copy link
Copy Markdown
Member

@wilzbach The changelog entry doesn't seem to be picked up, any idea why off the top of your head?

@CyberShadow CyberShadow self-assigned this Jul 17, 2017
@CyberShadow
Copy link
Copy Markdown
Member

Oh, I'm guessing dlang/dlang.org#1826 was the cause.

Well, LGTM, so merging.

@dlang-bot dlang-bot merged commit ee59ec9 into dlang:master Jul 17, 2017
@joakim-noah
Copy link
Copy Markdown
Contributor Author

Thanks, looking forward to trying rdmdwith ldc after this, because it now ships in ldc's tarfile and I don't put any particular D compiler on my path in my VPS.

@wilzbach
Copy link
Copy Markdown
Contributor

Oh, I'm guessing dlang/dlang.org#1826 was the cause.

Yep, Martin reverted the generation of the _pending file and _pre exists and won't be regenerated...

@wilzbach
Copy link
Copy Markdown
Contributor

The changelog entry doesn't seem to be picked up

@joakim-noah btw -> https://dlang.org/changelog/2.076.0_pre.html#b11997

@CyberShadow
Copy link
Copy Markdown
Member

Well, for the matter of public record, noting here that this broke the DMD setup on my laptop (as I don't build rdmd there), so that certainly wasn't a theoretical concern. The --compiler workaround does work (and it's backwards compatible).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge Type.Bug Things don't work as they were intended to, or the way they were intended to work doesn't make sense

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants