resolving environmental variables in uri

[anqixu]

Here's an interesting use case for the robot in our research lab: we have multiple developers, each of which has an unique account on an institution-wide NFS. Our robot uses a number of hg repositories, stored at shared paths on the NFS. We have multiple machines that can access the NFS via ssh, but no single one is reliable enough to be used as a single static host.

We would like a standardized way for any individual among us to 'wstool init' and 'wstool update' the packages, in addition to manually execute VCS commands. Due to institution regulations, we cannot use a common shared user, nor can we serve the repositories outside

My current attempt to realize this is to define USERNAME and SERVER environmental variables (in .bashrc). The .rosinstall file contains uri of the form 'ssh://${USERNAME}@${SERVER}//path/to/repo'. This .rosinstall works with wstool init, but it stores the resolved variables in the .hg/hgrc files. I can restore the environmental variable forms using a sed-in-a-find-loop bash script. However, the problem is that 'wstool update' doesn't work, as it detects a mismatch between the .rosinstall and the .hg/hgrc (which is weird since both of them have the unresolved environmental variable strings).

I'm pitching these two competing requests as potential solutions:
A) add an option for 'wstool update' to force-update the hgrc with the one from .rosinstall (presumably with resolved variable strings)
B) add an option, or by default, to have 'wstool update' not resolve environmental variables internally, and let hg (e.g. hg pull) worry about it

[wjwwood]

I don't know if hg has a similar setting, but you could use a generic url in your .rosinstall file and each individual could "rewrite" the generic url to a more specific one using the url.<base>.insteadOf option in the git configs, see:

https://git-scm.com/docs/git-config

This is often used in git to rewrite https urls to be git+ssh like urls, or the reverse:

http://stackoverflow.com/questions/1722807/git-convert-git-urls-to-http-urls

[tfoote]

Your use case I think is kind of getting luck that wstool under the hood is invoking hg via a shell subprocess that does the substitution on the command line which happens to get your environment variables.

The .hgrc does not get passed through the command line thus does not get substituted.

If you want to change the upstream values I'd suggest using a template rosinstall file with a simple script to evaluate the variables explicitly. And pass that to wstool merge to update the urls.

export SERVER=foo
export USER=bar
evaluate_template my_rosinstall.template | wstool merge -
wstool update

First you explicitly set your urls, second you update the content.

[tkruse]

I think it may help @anqixu to explain a bir about the internals of wstool / vcstools. The code uses the command line as the interface to mercurial, so whatever wstool command you run, it get's translated to one or more hg commands. wstool avoids to directly read or write hgrc.

The hg command code is all in this file: https://github.com/vcstools/vcstools/blob/master/src/vcstools/hg.py

So for your case, wstool init calls hg init with the url having the variables, and the shell process replaces them, that's how the substituted values are written to .hgrc (Same as what Tully said).

For checking the current workspace state against the .rosinstall, wstool similarly uses hg commands to detect the current branch name and remote of the local folder.

Currently, the behavior of wstool with respect to environment variables is unspecified, I believe (nobody thought of that usecase). So changing the code to invoke

hg clone 'url' localpath

instead of

hg clone url localpath

would achieve that the unsubstituted URL goes into hgrc (I tried, mercurial3.4 will accept such paths). That would change current behavior of wstool init, but it might be a change for the better (up to discussion). In your case that would avoid having to manipulate the hgrc after wstool init.

The logic to compare the .rosinstall contents with what is in hgrc during wstool update is in file:
https://github.com/vcstools/wstool/blob/master/src/wstool/config_elements.py
in method VCSConfigElement:prepareInstall()
in particular we call 2 methods of vcstools (the file above for hg):

# get the path stored in hgrc, using 'hg paths default'
cur_url = self._get_vcsc().get_url()
# compare the path with the one in .rosinstall, done in vcstools.vcs_base.py
# essentially url.rstrip('/') == url_or_shortcut.rstrip('/')
if not self._get_vcsc().url_matches(cur_url, self.uri): #...

The reason wstool update detects a difference in paths is that mercurial itself, in hg paths default replaces environment variables with their values. So you'd need an alternative hg command (other than hg paths default) to read the value from hgrc (without actually parsing that file).

Alternatively, wstool update could attempt to resolve paths using the shell first, as a 'normalization', in case the path contains variables (small risk of command injection, but that might be an open attack vector already anyway). So both the .rosinstall and hgrc could contain environment variables, but for wstool compare, wstool would resolve the one from .rosinstall, and hg already resolves the one in hgrc.

Note there may be other similar undesirable behaviors with respect to usage of environment variables in the .rosinstall file. This is a use-case that does not have any tests yet. IMHO wstool is underspecified with respect to that, and could be improved to define behavior for such cases, including breaking changes to the current behavior (after asking the community).

So @anqixu, the process to follow would be to write up a short suggestion to improve support of environment variables in rosinstall files for wstool, including the change to hg clone, and the comparison in wstool update resolving paths from .rosinstall in the shell first (basically running echo <path>, maybe only if a $ or % sign is in the string).
Then post that to the buildsystem-SIG to collect feedback, implement the changes and add tests, make a PR.

[dirk-thomas]

Closing since the repository is about to be archived: see #154.