Add a __version__ key to vp and n3fit

[scarlehoff]

There are so many corner cases... but I guess we can consider burning down a string only upon deploy and call the git code otherwise.

I am missing a very important use case* I don't know how to go around:
I have my repository in one folder and run the code from a different folder. So I cannot just "git it". Rely on installing the files in development mode doesn't work either.
I can burn, from setuptools, the path to git...

Anyway, I'd love any ideas for the previous situation.

For the burning the version on the conda package I think what I did is ok (with a bit more logic to go around cases in which there is something written in the __init__) as well as for when the code is ran inside the repo. Any comments?

*it is very important because it is probably the standard case in clusters

Closes #275

[Zaharid]

You can set the current working directory in subprocess, from Path(__file__).parent. That works for me at least.

I think this is more or less the right approach. A few minor nitpicks:

• I'd use a separate file called version.py or whatever with a big comment saying to not add anything else in there.

• I actually tried and am unsure you can get the same format as conda so easily with one invocation of git describe. There are various corner cases, notably the fact that when you are in a tag commit it returns the tag itself without info on the commit hash or a zero for the revision. We could declare that it doesn't matter or even change the logic in conda. Otherwise you probably need some extra logic.

• You should use text=True and check=True in subprocess.run.

[Zaharid]

Also repurposing double dunder names is frowned upon by the thought police. We should call the thing __version__ instead.

[scarlehoff]

You can set the current working directory in subprocess, from Path(__file__).parent. That works for me at least.

Only if you did something like python setup.py develop. You might be developing only vp or only n3fit and install the other, then you lose access to the repo.

[scarlehoff]

I touched a bit the logic to deal with corner cases I could think of (some of the ones you mentioned).

Still missing a satisfactory way to deal with the problem I mentioned before. One thing can be "if you install in development mode you don't have access to the tag, though luck" but the conda installation could be in a different hard drive from the repository which you don't have access to at run time.

Ideally setuptools would change the version.py file on the fly only on install...

[Zaharid]

I am confused as to what the issue is: either you install in develop mode in which case you have access to git (because it just symlinks the files) or you build and install with cmake, which you should use instead of setup.py install to set various other paths correctly anyway. What is the remaining case?

…

On Wed, 29 Apr 2020, 10:18 Juacrumar, ***@***.***> wrote: I touched a bit the logic to deal with corner cases I could think off (some of the ones you mentioned). Still missing a satisfactory way to deal with the problem I mentioned before. One thing can be "if you install in development mode you don't have access to the tag, though luck" but the conda installation could be in a different hard drive from the repository which you don't have access to at run time. Ideally setuptools would change the version.py file on the fly only on install... — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#748 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLJWUTJBSJO7UBNULUKPD3RO7PENANCNFSM4MTEVHGA> .

[scarlehoff]

I am in a cluster where the conda installation is in some remote disk which all nodes have access to (an amazon s3 bucket for instance). I develop, however, in the local disk of the login machine because otherwise files take a minute to save.

Since this is my workflow I install the package with python setup.py install and then things run in some cluster which have no access back to the repository

Note: one of the clusters I have access to in Milan work like this. I never use it anyway, but it is a case

Note 2: Saying "ok, in that case you still can use cmake and you should" wouldn't count as "satisfactory" but I am ok with that. I prefer the setuptools on the fly modification better but meh.

[Zaharid]

I guess this thing can do that, but the README is way too long for me:

https://github.com/pypa/setuptools_scm

[scarlehoff]

The trick is to instead of spend 10 minutes reading the readme to spend 4 hours trying and failing!
I'll try. There's 4 hours from now to the code meeting. I'll get it just in time.

[scarlehoff]

Apparently setuptools_scm does not work well with pip because of pip moving everything to some /tmp folder before installing.

The easy solution is using setup.py (if it works on travis). Not sure whether there's any downsides to this in our case.

[scarlehoff]

"if you install in development mode you don't have access to the tag, though luck"

This is my answer now to all corner I was thinking about before.

[Zaharid]

Honestly I much must prefer a simple minded approach that I understand which we had at the beginning than the overenginerred dependincy that triggers problems now and who knows what in the future

[scarlehoff]

Well, the thing is that the problem is not weird, given that pip is moving things to a /tmp folder and then you lose access to git. But there is no "easy solution" I can think of for this.

Anyway, I went back to the simple approach. Didn't want to reset and push in case someone wants to go back to setuptools_scm one day and try it out.

[Zaharid]

Ideally we would have a valid identifier as per
https://www.python.org/dev/peps/pep-0440/#local-version-segments
which the conda package implements. That is we would have something like

3.4.1874+gc4c7a1f5-dirty

Instead of

3.4-1874-gc4c7a1f5-dirty

It is a bit more work though.

There is also the corner case of being in a tagged commit, where git describe doesn't print the commit or the revision (but it prints "dirty" if you ask for it). We could probably forget about that, in that the

A bunch of lower level commands include (which may or may not make things easier):

#get tag
git rev-list --tags --max-count=1
# c25e438aeb64643d93dd3c1b3c88db21f9ecbffe
#get tag name
 git describe --tags c25e438aeb64643d93dd3c1b3c88db21f9ecbffe
# Get revision number
 git rev-list c25e438aeb64643d93dd3c1b3c88db21f9ecbffe.. --count

[scarlehoff]

In theory --long gives you the hash even in the tag case.

The only different between the good and the bad is the "+"?

[Zaharid]

In theory --long gives you the hash even in the tag case.

Cool! that is much better.

The only different between the good and the bad is the "+"?

And the dot for the revision number :)

[scarlehoff]

And the dot for the revision number :)

Don't seem worth it... it's a pity git describe doesn't have a --format option though. Damn Linus.

[Zaharid]

I would say it is:Tools like to know the revision number, for example to decide which package to install. For example the conda packages used to be problematic when the revision number was a "build number" rather than part of the version, and I still have nightmares.

[scarlehoff]

Sigh

version=$(git describe --long)
tag=$(git describe --abbrev=0 --tags)
version=${version/${tag}-/${tag}.}
hash=$(git rev-parse --short HEAD)
version=${version/-g${hash}/+g${hash}}

This seems to work fine...

[Zaharid]

Is there no way of having CMAKE doing this when the VP_DEV option is off?

[scarlehoff]

Is there no way of having CMAKE doing this when the VP_DEV option is off?

#748 (comment)

[Zaharid]

Right, I thought that was when you were not using cmake at all, but just pip install from the vp directory. I'd say this is a bit more of an issue, but not sure how easy a solution is.

[scarlehoff]

The problem is pip sends you to tmp when you install so you lose access to git. From cmake we can change the file in the installation folder after pip has installed.

Something like

echo $(git describe) > $(python -c 'import n3fit.version ; print(n3fit.version.__file__))

[Zaharid]

I like that. I have been trying to get the installation directory from setuptools and now I want to shout.

[scarlehoff]

This last commit does the trick.

I am not sure whether it is necessary to write anything from the build scripts anymore, i.e., I am not sure whether to create the package you need it to have the modification in the source file.

[Zaharid]

I think this is failing here

nnpdf/conda-recipe/run_test.sh

Line 15 in 76a8d36

mkdir bldtest

because this is happening outside a git repository. One solution is to have the fallback thing you have in python.

[scarlehoff]

I'm happy with the last one, let's see whether travis is also happy

[Zaharid]

If this is now done in cmake, we don't need it in the package scripts right?

[scarlehoff]

we don't need it in the package scripts right?

I thought so but then travis failed to find git.

[Zaharid]

Argh. I see. I guess conda is not copying the .git folder. I am not sure how much I like having 4 separate mechanisms (conda package, python dev, cmake, conda build), but I also don't have better ideas or feel particularly motivated to think about them.

[scarlehoff]

Mm, the linux travis is understanding the repository as "dirty", not 100% sure why... maybe the cmake and the deployment tag need to be set differently...

[Zaharid]

I wouldn't add git to the dependencies. That can do all sorts of bad things, and wouldn't solve much unless you also copy the .git folder. Now I prefer the previous idea of writing the tag both in cmake if possible and in the conda build script.

[scarlehoff]

I wouldn't add git to the dependencies.

Yeah, and it didn't solve the problem anyway (but only in mac for some reason!). Well it was a wild try. Reverting.

[scarlehoff]

I'm going to merge this and cross my fingers. If anything breaks I'll revert. If everything works I'll add next the gitstamp to the n3fit replicas.