Missing `tomlkit`

[cschwan]

At least on dom on the DIS-more branch I get:

Error processing line 1 of /home/cschwan/.local/lib/python3.10/site-packages/zzz_poetry_dynamic_versioning.pth:

  Traceback (most recent call last):
    File "/usr/lib/python3.10/site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/cschwan/.local/lib/python3.10/site-packages/poetry_dynamic_versioning/__init__.py", line 13, in <module>
      import tomlkit
  ModuleNotFoundError: No module named 'tomlkit'

[alecandido]

Can you add details on how to reproduce, please? (i.e. which command did you launch)

[cschwan]

./rr install for instance, but anything starting with ./rr gives these warnings.

[alecandido]

Not reproduced locally, I'm going to try on dom

[cschwan]

That's strange:

cschwan@dom runcards $ which pip
/usr/bin/pip
cschwan@dom runcards $ pip show tomlkit
Error processing line 1 of /home/cschwan/.local/lib/python3.10/site-packages/zzz_poetry_dynamic_versioning.pth:

  Traceback (most recent call last):
    File "/usr/lib/python3.10/site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/cschwan/.local/lib/python3.10/site-packages/poetry_dynamic_versioning/__init__.py", line 13, in <module>
      import tomlkit
  ModuleNotFoundError: No module named 'tomlkit'

Remainder of file ignored
Name: tomlkit
Version: 0.9.1
Summary: Style preserving TOML library
Home-page: https://github.com/sdispater/tomlkit
Author: Sébastien Eustace
Author-email: sebastien@eustace.io
License: UNKNOWN
Location: /usr/lib/python3.10/site-packages
Requires: 
Required-by: poetry-dynamic-versioning, poetry

[alecandido]

I've just run:

./rr run nnpdf31_proc/HERA_CC_318GEV_EM_SIGMARED theories/theory_213.yaml

on dom, and it worked.

[cschwan]

Even starting python complains about it:

Error processing line 1 of /home/cschwan/.local/lib/python3.10/site-packages/zzz_poetry_dynamic_versioning.pth:

  Traceback (most recent call last):
    File "/usr/lib/python3.10/site.py", line 186, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/cschwan/.local/lib/python3.10/site-packages/poetry_dynamic_versioning/__init__.py", line 13, in <module>
      import tomlkit
  ModuleNotFoundError: No module named 'tomlkit'

Remainder of file ignored
Python 3.10.2 (main, Jan 15 2022, 19:56:27) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

@alecandido could you please try to delete ~/.local and rerun ./rr install?

[felixhekhorn]

@cschwan this seems to me like your poetry-dynamic-versioning installation is somehow corrupted ... as far as I understood it has to be installed alongside poetry (meaning in the very same environment)

what do you get on

felix@FHe19b:~$ pip list | grep poetry
poetry                        1.1.12
poetry-core                   1.0.7
poetry-dynamic-versioning     0.13.0

and

felix@FHe19b:~$ which poetry
/home/felix/.local/bin/poetry

see also https://github.com/mtkennerly/poetry-dynamic-versioning#installation

and tomlkit is a direct dependency https://github.com/mtkennerly/poetry-dynamic-versioning/blob/eac2f68f6e01f6e336dfb6c23a4c90911c12f013/pyproject.toml#L28 so pip has messed up, maybe?

[cschwan]

There you go:

cschwan@dom runcards $ pip list | grep poetry
poetry                        1.1.13
poetry-core                   1.0.7
cschwan@dom runcards $ which poetry
/usr/bin/poetry

[alecandido]

I tried to delete .local and I reproduced the issue.

I solved deleting again, and simply using the system available poetry. I installed manually dynamic versioning with pip.

In general, I believe the solution to be the following: do not use ./rr install to install poetry, dynamic versioning, or pre-commit.
The idea to manage the developer's environment with a script is wrong on its own: each developer will have its own very personal environment, with all its quirks, and the developer itself has to be responsible for the management (it's hard to maintain something that works everywhere in any context, so better not to do).

[cschwan]

I understand what you're saying, but in a sense that defeats the purpose of ./rr - I don't care about poetry or any other Python package, I just want to generate processes.

[felixhekhorn]

Mmm but that's like saying "I just want to play that game" and then missing on the video driver, no?

[alecandido]

Actually, I believe this will be solved by NNPDF/pinecards#127 and NNPDF/pinecards#128.

The workflow I have in mind is the following:

• if you're a developer, you're in charge to install poetry & co. by yourself
• otherwise, you can just use pip install, both with the remote version (pip install <runcards-runner-final-name>) and with the local one (pip install .)

Once you install with pip, then rr or its replacement will be available as command, and it will take the place of ./rr.
Instructions for developers are delicate and we can even spell personally (we will write something, but I'd expect it might fail in general if very detailed), instructions for everyone else will be simple, even without ./rr.

Eventually, when poetry will release 1.2.0, we will get even editable mode with pip for free.

[cschwan]

Mmm but that's like saying "I just want to play that game" and then missing on the video driver, no?

True 😄, and I've worded my statement probably more strongly than I felt, because - at least subjectively - 'poetry' keeps breaking time and again for me.

I've just recently realized that what I had in mind concerning the design of our runner was the containerization of the runcards runner; this would solve all the version and installation problems reliably.

[alecandido]

I keep thinking about poetry, that's also the feeling of @scarlehoff for example.

But analyzing all the issues one by one, what I currently believe is that poetry failures are just the signal of us doing something bad (mostly with versions). So, in most cases it's leading to early detection of inconsistencies.

This applies to all the restrictions on the python versions, or the dependency on various packages, but it does not apply to the issues related to poetry installation and environment management.
About this second category I believe the solution to be to use more scripts (like ŧest or run and so on), that we can write once and they are proved to work, plus the one written above: for actual usage (not development) rely on pip installation.

Moreover, there are the problems related to the installation of poetry itself: this is almost a non-problem, I created it myself with the installation script (I believe)...