cli: renku dataset create data

[AWiton]

Describe the bug

A dataset 'data' was created and removed (renku dataset rm data).
A new dataset 'data' with the same name is going to be created.
This crash...

Details

Please verify and redact the details.

Renku version: 0.5.0
OS: Linux (#1 SMP PREEMPT Wed Mar 27 17:53:10 UTC 2019)
Python: 3.7.3

Traceback

Traceback (most recent call last):
  File "[...]/renku/cli/_exc.py", line 100, in main
    result = super().main(*args, **kwargs)
  File "[...]/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "[...]/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/_client.py", line 80, in new_func
    result = ctx.invoke(method, client, *args, **kwargs)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/dataset.py", line 197, in create
    with client.with_dataset(name=name) as dataset:
  File "[...]/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "[...]/renku/api/datasets.py", line 129, in with_dataset
    ).set_reference(path)
  File "[...]/renku/models/refs.py", line 86, in set_reference
    str(self.path)
FileExistsError: [Errno 17] File exists: '../../datasets/79ad345298d34dc3a7c9b56f6a95b7b7/metadata.yml' -> '/home/xxxpathxxx/TiMeS/times-electrophy/.renku/refs/datasets/data'

Additional context

When removing data/dataset, all the metadata are not removed(?).

[rokroskar]

Hi @AWiton thanks for the report - I'm actually not able to reproduce this issue:

renku dataset create mydata
Creating a dataset ... OK
renku dataset rm mydata
OK
renku dataset create mydata
Creating a dataset ... OK

Can you show me the output of renku --version?

[rokroskar]

Ah nevermind I see you are on v0.5.0. ;)

[rokroskar]

I just tried the same with v0.5.0 and I can't reproduce the issue. Can you provide maybe a few more commands that preceded the one that failed?

[AWiton]

Hi Rok, Thanks for your message. I will try to reproduce the issue and add more commands for reproducibility. Best, Adrien

…

________________________________ From: Rok Roškar <notifications@github.com> Sent: 16 May 2019 16:32 To: SwissDataScienceCenter/renku-python Cc: Witon Adrien Jean-Baptiste; Mention Subject: Re: [SwissDataScienceCenter/renku-python] cli: renku dataset create data (#535) I just tried the same with v0.5.0 and I can't reproduce the issue. Can you provide maybe a few more commands that preceded the one that failed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#535?email_source=notifications&email_token=AMB35RXDKJL6QI2ZAGM5HLTPVVWAHA5CNFSM4HMRUJRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVSAAHQ#issuecomment-493092894>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMB35RUSR7354PA5STRMMTTPVVWAHANCNFSM4HMRUJRA>.

[AWiton]

Hi, It is not the same bug, and not really what I was looking for, but at least it should be very reproducible: jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject$ renku dataset add MyData jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject$ ls data Dockerfile environment.yml notebooks README.md requirements.txt jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject$ ls data/ MyData jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject$ renku dataset add MyData/Pat1 Ahhhhhhhh! You have found a bug. 🐞 1. Open an issue by typing "open"; 2. Print human-readable information by typing "print"; 3. See the full traceback without submitting details (default: "ignore"). Please select an action by typing its name (open, print, ignore) [ignore]: ore Error: invalid choice: ore. (choose from open, print, ignore) Ahhhhhhhh! You have found a bug. 🐞 1. Open an issue by typing "open"; 2. Print human-readable information by typing "print"; 3. See the full traceback without submitting details (default: "ignore"). Please select an action by typing its name (open, print, ignore) [ignore]: Traceback (most recent call last): File "/home/jovyan/.local/bin/renku", line 10, in <module> sys.exit(cli()) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_exc.py", line 109, in main self._handle_github() File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_exc.py", line 155, in _handle_github getattr(self, '_process_' + value)() File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_exc.py", line 100, in main result = super().main(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_client.py", line 80, in new_func result = ctx.invoke(method, client, *args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/dataset.py", line 225, in add with client.with_dataset(name=name) as dataset: File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/api/datasets.py", line 129, in with_dataset ).set_reference(path) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/models/refs.py", line 83, in set_reference self.path.parent.mkdir(parents=True, exist_ok=True) File "/opt/conda/lib/python3.6/pathlib.py", line 1248, in mkdir self._accessor.mkdir(self, mode) File "/opt/conda/lib/python3.6/pathlib.py", line 387, in wrapped return strfunc(str(pathobj), *args) FileExistsError: [Errno 17] File exists: '/home/jovyan/testproject/.renku/refs/datasets/MyData' jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject$ Best,

…

________________________________ From: Rok Roškar <notifications@github.com> Sent: 16 May 2019 16:32 To: SwissDataScienceCenter/renku-python Cc: Witon Adrien Jean-Baptiste; Mention Subject: Re: [SwissDataScienceCenter/renku-python] cli: renku dataset create data (#535) I just tried the same with v0.5.0 and I can't reproduce the issue. Can you provide maybe a few more commands that preceded the one that failed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#535?email_source=notifications&email_token=AMB35RXDKJL6QI2ZAGM5HLTPVVWAHA5CNFSM4HMRUJRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVSAAHQ#issuecomment-493092894>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMB35RUSR7354PA5STRMMTTPVVWAHANCNFSM4HMRUJRA>.

[AWiton]

Hi again, I did a mistake with: $renku dataset add My Data ~/work/ [It should have been MyData, not My Data]. Anyway, this genereates a bug... (See below) And this creates a double pain error by making the repository dirty: jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData/Pat1$ renku dataset add MyData ~/work/TestFile.txt Error: The repository is dirty. Please use the "git" command to clean it. On branch master Your branch is ahead of 'origin/master' by 1 commit. (use "git push" to publish your local commits) Untracked files: (use "git add <file>..." to include in what will be committed) .renku/refs/datasets/My nothing added to commit but untracked files present (use "git add" to track) Once you have added the untracked files, commit them with "git commit". jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData/Pat1$ Then, it is not clear what is the best way to do. Do you prefer to have these bugs enter in GitHub? BTW, this is kind of related to the inital bug, where renku and git are not in the same state. All the best, Adrien jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData$ mkdir Pat1 jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData$ cd Pat1/ jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData/Pat1$ pwd /home/jovyan/testproject/data/MyData/Pat1 jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData/Pat1$ renku dataset add My Data ~/ .bash_logout .cache/ .config/ .jupyter/ .npm/ testproject/ .yarn/ .bashrc .conda/ .gitconfig .local/ .profile work/ jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData/Pat1$ renku dataset add My Data ~/work/ Ahhhhhhhh! You have found a bug. 🐞 1. Open an issue by typing "open"; 2. Print human-readable information by typing "print"; 3. See the full traceback without submitting details (default: "ignore"). Please select an action by typing its name (open, print, ignore) [ignore]: Traceback (most recent call last): File "/home/jovyan/.local/bin/renku", line 10, in <module> sys.exit(cli()) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_exc.py", line 109, in main self._handle_github() File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_exc.py", line 155, in _handle_github getattr(self, '_process_' + value)() File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_exc.py", line 100, in main result = super().main(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/_client.py", line 80, in new_func result = ctx.invoke(method, client, *args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/dataset.py", line 235, in add force=force, File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/api/datasets.py", line 149, in add_data_to_dataset git = git or check_for_git_repo(url) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/api/datasets.py", line 462, in check_for_git_repo Repo(u.path, search_parent_directories=True) File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.6/site-packages/git/repo/base.py", line 131, in __init__ raise NoSuchPathError(epath) git.exc.NoSuchPathError: /home/jovyan/testproject/data/MyData/Pat1/Data jovyan@jupyter-witon-witon-2dtestprojec-2dcaba3c8:~/testproject/data/MyData/Pat1$ Adrien

…

________________________________ From: Rok Roškar <notifications@github.com> Sent: 16 May 2019 16:32 To: SwissDataScienceCenter/renku-python Cc: Witon Adrien Jean-Baptiste; Mention Subject: Re: [SwissDataScienceCenter/renku-python] cli: renku dataset create data (#535) I just tried the same with v0.5.0 and I can't reproduce the issue. Can you provide maybe a few more commands that preceded the one that failed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#535?email_source=notifications&email_token=AMB35RXDKJL6QI2ZAGM5HLTPVVWAHA5CNFSM4HMRUJRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVSAAHQ#issuecomment-493092894>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMB35RUSR7354PA5STRMMTTPVVWAHANCNFSM4HMRUJRA>.

[rokroskar]

Hi @AWiton thanks for clarifying - however, it looks to me like this is an entirely different issue (handling whitespace) and may be related to #517. Are you still having the problem you reported initially, i.e. that re-creating a dataset with the same name fails?

[AWiton]

Ok Thanks Rok, Related to re-creating a dataset, it happens in a particular case, when we mix renku and filsystem command: $ renku dataset create Test $ rmdir Test/ $ renku dataset rm Test $ renku dataset create Test Ahhhhhhhh! You have found a bug. 🐞 (And can be solved be removing ourself: .renku/refs/datasets/Test) Another bug there: $ renku dataset ID NAME CREATED AUTHORS -------- ------ ------------------- -------------------------- 746ced25 MyData 2019-05-16 16:46:08 Adrien $ renku dataset create MyData/SubData Ahhhhhhhh! You have found a bug. 🐞 All the best, Adrien

…

________________________________ From: Rok Roškar <notifications@github.com> Sent: 22 May 2019 00:28:33 To: SwissDataScienceCenter/renku-python Cc: Witon Adrien Jean-Baptiste; Mention Subject: Re: [SwissDataScienceCenter/renku-python] cli: renku dataset create data (#535) Hi @AWiton<https://github.com/AWiton> thanks for clarifying - however, it looks to me like this is an entirely different issue (handling whitespace) and may be related to #517<#517>. Are you still having the problem you reported initially, i.e. that re-creating a dataset with the same name fails? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#535?email_source=notifications&email_token=AMB35RWEF4LFVOBAZLZ36WDPWRZRDA5CNFSM4HMRUJRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5LVXA#issuecomment-494582492>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMB35RX4MOKZLPLCNCOVKU3PWRZRDANCNFSM4HMRUJRA>.

[rokroskar]

Great, thanks for clarifying. I will have a look.

[rokroskar]

I still can't reproduce your first issue -- I just ran:

renku init
renku dataset create mydata
rmdir data/mydata
renku dataset rm mydata
renku dataset create mydata

This completes without issues. So I think we are still not quite getting to the bottom of this... could you try to run the sequence above?

[AWiton]

Maybe this is related to the SV instance: https://sv-renku.epfl.ch/j jovyan@jupyter-witon-witon-2dtestprojec-2dab9f108:~/testproject/RenkuTest$ renku init Initialized empty project in /home/jovyan/testproject/RenkuTest jovyan@jupyter-witon-witon-2dtestprojec-2dab9f108:~/testproject/RenkuTest$ renku dataset create mydata Creating a dataset ... OK jovyan@jupyter-witon-witon-2dtestprojec-2dab9f108:~/testproject/RenkuTest$ rmdir data/mydata/ jovyan@jupyter-witon-witon-2dtestprojec-2dab9f108:~/testproject/RenkuTest$ renku dataset rm mydata OK jovyan@jupyter-witon-witon-2dtestprojec-2dab9f108:~/testproject/RenkuTest$ renku dataset create mydata Ahhhhhhhh! You have found a bug. 🐞 Best, Adrien

…

________________________________ From: Rok Roškar <notifications@github.com> Sent: 22 May 2019 12:27 To: SwissDataScienceCenter/renku-python Cc: Witon Adrien Jean-Baptiste; Mention Subject: Re: [SwissDataScienceCenter/renku-python] cli: renku dataset create data (#535) I still can't reproduce your first issue -- I just ran: renku init renku dataset create mydata rmdir data/mydata renku dataset rm mydata renku dataset create mydata This completes without issues. So I think we are still not quite getting to the bottom of this... could you try to run the sequence above? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#535?email_source=notifications&email_token=AMB35RW4LK4L3MCOUZSNJZ3PWUNXJA5CNFSM4HMRUJRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV6TR5I#issuecomment-494745845>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMB35RUXMOHCNSF4TFPYIMDPWUNXJANCNFSM4HMRUJRA>.

[rokroskar]

Ah yes! you're right, turns out the renku version in our current production docker file has this bug. We will do a new release of the CLI soon and fix the underlying images. In the meantime you can upgrade the renku CLI by adding this line to your Dockerfile:

RUN pipx upgrade --pip-args="--pre" renku