renku dataset add failed

[AWiton]

Describe the bug

Add data from repository A to the project.
Did a git pull on repository B (mounted from smb) to get data from the project (with new data from A).
$ git pull
[get last data]
Add new data
$ renku dataset add --relative-to ~/Hummel-Arch/TiMeS/WP11/ WP1.1 ~/Hummel-Arch/TiMeS/WP11/raw/TiMeS_WP11_003/T3/eeg/

Ahhhhhhhh! You have found a bug. 🐞

Details

Please verify and redact the details.

Renku version: 0.5.0
OS: Linux (#1 SMP Thu Nov 29 14:49:43 UTC 2018)
Python: 3.6.4

Traceback

Traceback (most recent call last):
  File "[...]/renku/cli/_exc.py", line 100, in main
    result = super().main(*args, **kwargs)
  File "[...]/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "[...]/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/_client.py", line 80, in new_func
    result = ctx.invoke(method, client, *args, **kwargs)
  File "[...]/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/dataset.py", line 225, in add
    with client.with_dataset(name=name) as dataset:
  File "[...]/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "[...]/renku/api/datasets.py", line 113, in with_dataset
    dataset = self.load_dataset(name=name)
  File "[...]/renku/api/datasets.py", line 99, in load_dataset
    dataset = Dataset.from_yaml(path)
  File "[...]/renku/models/_jsonld.py", line 360, in from_yaml
    yaml.safe_load(fp) or {}, __reference__=path
  File "[...]/renku/models/datasets.py", line 244, in from_jsonld
    self = super().from_jsonld(data, *args, **kwargs)
  File "[...]/renku/models/_jsonld.py", line 310, in from_jsonld
    raise ValueError(data)
ValueError: ../../datasets/1aef1ffeca8a4533a5f770db2856a41a/metadata.yml

Additional context

Add any other context about the problem.

[rokroskar]

Hi @AWiton we have recently changed quite a bit about how the renku dataset add works - specifically, we replaced the --target and --relative-to flags with --source and --destination -- please see the docs. Can you try to upgrade your renku client to 0.7.0 and please let me know if this continues to be a problem?

[AWiton]

Thanks @rokroskar for the update. With the new version, there is a bug for the renku dataset command. Is there something else to update?
$renku dataset
Ahhhhhhhh! You have found a bug. 🐞

1. Open an issue by typing "open";
2. Print human-readable information by typing "print";
3. See the full traceback without submitting details (default: "ignore").

Please select an action by typing its name (open, print, ignore) [ignore]:
Traceback (most recent call last):
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 686, in compact
active_ctx = self.process_context(active_ctx, ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 1199, in process_context
return self._process_context(active_ctx, local_ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 2833, in _process_context
self._create_term_definition(rval, ctx, k, defined)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 4441, in _create_term_definition
}, code='invalid IRI mapping')
pyld.jsonld.JsonLdError: ('Invalid JSON-LD syntax; @context terms must define an @id.',)
Type: jsonld.SyntaxError
Code: invalid IRI mapping
Details: {'context': {'dcterms': 'http://purl.org/dc/terms/', 'dctypes': 'http://purl.org/dc/dcmitypes/', 'foaf': 'http://xmlns.com/foaf/0.1/', 'prov': 'http://www.w3.org/ns/prov#', 'scoro': 'http://purl.org/spar/scoro/', 'name': 'dcterms:name', 'created': 'http://schema.org/dateCreated', 'identifier': {'@id': 'dctypes:Dataset', '@type': '@id'}, 'authors': {'@container': '@list'}, 'email': 'dcterms:email', 'affiliation': 'scoro:affiliate', 'files': {'@container': '@index'}, 'url': 'http://schema.org/url', 'added': 'http://schema.org/dateCreated'}, 'term': 'authors'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/adi/.local/bin/renku", line 10, in
sys.exit(cli())
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_exc.py", line 109, in main
self._handle_github()
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_exc.py", line 155, in _handle_github
getattr(self, 'process' + value)()
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_exc.py", line 100, in main
result = super().main(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 1114, in invoke
return Command.invoke(self, ctx)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_client.py", line 80, in new_func
result = ctx.invoke(method, client, *args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/dataset.py", line 183, in dataset
datasets = client.datasets.values()
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/api/datasets.py", line 78, in datasets
result[path] = Dataset.from_yaml(path)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/models/_jsonld.py", line 360, in from_yaml
yaml.safe_load(fp) or {}, reference=path
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/models/datasets.py", line 244, in from_jsonld
self = super().from_jsonld(data, *args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/models/_jsonld.py", line 328, in from_jsonld
compacted = ld.compact(data, {'@context': cls.jsonld_context})
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 145, in compact
return JsonLdProcessor().compact(input, ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 690, in compact
'jsonld.CompactError', cause=cause)
pyld.jsonld.JsonLdError: ('Could not process context before compaction.',)
Type: jsonld.CompactError
Cause: ('Invalid JSON-LD syntax; @context terms must define an @id.',)
Type: jsonld.SyntaxError
Code: invalid IRI mapping
Details: {'context': {'dcterms': 'http://purl.org/dc/terms/', 'dctypes': 'http://purl.org/dc/dcmitypes/', 'foaf': 'http://xmlns.com/foaf/0.1/', 'prov': 'http://www.w3.org/ns/prov#', 'scoro': 'http://purl.org/spar/scoro/', 'name': 'dcterms:name', 'created': 'http://schema.org/dateCreated', 'identifier': {'@id': 'dctypes:Dataset', '@type': '@id'}, 'authors': {'@container': '@list'}, 'email': 'dcterms:email', 'affiliation': 'scoro:affiliate', 'files': {'@container': '@index'}, 'url': 'http://schema.org/url', 'added': 'http://schema.org/dateCreated'}, 'term': 'authors'} File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 686, in compact
active_ctx = self.process_context(active_ctx, ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 1199, in process_context
return self._process_context(active_ctx, local_ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 2833, in _process_context
self._create_term_definition(rval, ctx, k, defined)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 4441, in _create_term_definition
}, code='invalid IRI mapping')

[rokroskar]

can you try renku migrate datasets?

[AWiton]

This is the same.

$ renku migrate datasets
$ renku dataset
Ahhhhhhhh! You have found a bug. 🐞

1. Open an issue by typing "open";
2. Print human-readable information by typing "print";
3. See the full traceback without submitting details (default: "ignore").

Please select an action by typing its name (open, print, ignore) [ignore]:
Traceback (most recent call last):
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 686, in compact
active_ctx = self.process_context(active_ctx, ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 1199, in process_context
return self._process_context(active_ctx, local_ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 2833, in _process_context
self._create_term_definition(rval, ctx, k, defined)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 4441, in _create_term_definition
}, code='invalid IRI mapping')
pyld.jsonld.JsonLdError: ('Invalid JSON-LD syntax; @context terms must define an @id.',)
Type: jsonld.SyntaxError
Code: invalid IRI mapping
Details: {'context': {'dcterms': 'http://purl.org/dc/terms/', 'dctypes': 'http://purl.org/dc/dcmitypes/', 'foaf': 'http://xmlns.com/foaf/0.1/', 'prov': 'http://www.w3.org/ns/prov#', 'scoro': 'http://purl.org/spar/scoro/', 'name': 'dcterms:name', 'created': 'http://schema.org/dateCreated', 'identifier': {'@id': 'dctypes:Dataset', '@type': '@id'}, 'authors': {'@container': '@list'}, 'email': 'dcterms:email', 'affiliation': 'scoro:affiliate', 'files': {'@container': '@index'}, 'url': 'http://schema.org/url', 'added': 'http://schema.org/dateCreated'}, 'term': 'authors'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/adi/.local/bin/renku", line 10, in
sys.exit(cli())
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_exc.py", line 109, in main
self._handle_github()
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_exc.py", line 155, in _handle_github
getattr(self, 'process' + value)()
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_exc.py", line 100, in main
result = super().main(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 1114, in invoke
return Command.invoke(self, ctx)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/_client.py", line 80, in new_func
result = ctx.invoke(method, client, *args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/dataset.py", line 183, in dataset
datasets = client.datasets.values()
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/api/datasets.py", line 78, in datasets
result[path] = Dataset.from_yaml(path)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/models/_jsonld.py", line 360, in from_yaml
yaml.safe_load(fp) or {}, reference=path
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/models/datasets.py", line 244, in from_jsonld
self = super().from_jsonld(data, *args, **kwargs)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/models/_jsonld.py", line 328, in from_jsonld
compacted = ld.compact(data, {'@context': cls.jsonld_context})
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 145, in compact
return JsonLdProcessor().compact(input, ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 690, in compact
'jsonld.CompactError', cause=cause)
pyld.jsonld.JsonLdError: ('Could not process context before compaction.',)
Type: jsonld.CompactError
Cause: ('Invalid JSON-LD syntax; @context terms must define an @id.',)
Type: jsonld.SyntaxError
Code: invalid IRI mapping
Details: {'context': {'dcterms': 'http://purl.org/dc/terms/', 'dctypes': 'http://purl.org/dc/dcmitypes/', 'foaf': 'http://xmlns.com/foaf/0.1/', 'prov': 'http://www.w3.org/ns/prov#', 'scoro': 'http://purl.org/spar/scoro/', 'name': 'dcterms:name', 'created': 'http://schema.org/dateCreated', 'identifier': {'@id': 'dctypes:Dataset', '@type': '@id'}, 'authors': {'@container': '@list'}, 'email': 'dcterms:email', 'affiliation': 'scoro:affiliate', 'files': {'@container': '@index'}, 'url': 'http://schema.org/url', 'added': 'http://schema.org/dateCreated'}, 'term': 'authors'} File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 686, in compact
active_ctx = self.process_context(active_ctx, ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 1199, in process_context
return self._process_context(active_ctx, local_ctx, options)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 2833, in _process_context
self._create_term_definition(rval, ctx, k, defined)
File "/home/adi/.local/pipx/venvs/renku/lib/python3.7/site-packages/pyld/jsonld.py", line 4441, in _create_term_definition
}, code='invalid IRI mapping')

[rokroskar]

@AWiton it still looks like your client is not updated to the most recent version. Could you do renku --version? Since you are using pipx, you should be able to do pipx upgrade renku.

[AWiton]

Thanks, addition now is working with renku 0.7.1; but the dataset added are no more in LFS.

[AWiton]

With python Renku 0.7.2, here is the output:

renku dataset add newdataset --source data/young/TiMeS_WP12_001/eeg/TiMeS_WP12_001_0002.vhdr --destination TiMeS_WP12_001/TiMeS_WP12_001_0002.vhdr https://witon:xxxxxxxxxxxxxx@sv-renku-git.epfl.ch/witon/times_wp12_db.git

Ahhhhhhhh! You have found a bug. 🐞

1. Open an issue by typing "open";
2. Print human-readable information by typing "print";
3. See the full traceback without submitting details (default: "ignore").

Please select an action by typing its name (open, print, ignore) [ignore]: Traceback (most recent call last):
File "/home/jovyan/.local/bin/renku", line 8, in
sys.exit(cli())
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/exception_handler.py", line 126, in main
self._handle_github()
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/exception_handler.py", line 173, in _handle_github
getattr(self, 'process' + value)()
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/exception_handler.py", line 117, in main
result = super().main(*args, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/exception_handler.py", line 90, in main
return super().main(*args, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/cli/dataset.py", line 417, in add
urlscontext=progress
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/core/commands/client.py", line 86, in new_func
result = ctx.invoke(method, client, *args, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/core/commands/dataset.py", line 151, in add_file
urlscontext
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/core/commands/dataset.py", line 180, in add_to_dataset
destination=destination,
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/core/management/datasets.py", line 183, in add_data_to_dataset
dataset, dataset_path, url, sources, destination
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/renku/core/management/datasets.py", line 343, in _add_from_git
repo = Repo.clone_from(url, repo_path, recursive=True)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/git/repo/base.py", line 1026, in clone_from
return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/git/repo/base.py", line 967, in _clone
finalize_process(proc, stderr=stderr)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/git/util.py", line 332, in finalize_process
proc.wait(**kwargs)
File "/home/jovyan/.local/pipx/venvs/renku/lib/python3.7/site-packages/git/cmd.py", line 412, in wait
raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git clone --recursive -v https://witon:xxxxxxxxxxxxxx@sv-renku-git.epfl.ch/witon/times_wp12_db.git /tmp/tmp25i9d4k8
stderr: 'Cloning into '/tmp/tmp25i9d4k8'...
POST git-upload-pack (177 bytes)
fatal: read error: Bad address
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

'
nothing particular on git log or git status.

[rokroskar]

cc @mohammad-sdsc - any ideas here?

[mohammad-alisafaee]

@AWiton Can you clone the repo outside Renku: git clone --recursive -v https://witon:xxxxxxxxxxxxxx@sv-renku-git.epfl.ch/witon/times_wp12_db.git? Also, for git logs and status did you check were the repo is cloned (which is a random directory in /tmp, in this case, /tmp/tmp25i9d4k8)?

[mohammad-alisafaee]

I could not reproduce this problem with renku v0.7.2 and private repositories with an access token:

$ renku dataset add b -s Dockerfile https://mohammad.alisafaee:***@dev.renku.ch/gitlab/renku-devs/test-private-changed.git
Adding data to dataset  [      ]  1/1

[rokroskar]

Aha ok so it fails in an interactive session? Are you sure disk space is not a problem?

[AWiton]

The file updated is a text file, so it should not be a pb. I am able to upload files for ~100Mb.

[rokroskar]

When you say it works outside of renku, you mean you did the clone of the project in the interactive session manually? Or you did it locally on your machine?