renku tutorial bug

[EricaMoreira]

Describe the bug

On the last step of the Renku tutorial renku update, I encountered a bug.

The error I got was:

[workflow ] start
[workflow ] starting step step_2
[step step_2] start
[job step_2] /tmp/tmpwzrf_0az$ python \
    /tmp/tmpcg_bp1of/stgc255d2b1-f136-44c1-bd0e-da7d5874fa5a/clean_data.py \
    /tmp/tmpcg_bp1of/stg6a5cd75d-2f91-4f07-9f04-bd73af3dd61e/zhbikes \
    data/preprocessed/zhbikes.parquet
[job step_2] Job error:
Error collecting output for parameter 'output_0':
.renku/workflow/6950df46081147c587b5a94fa9d89b5e_python.cwl:40:7: Did not find output file with glob pattern: '['data/preprocessed/zhbikes.parquet']'
[job step_2] completed permanentFail
[step step_2] Output is missing expected field file:///home/jovyan/test-project/.renku/workflow/0812718ca50f4d89b39b8730814bd97c.cwl#step_2/output_0
[step step_2] completed permanentFail
[workflow ] completed permanentFail
Ahhhhhhhh! You have found a bug. 🐞

Details

Renku version: 0.5.0
OS: Linux (#1 SMP Wed Jul 10 08:46:17 EDT 2019)
Python: 3.6.8

Traceback

Traceback (most recent call last):
  File "[...]/renku/cli/_exc.py", line 100, in main
    result = super().main(*args, **kwargs)
  File "[...]/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "[...]/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/_client.py", line 80, in new_func
    result = ctx.invoke(method, client, *args, **kwargs)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/update.py", line 190, in update
    execute(client, output_file, output_paths=output_paths)
  File "[...]/renku/cli/_cwl.py", line 71, in execute
    outputs = process()
  File "[...]/cwltool/factory.py", line 35, in __call__
    raise WorkflowStatus(out, status)
cwltool.factory.WorkflowStatus: Completed permanentFail

[rokroskar]

Hi @EricaMoreira thanks for reporting this - could you share more of the output - what happened during step1 and more of the stack trace if possible?

[EricaMoreira]

I redid the tutorial from scratch (made a new project, started and connected to a new server) and this time the renku update step didn’t give me an error, but at the very beginning of the tutorial I got an error saying the repo is dirty. I ended up having to commit and push a .renku.lock file in order to move forward… but then I was able to complete the tutorial without issue.

I actually went ahead and started the tutorial from scratch a third time, just to understand when exactly the .renku.lock file was being created. When I make a new project the .renku.lock file is not there, but it appears after I run the first command renku dataset create zhbikes.

Here is the error:

jovyan@jupyter-ericamorei-d0332fd74a22ed2b:~/test-project-3$ renku dataset create zhbikes
Creating a dataset ... OK
jovyan@jupyter-ericamorei-d0332fd74a22ed2b:~/test-project-3$ renku dataset add zhbikes https://data.stadt-zuerich.ch/dataset/verkehrszaehlungen_werte_fussgaenger_velo/resource/d17a0a74-1073-46f0-a26e-46a403c061ec/download/2019_verkehrszaehlungen_werte_fussgaenger_velo.csv
Error: The repository is dirty. Please use the "git" command to clean it.

On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .renku.lock

nothing added to commit but untracked files present (use "git add" to track)

Once you have added the untracked files, commit them with "git commit".

[ableuler]

@EricaMoreira Thanks again for reporting this issue. There was an error in the template which is used when creating a new project through the UI which led to renku complaining about the "dirty repo". This is fixed now.
I ran through the rest of the first step tutorial and - just like you - I could not reproduce your initially reported bug during renku update. I'm therefore closing this issue. Do not hesitate to reopen it or open a new one should you run into this or another problem again.