FileNotFoundError with renku run

[vfdev-5]

Describe the bug

I'm trying to run with renku the following command

PYTHONPATH=$PYTHONPATH:$PWD/src OUTPUT_PATH=$PWD/data python src/train_fast.py src/config.py

from this project

So, I run

renku run PYTHONPATH=$PYTHONPATH:$PWD/src OUTPUT_PATH=$PWD/data python src/train_fast.py src/config.py

and I got an error with a message to fill a bug.

Details

Please verify and redact the details.

Renku version: 0.5.0
OS: Linux (#1 SMP Mon Oct 22 10:40:32 EDT 2018)
Python: 3.6.8

Traceback

Traceback (most recent call last):
  File "[...]/renku/cli/_exc.py", line 100, in main
    result = super().main(*args, **kwargs)
  File "[...]/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "[...]/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/_client.py", line 80, in new_func
    result = ctx.invoke(method, client, *args, **kwargs)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/run.py", line 195, in run
    for key in mapped_std.keys()},
  File "[...]/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "[...]/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "[...]/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'PYTHONPATH=:/home/jovyan/cifar10-project/src': 'PYTHONPATH=:/home/jovyan/cifar10-project/src'

Additional context

[jirikuncar]

The environment variables are not yet directly recorded.

[rokroskar]

@vfdev-5 you need to persist the PYTHONPATH in your environment before running renku run. Could you try with

export PYTHONPATH=$PYTHONPATH:$PWD/src
export OUTPUT_PATH=$PWD/data
renku run python src/train_fast.py src/config.py

Note that as @jirikuncar, these variables will not be recorded in the metadata we store about this execution, therefore other renku commands e.g. renku update may fail.

For the output path, it would be better to read it in to the script from the command line with something like e.g.

renku run python src/train_fast.py src/config.py --output-path=${PWD}/data

This way, the output directory will be recorded as a dependency.

[rokroskar]

@vfdev-5 were you able to fix the problem?

[vfdev-5]

@rokroskar thanks for the suggestions and sorry for the delay ! No, I haven't yet time to retest this but the suggestions look good. I close the issue as solved if it is OK for you too.

[rokroskar]

Sure, sounds good - please reopen if you continue having trouble or find us on gitter: https://gitter.im/SwissDataScienceCenter/renku