Skip to content

renku workflow iterate crashes with SLURM #2587

@cmdoret

Description

@cmdoret

Attempting to use renku workflow iterate -p toil with SLURM results in a crash.
Additionally, it causes subsequent runs of renku log in the project to error.

Using the same workflow:

  • renku workflow execute works with SLURM
  • renku workflow iterate works with -p toil without SLURM

To Reproduce
Steps to reproduce the behavior:
0. renku init

  1. Create workflow:
renku -S run --name demo1 -- echo 'demo1' > demo1.out
  1. Run iterate on SLURM through toil:
TOIL_SLURM_ARGS="-p parallel -A free --export=ALL" renku -S workflow iterate \
    -m 'parameter-1=[1, 2, 3]' \
    -m output-2='demo1_iter{iter_index}.out' \
    -p toil \
    -c conf.yaml \
    demo1

Expected behavior
The workflow runs 3 times, creating 3 output files demo1_{0,1,2}.

Observed behaviour
The command crashes with the following traceback:

Traceback (most recent call last):
  File "[...]/site-packages/renku/cli/exception_handler.py", line 126, in main
    result = super().main(*args, **kwargs)
  File "[...]/site-packages/renku/cli/exception_handler.py", line 92, in main
    return super().main(*args, **kwargs)
  File "[...]/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "[...]/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "[...]/site-packages/renku/cli/workflow.py", line 1148, in iterate
    iterate_workflow_command().with_communicator(communicator).build().execute(
  File "[...]/site-packages/renku/core/management/command_builder/command.py", line 242, in execute
    output = context["click_context"].invoke(self._operation, *args, **kwargs)
  File "[...]/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "[...]/site-packages/renku/core/commands/workflow.py", line 784, in _iterate_workflow
    execute_workflow(dag=graph.workflow_graph, command_name="iterate", provider=provider, config=config)
  File "[...]/site-packages/inject/__init__.py", line 342, in injection_wrapper
    return sync_func(*args, **kwargs)
  File "[...]/site-packages/renku/core/commands/workflow.py", line 463, in execute_workflow
    execute(dag=dag, basedir=client.path, provider=provider, config=config)
  File "[...]/site-packages/renku/core/plugins/provider.py", line 83, in execute
    return executor(dag=dag, basedir=basedir, config=config)
  File "[...]/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "[...]/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "[...]/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "[...]/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "[...]/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "[...]/site-packages/renku/core/management/workflow/providers/toil.py", line 227, in workflow_execute
    [setattr(options, k, v) for k, v in config.items()]
AttributeError: 'str' object has no attribute 'items'

Renku version: 1.0.1
OS: Linux (#1 SMP Thu Jul 8 00:48:33 UTC 2021)
Python: 3.9.7

After this crash, running renku log also causes the following error:

Traceback (most recent call last):
  File "[...]/site-packages/inject/__init__.py", line 342, in injection_wrapper
    return sync_func(*args, **kwargs)
  File "[...]/site-packages/renku/core/commands/log.py", line 38, in _log
    log_entries = [LogViewModel.from_activity(a) for a in activities]
  File "[...]/site-packages/renku/core/commands/log.py", line 38, in <listcomp>
    log_entries = [LogViewModel.from_activity(a) for a in activities]
  File "[...]/site-packages/renku/core/commands/view_model/log.py", line 74, in from_activity
    description=" ".join(activity.plan_with_values.to_argv()),
TypeError: sequence item 1: expected str instance, int found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[...]/site-packages/renku/cli/exception_handler.py", line 126, in main
    result = super().main(*args, **kwargs)
  File "[...]/site-packages/renku/cli/exception_handler.py", line 92, in main
    return super().main(*args, **kwargs)
  File "[...]/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "[...]/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "[...]/site-packages/renku/cli/log.py", line 53, in log
    result = log_command().with_database().build().execute(workflows_only=workflows).output
  File "[...]/site-packages/renku/core/management/command_builder/command.py", line 242, in execute
    output = context["click_context"].invoke(self._operation, *args, **kwargs)
  File "[...]/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "[...]/site-packages/inject/__init__.py", line 344, in injection_wrapper
    raise ConstructorTypeError(func, previous_error)
inject.ConstructorTypeError: <function _log at 0x2b95c7b65ca0> raised an error: sequence item 1: expected str instance, int found

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions