workflow UX improvements

[Panaetius]

As a result of multiple discussions over the past year, we've come to the conclusion that we don't just want to overhaul the metadata for workflows and how it's stored, but also the UX regarding workflows.

This epic serves as the central place for keeping discussions and to outline the future work to be done.

See https://github.com/SwissDataScienceCenter/renku-python/blob/master/design/001-new-workflow-commands/001-new-workflow-commands.md for more details.

[schymans]

Thanks a lot for this vision of workflows in renku! In the last example (Recurring data import), I wonder how the link to external data would be saved and show up in the knowledge graph. As I understand it, linking knowledge graphs from different projects, even different renku deployments, was the great aim from the start, and I wondered if this could be the place to start implementing it. Moreover, would it be possible to not just get data from an external source, but also code? For example, if I develop a script to process a certain type of data in Repo1, then use this script in Repo2, I would like to be able to include that link in the workflow, i.e. transfer any updates to the script in Repo1 to Repo2 and then process the data in Repo2 using this updated script. Is such a use scenario on the horizon at all?

[Panaetius]

If the link to the external data was a parameter in the command used to get it (e.g. a URL), it'd show up as a parameter. In the example above it'd be hidden inside the download.sh file and not immediately apparent (Only that script would show up in the knowledge graph).

We do envision that the changes outlined above will lead to a command like renku workflow import <id> that allows importing (and updating from) a workflow from a different renku project, similar to how renku dataset import works. But this isn't fully fleshed out yet.

[schymans]

Actually, I didn't realise that your examples don't use any of the renku dataset functions. If we do renku run wget URL, where the URL would be the URL of a code file in a different project, would this link between project show up in the KG? Maybe I was too much fixated on the dataset method when thinking about getting code from another project.

[Panaetius]

It wouldn't exactly link to the other project, it would just show up as a string parameter value to a run in the KG