Pipeline: log file preservation for failing steps and new backward compatibility system#276
Pipeline: log file preservation for failing steps and new backward compatibility system#276taimontgomery merged 11 commits intomasterfrom
Conversation
…nputs to subdirectory subworkflows. These are now handled by cwltool's --log-dir option, which has the added benefit of preserving captured outputs from failing steps
… stdout/stderr captures are preserved for steps that produce errors
…ation so that it conforms to the most recent Features Sheet format
|
What about storing the log files for each step in the same directory, for example, one directory for bowtie with each library, one directory for collapser, etc.? Its a little bit busy with each library in its own directory. |
|
That was my initial preference too but I think this approach actually makes the most sense from a troubleshooting perspective because the job ID is printed several times around each step including in the final error output (hard to miss). We've been looking at the terminal output from tinyRNA a lot so we know exactly what to look for when identifying which library a job is addressing, but for new users it would be a needle in the haystack that is the cwltool output. I anticipate that most users will only look at the logs when they are troubleshooting a failed step. |
…of tinyRNA tools. Updating CWL step names for tinyRNA tools so that the step name matches the tool name. Workflow step IDs are derived from the step name, and log file subdirectories are currently named from step IDs in the cwltool runner. Similarly, defaults for output subdirectory names have been updated to follow the same format
…of tinyRNA tools. Updating CWL step names for tinyRNA tools so that the step name matches the tool name. Workflow step IDs are derived from the step name, and log file subdirectories are currently named from step IDs in the cwltool runner. Similarly, defaults for output subdirectory names have been updated to follow the same format
…gn inconsistency in ResumePlotterConfig
|
I've updated the workflow CWL to produce log and step output subdirectories that match tinyRNA's current tool names. Users won't have to infer that collapser, counter, dge, and plotter refer to tiny-collapse, tiny-count, etc. which might help with the log directory optics. Unfortunately cwltool only uses the New backward compatibility systemI've also introduced a basic system for backward compatibility with configuration files. In the past I made efforts to avoid compatibility issues with old configuration files, but the approach was disorganized and meant to be temporary. The new system centralizes this patchwork and provides a more robust means of documenting changes in a way that is clear and easy to maintain. Right now only the Run Config is supported for only the changes introduced in this PR, but once more pressing tasks are completed I'll loop back to this and finish the implementation. The idea is that changes (keys added, renamed, and removed) are defined in a new YAML file ( |
The new subdirectory
logshas been added to the Run Directory. Log files for all workflow steps will be stored here rather than in their respective step subdirectories. This allows for preservation of logs that are produced by failing workflow steps, which is critical for troubleshooting.Closes #275