-
Notifications
You must be signed in to change notification settings - Fork 10
Description
Is your feature request related to a problem? Please describe.
As of today, execution of tasks has no guarantees around writing data in failed units. This should be handled.
Describe the solution you'd like
I propose that we had a new method that will both expand the functionality of unit contexts to be storage agnostic, but also provide robust failover capabilities. This will be done by rewriting the Context class to leverage the ExternalStorage for filesystem operations in shared/permanent cases. Furthermore, execute_DAG will be expanded to leverage these new primitives to provide a failure case for writing files back to storage mediums.
Describe alternatives you've considered
n/a
Additional context
This work is being done as part of exorcist to enable robust campaign-scale orchestration.