At-most-once execution for programs running inside containers (ContainerExec)

[karolz-ms]

Feature overview

DCP users should be able to specify that a particular program execution inside a container should happen only once. The main difficulty here lies with persistent containers. ContainerExec instances can be created at any time during corresponding Container lifetime, including after a DCP-orchestrated application is restarted and after the persistent Contaienr (or, more precisely, the underlying Docker/Podman container) is reused. This means we need to have a way to memorize if an instance of ContainerExec has already been run for a partincualr persistent Container instance, and if yes, retrieve the result of previous run and apply it to ContainerExec Status.

The previous run data that needs to be restored (if applicable) contains the following:

1. Startup and finish timestamps
2. Exit code
3. Logs (stdout and stderr)
4. Effective environment variable values (after substitutions)
5. Effective invocation arguments (after substitutions)

Some implementation options

The key design question is where to place the data related to at-most-once ContainerExec runs. Two main options were discussed at initial design meeting (they are not necessarily exclusive).

"System" container

The idea is to create a DCP-managed container and use its filesystem to store ContainerExec data..

Pros

1. Data can be easily separated per-application by using different container.
2. Space can be reclaimed easily by deleting the container.
3. The container can be used in "file share" mode (not running, just having the data copied to/from the container), or in "active" mode (running with a server that provides a higher-level API and synchronizes access to data, potentially for multiple instances).

Cons

1. Creating the "system" container represents additional delay during startup sequence
2. Storing the data requires the container orchestrator to be running and healthy. Error handling may be complicated than with the alternative.
3. Does not help with storing machine-wide user data for applications that do not use containers

Database stored inside user profile

Here we are talking about a SQLite-like database with ContainerExec data stored in the database alongside other DCP system data. Part of the data (e.g. logs) might occupy separate files, although this is probably not necessary given a good support for TEXT/BLOB data in SQLite.

Pros

1. Data can be stored in optimal, relational format with very rich querying capabilities and transaction support.
2. SQLite has very good support for accessing and modifying data from multiple concurrent program instances.
3. Data is accessible whenever DCP is running--there is no dependency on container orchestration.
4. The database can be used for storing data related to all DCP-orchestrated applications, including those that do not use containers.

Cons

1. Different versions of DCP may need different schemas, requiring versioning strategy, or at least a strategy for using different database instances between versions.
2. Reclamation of the space (cleaning up stale data) might be more complicated and require explicit code to delete state data and compact the database.
3. Using SQLite from a Go program might increase the executable size and represents somewhat novel architecture (with original SQLite code cross-compiled into Go).

[karolz-ms]

@mitchdenny FYI

[mitchdenny]

Related: microsoft/aspire#10951