Cleanup volumes: 3 locations for volumes with each different behaviors is too much

[thomas-riccardi]

dmake volume current state

dmake supports 3 locations for volumes definitions:

services[].config.volumes

• shared (named) volume mount: (there is only one shared volume behavior !)

  • no host path by definition (like docker(-compose) named volumes)
  • supports variable substitution in target path

  => used to shared data between services and docker_links

• normal (host) volume mount:

  • full host paths only: explicit error with relative host paths
  • host paths are rewritten as relative to repo/.dmake/volumes/, but in fact it doesn work: it just mounts on the standard host path: dmake local mount volume: doesn't work as expected #241

  => bizarre, probably not used a lot, use tests.data_volumes instead: it supports variable substitution and no hack on host path .dmake/volumes. In practice only deprecated repos use it, others have migrated to tests.data_volumes.

• always mounted
  (- also SSH deploy usage: unused and deprecated for multiple years)

services[].tests.data_volumes

• no shared (named) volume support (yet?)
• supports file:// and s3:// source path (but s3 not used anymore IIRC)
• supports variable substitution in container_volume and source paths
• supports optional read-only
• supports relative host path for source
• always mounted

=> used for cache: avoid re-downloading large resources on each dmake execution.

docker_links[].volumes

• shared (named) volume mount: (there is only one shared volume behavior !)

  • no host path by definition (like docker(-compose) named volumes)
  • supports variable substitution in target path
  • always mounted

  => used to shared data between services and docker_links

• host volume mount:

  • full host paths only: explicit error with relative host paths
  • supports variable substitution in host_volume path
  • strangely in validator we require starting with / (before variable substitution), but later when generating docker run options allow relative path, and reconstruct full path for docker
  • only mounted for dmake shell and dmake run --docker-links-volumes-persistence

  => used for developers persistence, mainly on postgresql database

Needs

current usages/needs:

• cache (for both dev and tests): tests.data_volumes
• sharing (for both dev and tests): shared volumes: volumes on services[].config and docker_links[]
• persistence (for dev only): only available for docker_links for now (but only with full host path: not great to write randomly in host; ideally would store that somewhere in repo root/.dmake)

new needs:

• need persistence for services too.
  • cleaner persistence: don't store on use-chosen full host paths: maybe in repo_root/.dmake?
• (variable substitution everywhere)
• replace docker_links by real services, see Deprecate docker links: services can fully replace them #366: need persistence for services[].config.volumes like docker_links.volumes have

deprecated needs:

• s3 not used anymore
  (- delete other volumes related to deprecated Deployment types: ssh, aws beanstalk...)

maybe still allow persistence for dev vs cache on services but maybe do that with dmake.yml extra values in same volumes instead of split between config.volumes and tests.data_volumes ?

[thomas-riccardi]

For persistence, maybe explicit things:
kind: temporary/persistent/cache/shared-named

• temporary: just use shared/named volumes: it can be seen as the same thing
• persistent: somehow make it clear that there are still some magic:
  • shell: persistence
  • run: no persistence, except with --docker-links-volumes-persistence
  • test: never persistence
• cache:
  • either use static host path
  • or use .dmake/ stable subdir
  • or use stable named volume?

subdir or real named/docker volumes?

• subdir makes it easy to manipulate to switch between multiple (no docker volume rename...), which can be useful for persistence
• docker volumes
  • harder to control where it lands, (which partition), harder to see disk usage
  • but maybe clearer in dmake.yml: define the volume somewhere, with its kind: cache; then just refer to it in containers definitions; but we already have a full verbose format for non-named volumes, so that's fine