Credentialspec from Swarm config not working

[jorisscheppers]

Description
In Docker 19.03 support was added to provide Swarm services with credentialspec json files loaded from configs instead of binding the file from the local filesystem (#1781). This does not seem to work in my case. I suspect a bug but it may be that I missed something.

Steps to reproduce the issue:
I created a config on the manager via UCP client bundle on client machine:

docker config create gmsa_ontcaching gmsa_ontcaching.json

gmsa_ontcaching.json file is located on the client machine, config is created successfully:

PS D:\> docker config ls
ID                          NAME                                                      CREATED             UPDATED
xs7b2ueepch3l7wpewnswdvf8   gmsa_ontcaching                                 32 hours ago        32 hours ago

Service definition in docker-compose.yml:

version: '3.8'
configs:
  gmsa_ontcaching:
    external:
      name: gmsa_ontcaching
networks:
  fzbo-network:
    driver: overlay
services:
  fzbo-caching:
    credential_spec:
      config: gmsa_ontcaching
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s
    hostname: gmsa_ontcaching
    image: repo-name/caching:tag-162
    networks:
      fzbo-network:
        aliases:
        - orleans

Describe the results you received:

PS D:\sources\compose> docker stack deploy -c .\docker-compose.yml cachetest
time="2019-09-25T13:54:21+02:00" level=warning msg="config gmsa_ontcaching: config.external.name is deprecated in favor of config.name"
Creating network cachetest_fzbo-network
Creating service cachetest_fzbo-caching
failed to create service cachetest_fzbo-caching: Error response from daemon: invalid CredentialSpec: must either provide "file", "registry", or "config" for credential spec

Describe the results you expected:
No errors and a service where the gMSA credentialspec was added from a Swarm config.

Additional information you deem important (e.g. issue happens only occasionally):
There are other methods of defining the config for the credentialspec, according to documentation (https://docs.docker.com/compose/compose-file/#example-gmsa-configuration):

version: "3.8"
services:
  myservice:
    image: myimage:latest
    credential_spec:
      config: my_credential_spec

configs:
  my_credentials_spec:
    file: ./my-credential-spec.json|

First of all, what is the pipe-symbol doing in the file definition?
Second: this specifies a service-specific config from a local file which is not my goal. But as a test I tried it this way as well, and the response was the same:

docker-compose.yml:

version: '3.8'
configs:
  gmsa_ontcaching:
    file: gmsa_ontcaching.json
networks:
  fzbo-network:
    driver: overlay
services:
  fzbo-caching:
    credential_spec:
      config: gmsa_ontcaching
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s
    hostname: gmsa_ontcaching
    image: repo-name/caching:tag-162
    networks:
      fzbo-network:
        aliases:
        - orleans

Output:

PS D:\sources\cz-docker\ForCZ\compose\ont> docker stack deploy -c .\docker-compose.caching.yml cachetest
Creating config cachetest_gmsa_ontcaching
Creating service cachetest_fzbo-caching
failed to create service cachetest_fzbo-caching: Error response from daemon: invalid CredentialSpec: must either provide "file", "registry", or "config" for credential spec

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.39 (downgraded from 1.40)
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:17:08 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Enterprise 2.1
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       c92ab06
  Built:            Tue Sep  3 15:55:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
 Universal Control Plane:
  Version:          3.1.4
  ApiVersion:       1.39
  Arch:             amd64
  BuildTime:        Wed Feb 27 22:26:43 UTC 2019
  GitCommit:        29b16f9
  GoVersion:        go1.10.6
  MinApiVersion:    1.20
  Os:               linux
 Kubernetes:
  Version:          1.11+
  buildDate:        2019-02-05T21:33:23Z
  compiler:         gc
  gitCommit:        8249259df545d6e853b7de9d050a34ec24e8b2f5
  gitTreeState:     clean
  gitVersion:       v1.11.7-docker-1
  goVersion:        go1.10.7
  major:            1
  minor:            11+
  platform:         linux/amd64
 Calico:
  Version:          v3.5.0
  cni:              v3.5.0
  kube-controllers: v3.5.0
  node:             v3.5.0

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 421
  Running: 240
  Paused: 0
  Stopped: 181
 Images: 599
 Server Version: ucp/3.1.4
 Role: primary
 Strategy: spread
 Filters: health, port, containerslots, dependency, affinity, constraint, whitelist
 Nodes: 18
  REDACTED: 10.5.105.65:12376
   └ ID: D3VP:6S42:674O:VPKF:X7LS:2TNG:DSNZ:I6U5:FKZT:5F34:2D73:3XKA|10.5.105.65:12376
   └ Status: Healthy
   └ Containers: 39 (27 Running, 0 Paused, 12 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 340 MiB / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.66:12376
   └ ID: USZW:IHMU:JJRB:U6PY:3DOH:IAHE:E5CE:ECRZ:XV4U:LTON:XGD6:HZ4B|10.5.105.66:12376
   └ Status: Healthy
   └ Containers: 24 (21 Running, 0 Paused, 3 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:20Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.67:12376
   └ ID: RPWP:MUXF:CKW5:FXRQ:J2AB:AKP5:QSSP:ZWWK:YGEA:UZFJ:KFYG:CSTJ|10.5.105.67:12376
   └ Status: Healthy
   └ Containers: 16 (16 Running, 0 Paused, 0 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.209:12376
   └ ID: 6S26:RXWW:FRZJ:QK5S:EGSQ:XLDD:MMN6:B6NX:GBLB:BFUX:FCNX:RWKH|10.1.66.209:12376
   └ Status: Healthy
   └ Containers: 7 (4 Running, 0 Paused, 3 Stopped)
   └ Reserved CPUs: 0 / 4
   └ Reserved Memory: 0 B / 8.02 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:17Z
   └ ServerVersion: 18.09.2
  REDACTED: 10.2.66.47:12376
   └ ID: X3CR:SVG2:GOBW:MAVC:ZRSL:E6AR:6INB:TNSM:P5W6:GGY4:W2K4:EEYT|10.2.66.47:12376
   └ Status: Healthy
   └ Containers: 28 (28 Running, 0 Paused, 0 Stopped)
   └ Reserved CPUs: 0 / 25
   └ Reserved Memory: 0 B / 67.2 GiB
   └ Labels: kernelversion=10.0 14393 (14393.3179.amd64fre.rs1_release_1.190806-2245), operatingsystem=Windows Server 2016 Standard Version 1607 (OS Build 14393.3181), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:20Z
   └ ServerVersion: 18.09.3
  REDACTED: 10.1.66.35:12376
   └ ID: CGUM:LTD2:7ACF:CI6Y:MISJ:LN7F:J6R7:4UUI:4WI5:WXG2:CP75:RNRS|10.1.66.35:12376
   └ Status: Healthy
   └ Containers: 105 (28 Running, 0 Paused, 77 Stopped)
   └ Reserved CPUs: 0 / 12
   └ Reserved Memory: 0 B / 16.8 GiB
   └ Labels: kernelversion=10.0 14393 (14393.3179.amd64fre.rs1_release_1.190806-2245), operatingsystem=Windows Server 2016 Standard Version 1607 (OS Build 14393.3181), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 18.09.3
  REDACTED: 10.1.66.215:12376
   └ ID: NOTJ:526Z:TRKL:W5F5:GPOA:EKYJ:M6QE:KYL5:D3ZZ:PQQD:4HL2:B5NB|10.1.66.215:12376
   └ Status: Healthy
   └ Containers: 3 (2 Running, 0 Paused, 1 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-25T07:49:40Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.216:12376
   └ ID: 65GA:LZSK:KKGJ:2AM5:HMHR:YZ7L:QRXL:JJY7:K266:RNG4:AGLL:LHZY|10.1.66.216:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:00Z
   └ ServerVersion: 19.03.2
 REDACTED: 10.1.66.217:12376
   └ ID: X6TE:BHLD:QMPO:ISZ2:FJV5:QTHO:RJGJ:LMGM:6CYK:YAZC:ZF42:CWXC|10.1.66.217:12376
   └ Status: Healthy
   └ Containers: 34 (26 Running, 0 Paused, 8 Stopped)
   └ Reserved CPUs: 0 / 8
   └ Reserved Memory: 0 B / 33.6 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.737), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:10Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.220:12376
   └ ID: C26O:JYTW:BSGK:HZT6:2NFI:5P7C:MXJI:VHPG:PY2B:XYOY:JQS3:UMKY|10.1.66.220:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:52Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.221:12376
   └ ID: NDTX:762Z:GMUK:EHOY:BSYX:5WVE:MI5Q:LU4S:XCSW:4P2P:OPFK:LT4F|10.1.66.221:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:08Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.222:12376
   └ ID: GVZ3:WFFM:V7YR:L5VS:CLXH:E4TM:R47J:LGRD:UTWB:XSQT:G5NJ:BBZW|10.1.66.222:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:57Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.2.66.109:12376
   └ ID: FVM4:SVVX:EGTB:2VVG:TCYE:D5Z2:2RBY:RUBH:M3RT:RSOW:ZDHU:QICE|10.2.66.109:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:48Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.2.66.115:12376
   └ ID: NQYA:KGYU:JTCB:727A:4ZBA:XDEY:H7NX:V3U5:EECI:T5KD:HPQL:4HY5|10.2.66.115:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:35Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.30:12376
   └ ID: 6ZPK:PMKY:7ERT:AHHM:3BZT:W24I:KOW5:VI2C:N22X:7SUA:AHL5:6IQ7|10.5.105.30:12376
   └ Status: Healthy
   └ Containers: 31 (25 Running, 0 Paused, 6 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.31:12376
   └ ID: 6ZPK:PMKY:7ERT:AHHM:3BZT:W24I:KOW5:VI2C:N22X:7SUA:AHL5:6IQ7|10.5.105.31:12376
   └ Status: Healthy
   └ Containers: 44 (28 Running, 0 Paused, 16 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay
   └ UpdatedAt: 2019-09-26T20:43:20Z
   └ ServerVersion: 19.03.2
 REDACTED: 10.5.105.32:12376
   └ ID: 6ZPK:PMKY:7ERT:AHHM:3BZT:W24I:KOW5:VI2C:N22X:7SUA:AHL5:6IQ7|10.5.105.32:12376
   └ Status: Healthy
   └ Containers: 24 (16 Running, 0 Paused, 8 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.108:12376
   └ ID: 6S26:RXWW:FRZJ:QK5S:EGSQ:XLDD:MMN6:B6NX:GBLB:BFUX:FCNX:RWKH|10.1.66.108:12376
   └ Status: Healthy
   └ Containers: 42 (7 Running, 0 Paused, 35 Stopped)
   └ Reserved CPUs: 8 / 4
   └ Reserved Memory: 0 B / 8.02 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:42:57Z
   └ ServerVersion: 18.09.2
 Cluster Managers: 3
  REDACTED: Healthy
   └ Orca Controller: https://10.5.105.31:443
   └ Classic Swarm Manager: tcp://10.5.105.31:2376
   └ Engine Swarm Manager: tcp://10.5.105.31:12376
   └ KV: etcd://10.5.105.31:12379
 REDACTED: Healthy
   └ Orca Controller: https://10.5.105.30:443
   └ Classic Swarm Manager: tcp://10.5.105.30:2376
   └ Engine Swarm Manager: tcp://10.5.105.30:12376
   └ KV: etcd://10.5.105.30:12379
  REDACTED: Healthy
   └ Orca Controller: https://10.5.105.65:443
   └ Classic Swarm Manager: tcp://10.5.105.65:2376
   └ Engine Swarm Manager: tcp://10.5.105.65:12376
   └ KV: etcd://10.5.105.65:12379
 Plugins:
  Volume:
  Network:
  Log:
 Swarm: active
  NodeID: freo6wpjggr90494o7t98qd6p
  Is Manager: true
  ClusterID: nivf50h6ij2o3cw251uybflse
  Managers: 3
  Nodes: 18
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 3
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
   External CAs:
     cfssl: https://10.5.105.30:12381/api/v1/cfssl/sign
     cfssl: https://10.5.105.65:12381/api/v1/cfssl/sign
     cfssl: https://10.5.105.31:12381/api/v1/cfssl/sign
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.5.105.30
  Manager Addresses:
   10.5.105.30:2377
   10.5.105.31:2377
   10.5.105.65:2377
 Kernel Version: 3.10.0-957.27.2.el7.x86_64
 Operating System: linux
 Architecture: amd64
 CPUs: 79
 Total Memory: 290.2GiB
 Name: ucp-controller-10.5.105.30
 ID: nivf50h6ij2o3cw251uybflse
 Docker Root Dir:
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  com.docker.ucp.license_key=REDACTED
  com.docker.ucp.license_max_engines=20
  com.docker.ucp.license_expires=REDACTED
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Quantity: 20 Nodes    Expiration date: REDACTED   License is currently active

[jorisscheppers]

@dperny do you know what is going on here?

[esrahofstede]

Any updates on this?

[dperny]

Sorry, this issue must have gotten buried in my inbox and I missed it. Do y'all know if it works correctly through the command line, or is it broken in both the CLI and in compose?

[jimbo8098]

Hey dudes, I know this is really old but I believe the error is alluding to the format of the location you are using for your credspec. It should be file://<filename> where <filename> is within the C:\ProgramData\Docker\credentialspecs directory.

Some more info can be found here:

https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/gmsa-orchestrate-containers

I suppose it's possible that with compose files this may be different but when I delegate a service I'm able to use that syntax in docker CLI.