Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Credentialspec from Swarm config not working #2109

Open
jorisscheppers opened this issue Sep 26, 2019 · 4 comments
Open

Credentialspec from Swarm config not working #2109

jorisscheppers opened this issue Sep 26, 2019 · 4 comments
Labels
platform/windows

Comments

@jorisscheppers
Copy link

Description
In Docker 19.03 support was added to provide Swarm services with credentialspec json files loaded from configs instead of binding the file from the local filesystem (#1781). This does not seem to work in my case. I suspect a bug but it may be that I missed something.

Steps to reproduce the issue:
I created a config on the manager via UCP client bundle on client machine:

docker config create gmsa_ontcaching gmsa_ontcaching.json

gmsa_ontcaching.json file is located on the client machine, config is created successfully:

PS D:\> docker config ls
ID                          NAME                                                      CREATED             UPDATED
xs7b2ueepch3l7wpewnswdvf8   gmsa_ontcaching                                 32 hours ago        32 hours ago

Service definition in docker-compose.yml:

version: '3.8'
configs:
  gmsa_ontcaching:
    external:
      name: gmsa_ontcaching
networks:
  fzbo-network:
    driver: overlay
services:
  fzbo-caching:
    credential_spec:
      config: gmsa_ontcaching
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s
    hostname: gmsa_ontcaching
    image: repo-name/caching:tag-162
    networks:
      fzbo-network:
        aliases:
        - orleans

Describe the results you received:

PS D:\sources\compose> docker stack deploy -c .\docker-compose.yml cachetest
time="2019-09-25T13:54:21+02:00" level=warning msg="config gmsa_ontcaching: config.external.name is deprecated in favor of config.name"
Creating network cachetest_fzbo-network
Creating service cachetest_fzbo-caching
failed to create service cachetest_fzbo-caching: Error response from daemon: invalid CredentialSpec: must either provide "file", "registry", or "config" for credential spec

Describe the results you expected:
No errors and a service where the gMSA credentialspec was added from a Swarm config.

Additional information you deem important (e.g. issue happens only occasionally):
There are other methods of defining the config for the credentialspec, according to documentation (https://docs.docker.com/compose/compose-file/#example-gmsa-configuration):

version: "3.8"
services:
  myservice:
    image: myimage:latest
    credential_spec:
      config: my_credential_spec

configs:
  my_credentials_spec:
    file: ./my-credential-spec.json|

First of all, what is the pipe-symbol doing in the file definition?
Second: this specifies a service-specific config from a local file which is not my goal. But as a test I tried it this way as well, and the response was the same:

docker-compose.yml:

version: '3.8'
configs:
  gmsa_ontcaching:
    file: gmsa_ontcaching.json
networks:
  fzbo-network:
    driver: overlay
services:
  fzbo-caching:
    credential_spec:
      config: gmsa_ontcaching
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 30s
    hostname: gmsa_ontcaching
    image: repo-name/caching:tag-162
    networks:
      fzbo-network:
        aliases:
        - orleans

Output:

PS D:\sources\cz-docker\ForCZ\compose\ont> docker stack deploy -c .\docker-compose.caching.yml cachetest
Creating config cachetest_gmsa_ontcaching
Creating service cachetest_fzbo-caching
failed to create service cachetest_fzbo-caching: Error response from daemon: invalid CredentialSpec: must either provide "file", "registry", or "config" for credential spec

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.39 (downgraded from 1.40)
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:17:08 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Enterprise 2.1
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       c92ab06
  Built:            Tue Sep  3 15:55:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
 Universal Control Plane:
  Version:          3.1.4
  ApiVersion:       1.39
  Arch:             amd64
  BuildTime:        Wed Feb 27 22:26:43 UTC 2019
  GitCommit:        29b16f9
  GoVersion:        go1.10.6
  MinApiVersion:    1.20
  Os:               linux
 Kubernetes:
  Version:          1.11+
  buildDate:        2019-02-05T21:33:23Z
  compiler:         gc
  gitCommit:        8249259df545d6e853b7de9d050a34ec24e8b2f5
  gitTreeState:     clean
  gitVersion:       v1.11.7-docker-1
  goVersion:        go1.10.7
  major:            1
  minor:            11+
  platform:         linux/amd64
 Calico:
  Version:          v3.5.0
  cni:              v3.5.0
  kube-controllers: v3.5.0
  node:             v3.5.0

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 421
  Running: 240
  Paused: 0
  Stopped: 181
 Images: 599
 Server Version: ucp/3.1.4
 Role: primary
 Strategy: spread
 Filters: health, port, containerslots, dependency, affinity, constraint, whitelist
 Nodes: 18
  REDACTED: 10.5.105.65:12376
   └ ID: D3VP:6S42:674O:VPKF:X7LS:2TNG:DSNZ:I6U5:FKZT:5F34:2D73:3XKA|10.5.105.65:12376
   └ Status: Healthy
   └ Containers: 39 (27 Running, 0 Paused, 12 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 340 MiB / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.66:12376
   └ ID: USZW:IHMU:JJRB:U6PY:3DOH:IAHE:E5CE:ECRZ:XV4U:LTON:XGD6:HZ4B|10.5.105.66:12376
   └ Status: Healthy
   └ Containers: 24 (21 Running, 0 Paused, 3 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:20Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.67:12376
   └ ID: RPWP:MUXF:CKW5:FXRQ:J2AB:AKP5:QSSP:ZWWK:YGEA:UZFJ:KFYG:CSTJ|10.5.105.67:12376
   └ Status: Healthy
   └ Containers: 16 (16 Running, 0 Paused, 0 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.209:12376
   └ ID: 6S26:RXWW:FRZJ:QK5S:EGSQ:XLDD:MMN6:B6NX:GBLB:BFUX:FCNX:RWKH|10.1.66.209:12376
   └ Status: Healthy
   └ Containers: 7 (4 Running, 0 Paused, 3 Stopped)
   └ Reserved CPUs: 0 / 4
   └ Reserved Memory: 0 B / 8.02 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:43:17Z
   └ ServerVersion: 18.09.2
  REDACTED: 10.2.66.47:12376
   └ ID: X3CR:SVG2:GOBW:MAVC:ZRSL:E6AR:6INB:TNSM:P5W6:GGY4:W2K4:EEYT|10.2.66.47:12376
   └ Status: Healthy
   └ Containers: 28 (28 Running, 0 Paused, 0 Stopped)
   └ Reserved CPUs: 0 / 25
   └ Reserved Memory: 0 B / 67.2 GiB
   └ Labels: kernelversion=10.0 14393 (14393.3179.amd64fre.rs1_release_1.190806-2245), operatingsystem=Windows Server 2016 Standard Version 1607 (OS Build 14393.3181), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:20Z
   └ ServerVersion: 18.09.3
  REDACTED: 10.1.66.35:12376
   └ ID: CGUM:LTD2:7ACF:CI6Y:MISJ:LN7F:J6R7:4UUI:4WI5:WXG2:CP75:RNRS|10.1.66.35:12376
   └ Status: Healthy
   └ Containers: 105 (28 Running, 0 Paused, 77 Stopped)
   └ Reserved CPUs: 0 / 12
   └ Reserved Memory: 0 B / 16.8 GiB
   └ Labels: kernelversion=10.0 14393 (14393.3179.amd64fre.rs1_release_1.190806-2245), operatingsystem=Windows Server 2016 Standard Version 1607 (OS Build 14393.3181), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 18.09.3
  REDACTED: 10.1.66.215:12376
   └ ID: NOTJ:526Z:TRKL:W5F5:GPOA:EKYJ:M6QE:KYL5:D3ZZ:PQQD:4HL2:B5NB|10.1.66.215:12376
   └ Status: Healthy
   └ Containers: 3 (2 Running, 0 Paused, 1 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-25T07:49:40Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.216:12376
   └ ID: 65GA:LZSK:KKGJ:2AM5:HMHR:YZ7L:QRXL:JJY7:K266:RNG4:AGLL:LHZY|10.1.66.216:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:00Z
   └ ServerVersion: 19.03.2
 REDACTED: 10.1.66.217:12376
   └ ID: X6TE:BHLD:QMPO:ISZ2:FJV5:QTHO:RJGJ:LMGM:6CYK:YAZC:ZF42:CWXC|10.1.66.217:12376
   └ Status: Healthy
   └ Containers: 34 (26 Running, 0 Paused, 8 Stopped)
   └ Reserved CPUs: 0 / 8
   └ Reserved Memory: 0 B / 33.6 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.737), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:10Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.220:12376
   └ ID: C26O:JYTW:BSGK:HZT6:2NFI:5P7C:MXJI:VHPG:PY2B:XYOY:JQS3:UMKY|10.1.66.220:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:52Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.221:12376
   └ ID: NDTX:762Z:GMUK:EHOY:BSYX:5WVE:MI5Q:LU4S:XCSW:4P2P:OPFK:LT4F|10.1.66.221:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:43:08Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.222:12376
   └ ID: GVZ3:WFFM:V7YR:L5VS:CLXH:E4TM:R47J:LGRD:UTWB:XSQT:G5NJ:BBZW|10.1.66.222:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:57Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.2.66.109:12376
   └ ID: FVM4:SVVX:EGTB:2VVG:TCYE:D5Z2:2RBY:RUBH:M3RT:RSOW:ZDHU:QICE|10.2.66.109:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:48Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.2.66.115:12376
   └ ID: NQYA:KGYU:JTCB:727A:4ZBA:XDEY:H7NX:V3U5:EECI:T5KD:HPQL:4HY5|10.2.66.115:12376
   └ Status: Healthy
   └ Containers: 4 (2 Running, 0 Paused, 2 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 8.399 GiB
   └ Labels: kernelversion=10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434), operatingsystem=Windows Server 2019 Standard Version 1809 (OS Build 17763.379), ostype=windows, storagedriver=windowsfilter
   └ UpdatedAt: 2019-09-26T20:42:35Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.30:12376
   └ ID: 6ZPK:PMKY:7ERT:AHHM:3BZT:W24I:KOW5:VI2C:N22X:7SUA:AHL5:6IQ7|10.5.105.30:12376
   └ Status: Healthy
   └ Containers: 31 (25 Running, 0 Paused, 6 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.5.105.31:12376
   └ ID: 6ZPK:PMKY:7ERT:AHHM:3BZT:W24I:KOW5:VI2C:N22X:7SUA:AHL5:6IQ7|10.5.105.31:12376
   └ Status: Healthy
   └ Containers: 44 (28 Running, 0 Paused, 16 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay
   └ UpdatedAt: 2019-09-26T20:43:20Z
   └ ServerVersion: 19.03.2
 REDACTED: 10.5.105.32:12376
   └ ID: 6ZPK:PMKY:7ERT:AHHM:3BZT:W24I:KOW5:VI2C:N22X:7SUA:AHL5:6IQ7|10.5.105.32:12376
   └ Status: Healthy
   └ Containers: 24 (16 Running, 0 Paused, 8 Stopped)
   └ Reserved CPUs: 0 / 2
   └ Reserved Memory: 0 B / 16.29 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay
   └ UpdatedAt: 2019-09-26T20:43:21Z
   └ ServerVersion: 19.03.2
  REDACTED: 10.1.66.108:12376
   └ ID: 6S26:RXWW:FRZJ:QK5S:EGSQ:XLDD:MMN6:B6NX:GBLB:BFUX:FCNX:RWKH|10.1.66.108:12376
   └ Status: Healthy
   └ Containers: 42 (7 Running, 0 Paused, 35 Stopped)
   └ Reserved CPUs: 8 / 4
   └ Reserved Memory: 0 B / 8.02 GiB
   └ Labels: com.docker.security.seccomp=enabled, kernelversion=3.10.0-957.27.2.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), ostype=linux, storagedriver=overlay2
   └ UpdatedAt: 2019-09-26T20:42:57Z
   └ ServerVersion: 18.09.2
 Cluster Managers: 3
  REDACTED: Healthy
   └ Orca Controller: https://10.5.105.31:443
   └ Classic Swarm Manager: tcp://10.5.105.31:2376
   └ Engine Swarm Manager: tcp://10.5.105.31:12376
   └ KV: etcd://10.5.105.31:12379
 REDACTED: Healthy
   └ Orca Controller: https://10.5.105.30:443
   └ Classic Swarm Manager: tcp://10.5.105.30:2376
   └ Engine Swarm Manager: tcp://10.5.105.30:12376
   └ KV: etcd://10.5.105.30:12379
  REDACTED: Healthy
   └ Orca Controller: https://10.5.105.65:443
   └ Classic Swarm Manager: tcp://10.5.105.65:2376
   └ Engine Swarm Manager: tcp://10.5.105.65:12376
   └ KV: etcd://10.5.105.65:12379
 Plugins:
  Volume:
  Network:
  Log:
 Swarm: active
  NodeID: freo6wpjggr90494o7t98qd6p
  Is Manager: true
  ClusterID: nivf50h6ij2o3cw251uybflse
  Managers: 3
  Nodes: 18
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 3
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
   External CAs:
     cfssl: https://10.5.105.30:12381/api/v1/cfssl/sign
     cfssl: https://10.5.105.65:12381/api/v1/cfssl/sign
     cfssl: https://10.5.105.31:12381/api/v1/cfssl/sign
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.5.105.30
  Manager Addresses:
   10.5.105.30:2377
   10.5.105.31:2377
   10.5.105.65:2377
 Kernel Version: 3.10.0-957.27.2.el7.x86_64
 Operating System: linux
 Architecture: amd64
 CPUs: 79
 Total Memory: 290.2GiB
 Name: ucp-controller-10.5.105.30
 ID: nivf50h6ij2o3cw251uybflse
 Docker Root Dir:
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  com.docker.ucp.license_key=REDACTED
  com.docker.ucp.license_max_engines=20
  com.docker.ucp.license_expires=REDACTED
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Quantity: 20 Nodes    Expiration date: REDACTED   License is currently active
@jorisscheppers
Copy link
Author

@dperny do you know what is going on here?

@esrahofstede
Copy link

Any updates on this?

@dperny
Copy link
Contributor

dperny commented Jan 27, 2020

Sorry, this issue must have gotten buried in my inbox and I missed it. Do y'all know if it works correctly through the command line, or is it broken in both the CLI and in compose?

@jimbo8098
Copy link

jimbo8098 commented Sep 28, 2020 •
edited
Loading

Hey dudes, I know this is really old but I believe the error is alluding to the format of the location you are using for your credspec. It should be file://<filename> where <filename> is within the C:\ProgramData\Docker\credentialspecs directory.

Some more info can be found here:

https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/gmsa-orchestrate-containers

I suppose it's possible that with compose files this may be different but when I delegate a service I'm able to use that syntax in docker CLI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
platform/windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dperny @esrahofstede @GordonTheTurtle @jimbo8098 @jorisscheppers