"docker manifest inspect --insecure" doesn't work #1358
Comments
|
Taking a look |
|
@ijc -- What registry are you running? If it's one you created using Also, unfortunately, you can't override that certs dir location AFAIK. See @dnephin's comment at #138 (comment). (Just FYI) If anyone knows otherwise, lmk. |
|
@clnperez it's a container I've had for a while, my notes claim I created it with: I've not (knowingly) reconfigured anything so I think this is the default: But I don't think it is getting so far as to be even opening a socket, it hits a client side error before it gets that far. Let me know if you need anything else. |
|
@ijc thanks. just one more thing -- the hash of the registry:2 image you've got running. |
|
|
@ijc, looks like that one isn't in the registry any more can you give me output from |
|
@clnperez I just wanted to reiterate something I've said a couple of times now: The failure is occurring before the CLI has made any network calls at all, so I don't think the version of the registry it is talking to can be in any way relevant (infact it doesn't even need to exist or be running). $ grep -E 'socket|accept|listen|bind|172.17.0.4|AF_INET|gethostbyaddr' /tmp/docker-manifest-inspect.strace.*
/tmp/docker-manifest-inspect.strace.17582:execve("/usr/bin/docker", ["docker", "-D", "manifest", "inspect", "--insecure", "172.17.0.4:5000/foo:latest"], 0x7ffd843df480 /* 55 vars */) = 0
/tmp/docker-manifest-inspect.strace.17582:socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
/tmp/docker-manifest-inspect.strace.17582:connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
/tmp/docker-manifest-inspect.strace.17582:socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
/tmp/docker-manifest-inspect.strace.17582:connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
/tmp/docker-manifest-inspect.strace.17582:socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
/tmp/docker-manifest-inspect.strace.17582:socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
/tmp/docker-manifest-inspect.strace.17586:openat(AT_FDCWD, "/home/ijc/.docker/manifests/172.17.0.4-5000_foo-latest", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
/tmp/docker-manifest-inspect.strace.17586:openat(AT_FDCWD, "/etc/docker/certs.d/172.17.0.4:5000", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
/tmp/docker-manifest-inspect.strace.17586:openat(AT_FDCWD, "/etc/docker/certs.d/172.17.0.4:5000", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)You can see that there are no socket calls other than the Unix domain ones for For some reason (unknown to me) my |
|
I've reported the |
|
@ijc i read what you were saying and i don't think it needs reiterating. i'm sorry if i didn't acknowledge what you'd said. but what's still strange to me is that your list of endpoints is empty. the permissions one is probably something that the manifest command can/should work around, but what i'm seeing in the current code doesn't make sense that it got to that point. |
|
However the empty list of endpoints is arising it's nothing to do with the version or configuration of the registry on the server side, since the client never even tries to talk to the server, but you were asking questions specific to the server side of things. From So it does seem like the empty endpoints is pretty directly related to the If I correct the permissions on $ docker -D manifest inspect --insecure 172.17.0.3:5000/foo:latest
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.3:5000
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.3:5000
DEBU[0000] endpoints for 172.17.0.3:5000/foo:latest: [{false https://172.17.0.3:5000 v2 false false true 0xc4204c6c00} {false https://172.17.0.3:5000 v1 false false true 0xc4204c6d80}]
DEBU[0000] error with repo endpoint {%!s(*registry.RepositoryInfo=&{{172.17.0.3:5000 foo} 0xc4202e09c0 false }) {%!s(bool=false) %!s(*url.URL=&{https <nil> 172.17.0.3:5000 false }) %!s(registry.APIVersion=2) %!s(bool=false) %!s(bool=false) %!s(bool=true) %!s(*tls.Config=&{<nil> <nil> [] map[] <nil> <nil> <nil> <nil> <nil> [] 0 <nil> true [49196 49200 49195 49199 49162 49161 49172 49171 53 47] true false [123 26 177 2 12 180 4 49 139 85 145 57 51 219 174 34 158 52 4 193 205 74 40 116 204 145 78 129 114 212 233 189] <nil> 769 0 [] false 0 <nil> {{0 0} 1} {{0 0} 0 0 0 0} [{[227 213 248 79 41 196 201 98 205 114 18 228 223 136 13 1] [208 147 212 136 191 26 213 163 113 236 105 77 100 109 10 13] [197 138 17 48 72 221 91 95 152 147 185 166 152 158 79 144]}]})}}: failed to configure transport: error pinging v2 registry: Get https://172.17.0.3:5000/v2/: http: server gave HTTP response to HTTPS client
DEBU[0000] skipping v1 endpoint https://172.17.0.3:5000
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.3:5000
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.3:5000
DEBU[0000] endpoints for 172.17.0.3:5000/foo:latest: [{false https://172.17.0.3:5000 v2 false false true 0xc4204c7500} {false https://172.17.0.3:5000 v1 false false true 0xc4204c7680}]
DEBU[0000] error with repo endpoint {%!s(*registry.RepositoryInfo=&{{172.17.0.3:5000 foo} 0xc4202e11a0 false }) {%!s(bool=false) %!s(*url.URL=&{https <nil> 172.17.0.3:5000 false }) %!s(registry.APIVersion=2) %!s(bool=false) %!s(bool=false) %!s(bool=true) %!s(*tls.Config=&{<nil> <nil> [] map[] <nil> <nil> <nil> <nil> <nil> [] 0 <nil> true [49196 49200 49195 49199 49162 49161 49172 49171 53 47] true false [34 46 124 210 178 22 157 181 68 91 245 180 234 75 45 56 200 226 120 142 124 158 92 153 189 194 125 158 90 70 211 68] <nil> 769 0 [] false 0 <nil> {{0 0} 1} {{0 0} 0 0 0 0} [{[244 139 193 158 233 153 230 227 247 97 164 238 52 208 83 230] [70 233 228 182 253 84 246 155 55 129 117 2 192 1 249 2] [159 27 220 251 253 6 92 11 166 225 249 108 109 85 103 137]}]})}}: failed to configure transport: error pinging v2 registry: Get https://172.17.0.3:5000/v2/: http: server gave HTTP response to HTTPS client
DEBU[0000] skipping v1 endpoint https://172.17.0.3:5000
no such manifest: 172.17.0.3:5000/foo:latest(Note that between my initial report and now the registries P address changed because I restarted). If I hit a non-existent server address with the perms corrected then I see: $ docker -D manifest inspect --insecure 172.17.0.4:5000/foo:latest
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.4:5000
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.4:5000
DEBU[0000] endpoints for 172.17.0.4:5000/foo:latest: [{false https://172.17.0.4:5000 v2 false false true 0xc4200b1200} {false https://172.17.0.4:5000 v1 false false true 0xc4200b1380}]
DEBU[0000] not continuing on error (*url.Error) Get https://172.17.0.4:5000/v2/foo/manifests/latest: dial tcp 172.17.0.4:5000: connect: connection refused
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.4:5000
DEBU[0000] hostDir: /etc/docker/certs.d/172.17.0.4:5000
DEBU[0000] endpoints for 172.17.0.4:5000/foo:latest: [{false https://172.17.0.4:5000 v2 false false true 0xc42009d380} {false https://172.17.0.4:5000 v1 false false true 0xc42009d500}]
DEBU[0000] not continuing on error (*url.Error) Get https://172.17.0.4:5000/v2/foo/manifests/latest: dial tcp 172.17.0.4:5000: connect: connection refused
Get https://172.17.0.4:5000/v2/foo/manifests/latest: dial tcp 172.17.0.4:5000: connect: connection refusedBoth of those seem pretty sensible (there is no |
|
I was initially unable to recreate this, but forgetting that by default loopback (which is what I was using) is in the engine's insecure registries list. I mapped it to host's actual IP and, with the permissions set to 700, can recreate. Also, I realized what you were trying to get at after I sent that last message. I was looking for something the wrong way. So, apologies for being a little slow there. Thanks for all that detail. |
|
Opened moby/moby#37847 to address this issue |
|
@thaJeztah @vdemeester I think this one should stay open until the permissions handling on the CLI side is fixed too. |
|
Had some time to look at this today and figured out what I missed. Now I need to figure out the fix. Will try to get it done this week. |
If, for some reason, the certs directory has permissions that are inaccessible by docker, we should still be able to fetch manifests using the `insecure` flag. Since the cli doesn't access the engine's list of insecure registries, the registry client should make a singleton list of the registry being queried with the `insecure` flag. Fixes docker#1358 Signed-off-by: Christy Norman <[email protected]>
If, for some reason, the certs directory has permissions that are inaccessible by docker, we should still be able to fetch manifests using the `insecure` flag. Since the cli doesn't access the engine's list of insecure registries, the registry client should make a singleton list of the registry being queried with the `insecure` flag. Closes docker#1358 Signed-off-by: Christy Norman <[email protected]>
If, for some reason, the certs directory has permissions that are inaccessible by docker, we should still be able to fetch manifests using the `insecure` flag. Since the cli doesn't access the engine's list of insecure registries, the registry client should make a singleton list of the registry being queried with the `insecure` flag. Closes docker#1358 Signed-off-by: Christy Norman <[email protected]>
If, for some reason, the certs directory has permissions that are inaccessible by docker, we should still be able to fetch manifests using the `insecure` flag. Since the cli doesn't access the engine's list of insecure registries, the registry client should make a singleton list of the registry being queried with the `insecure` flag. Closes docker#1358 Signed-off-by: Christy Norman <[email protected]>
If, for some reason, the certs directory has permissions that are inaccessible by docker, we should still be able to fetch manifests using the `insecure` flag. Since the cli doesn't access the engine's list of insecure registries, the registry client should make a singleton list of the registry being queried with the `insecure` flag. Closes docker#1358 Signed-off-by: Christy Norman <[email protected]> (cherry picked from commit d57adbc) Signed-off-by: Sebastiaan van Stijn <[email protected]>
|
Thanks for using the manifest command @ijc, and the bug report! |
If, for some reason, the certs directory has permissions that are inaccessible by docker, we should still be able to fetch manifests using the `insecure` flag. Since the cli doesn't access the engine's list of insecure registries, the registry client should make a singleton list of the registry being queried with the `insecure` flag. Closes docker#1358 Signed-off-by: Christy Norman <[email protected]>
Description
I have a local (insecure) registry for debugging purposes. But I cannot run
manifest inspectagainst it, even with--insecure.Steps to reproduce the issue:
Just run
docker -D manifest inspect --insecure 172.17.0.4:5000/foo:latest.The registry and the image don't actually need to exist to hit the problem since it occurs before any sockets are opened etc (according to my quick look over the strace logs).
Describe the results you received:
Describe the results you expected:
The manifest to be printed.
Additional information you deem important (e.g. issue happens only occasionally):
The directory
/etc/docker/certs.ddoesn't exist, but in any case the permissions on/etc/dockerare0700, also it seems odd to rely only on a system wide directory here. I couldn't find any setting or config file option which would redirect this to e.g.~/.docker/certs.d.Output of
docker version:Also reproduced with 96dba79 (recent master).
Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
Native Linux (Debian) running docker-ce packages from
download.docker.comapt repo.The text was updated successfully, but these errors were encountered: