Dragonfly with harbor behind corporate proxy

[guillaumelfv]

Question

Hello, i did follow the docs about dragonfly with Harbor Using dragonfly with harbor. Everything work correctly when i pull images from my private Harbor registry.

But i am unable to pull images from the public docker registry anymore. I got the following error:

Error response from daemon: Get https://registry-1.docker.io/v2/: Service Unavailable

and dfdaemon debugging just throw:

2020-04-28 15:32:44.103 DEBU sign:9625 : Tunneling https request for registry-1.docker.io:443

This is the dfdaemon configuration on one client node:

registry_mirror:
   remote: https://index.docker.io
   insecure: false
   certs: []

proxies:
   # proxy all http image layer download requests with dfget
   - regx: blobs/sha256.*

hijack_https:
   cert: /root/dfclient/df.crt
   key: /root/dfclient/df.key
   hosts:
      - regx: redacted.company.com
        certs: [/root/dfclient/company.com]

With that i did modify the docker service accordingly to the doc and this is where the issue is i think. I did modify the /etc/systemd/system/docker.service.d/https-proxy.conf and /etc/systemd/system/docker.service.d/https-proxy.conf to use dfdaemon as proxy:

[Service]
Environment="HTTPS_PROXY=http://127.0.0.1:65001"

But before that it was set to my corporate proxy. If i reset it to my corporate proxy value then i can pull from my private registry and from the public docker registry, but it does not use dragonfly anymore.

It seems that:

• pulling from Harbor private registry is correctly hijack and pull is working (its internal so does not need corporate proxy)
• pulling from public docker registry get proxy by regx: blobs/sha256.* and use dfget but it does not goes to my corporate proxy (or use system proxy) hence the service unavailable

Do I understand it correctly ?

Ultimately Is there a way of configuring dragonfly to:

• pull images from harbor private registry through dfget (no proxy)
• pull images from docker public registry through dfget (using corporate proxy)

If you need other info or if its not clear let me know !

[lowzj]

Could you paste related logs in $HOME/.small-dragonfly/logs/dfclient.log and $HOME/.small-dragonfly/logs/dfdaemon.log?

[guillaumelfv]

Appreciate the quick answer ! Unfortunately the only logs i got when doing docker pull nginx are the following:

[root@host centos]# docker pull nginx
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: Service Unavailable

and from dfdaemon.log :
2020-04-28 17:22:47.342 DEBU sign:3255 : Tunneling https request for registry-1.docker.io:443

No logs from dfclient when i pull the nginx image...

[lowzj]

It seems that the network is not reachable from your dfclient container to registry-1.docker.io.
Could you test it on the dfclient container by using the command:

curl -iv 'https://registry-1.docker.io/v2/'

or you can type this command to get the error response:

# note: http, not https
http_proxy=http://127.0.0.1:65001 curl -i 'http://registry-1.docker.io/v2/'

[guillaumelfv]

I am not using container i did deploy on a server with the go binary, so i ran the command from the server itself to test (proxy url and IP redacted):

• curl -iv 'https://registry-1.docker.io/v2/'

[root@host logs]# curl -iv 'https://registry-1.docker.io/v2/'
* About to connect() to proxy proxy.company.com port 8080 (#0)
*   Trying X.X.X.X...
* Connected to proxy.company.com (X.X.X.X) port 8080 (#0)
* Establish HTTP proxy tunnel to registry-1.docker.io:443
> CONNECT registry-1.docker.io:443 HTTP/1.1
> Host: registry-1.docker.io:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
HTTP/1.1 200 Connection established
<

* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=*.docker.io
* 	start date: Jun 07 00:00:00 2019 GMT
* 	expire date: Jul 07 12:00:00 2020 GMT
* 	common name: *.docker.io
* 	issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
> GET /v2/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: registry-1.docker.io
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
HTTP/1.1 401 Unauthorized
< Content-Type: application/json
Content-Type: application/json
< Docker-Distribution-Api-Version: registry/2.0
Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
< Date: Tue, 28 Apr 2020 11:26:58 GMT
Date: Tue, 28 Apr 2020 11:26:58 GMT
< Content-Length: 87
Content-Length: 87
< Strict-Transport-Security: max-age=31536000
Strict-Transport-Security: max-age=31536000

<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host proxy.company.com left intact

• http_proxy=http://127.0.0.1:65001 curl -i 'http://registry-1.docker.io/v2/'

[root@host logs]# http_proxy=http://127.0.0.1:65001 curl -i 'http://registry-1.docker.io/v2/'
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 28 Apr 2020 11:44:34 GMT
Content-Length: 60

dial tcp 54.152.200.115:80: connect: network is unreachable

Which make sense because the first one use the corporate proxy but the second one is not apparently

[lowzj]

I see, your host cannot connect to registry without your corporate proxy, but dfdaemon of Dragonfly doesn't support using http(s)_proxy to send request, so dockerd cannot connect to registry via dfdaemon(127.0.0.1:65001).

Maybe dfdaemon should provide a flag to enable proxy, then it can send request via user's proxy.

WDYT? @starnop @inoc603

[inoc603]

I see, your host cannot connect to registry without your corporate proxy, but dfdaemon of Dragonfly doesn't support using http(s)_proxy to send request, so dockerd cannot connect to registry via dfdaemon(127.0.0.1:65001).

Maybe dfdaemon should provide a flag to enable proxy, then it can send request via user's proxy.

WDYT? @starnop @inoc603

I think we can first try supporting environment variables like http_proxy, https_proxy and no_proxy. This avoids adding more configuration to the config file.

[guillaumelfv]

@inoc603 Would be awesome !

[zanglang]

Seems to me that the issue is DFRoundTripper does not call ProxyFromEnvironment.

A different way to fix this may be to let, NeedUseGetter() work on all URLs besides "blobs/sha256/.*" URLs so that the "GET https://index.docker.io/v2/" HTTP request will be called through dfget, which will be handled by the CDN.