Build and proxies

[rumstead]

Are there any docs on how to use proxies with rancher desktop and the underlying components (lima and nerdctl)? I have seen a couple of issues (#384, #699) around proxies/env variables but I can't find any docs or a clear path forward for setting them and being able to build. I am happy to contribute a README or something after this is sorted.

For bugs, describe what you're seeing
I was able to get rancher desktop to launch by setting the http/https env variables, but get issues building images that need to come through a proxy.

During a build, my internal image pulls fine but the external gcr.io one times out. I did try adding build-args with the envs but didn't work (described here moby/buildkit#971)

❯ nerdctl build .                                                                                                                                             
[+] Building 30.1s (4/4) FINISHED                                                                                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                                                                                               0.0s
 => => transferring dockerfile: 32B                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                  0.0s
 => => transferring context: 35B                                                                                                                                                                                                   0.0s
 => ERROR [internal] load metadata for gcr.io/distroless/static:nonroot                                                                                                                                                           30.0s
 => [internal] load metadata for internal-reg/golang:1.17.1                                                                                                                                           2.8s
------
 > [internal] load metadata for gcr.io/distroless/static:nonroot:
------
error: failed to solve: failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head https://gcr.io/v2/distroless/static/manifests/nonroot: dial tcp 74.125.142.82:443: i/o timeout
FATA[0030] unrecognized image format

But nerdctl pull works "outside" the VM

❯ nerdctl pull gcr.io/distroless/static:nonroot                                                                                                            
gcr.io/distroless/static:nonroot:                                                 resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:bca3c203cdb36f5914ab8568e4c25165643ea9b711b41a8a58b42c80a51ed609:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:213a6d5205aa1421bd128b0396232a22fbb4eec4cbe510118f665398248f6d9a: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:bff4de2cb7e1dd0ed9797c6e33688f32f2ff0293ecee6fa069051081710bb61b:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e8614d09b7bebabd9d8a450f44e88a8807c98a438a2ddd63146865286b132d1b:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 5.0 s                                                                    total:  787.5  (157.4 KiB/s)    

nerdctl pull also works "inside" the VM

❯ LIMA_HOME="$HOME/Library/Application Support/rancher-desktop/lima" ./limactl shell 0
lima-rancher-desktop:~$ nerdctl pull gcr.io/distroless/static:nonroot
gcr.io/distroless/static:nonroot:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:bca3c203cdb36f5914ab8568e4c25165643ea9b711b41a8a58b42c80a51ed609:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:213a6d5205aa1421bd128b0396232a22fbb4eec4cbe510118f665398248f6d9a: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:bff4de2cb7e1dd0ed9797c6e33688f32f2ff0293ecee6fa069051081710bb61b:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e8614d09b7bebabd9d8a450f44e88a8807c98a438a2ddd63146865286b132d1b:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.1 s                                                                    total:  787.5  (154.4 KiB/s)

lima vm has my proxies

❯ LIMA_HOME="$HOME/Library/Application Support/rancher-desktop/lima" ./limactl shell 0
lima-rancher-desktop:~$ env | grep -i proxy | wc -l
8
lima-rancher-desktop:~$ env | grep -i proxy
HTTPS_PROXY=http://my-proxy
no_proxy=my-noproxy
NO_PROXY=my-noproxy
https_proxy=http://my-proxy
http_proxy=http://my-proxy
ftp_proxy=ftp://my-proxy
FTP_PROXY=ftp://my-proxy
HTTP_PROXY=http://my-proxy

To Reproduce
Steps to reproduce the behaviour:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'

Result
Try to build an image referencing a public image behind a proxy.

Expected behaviour
Public image can be pulled and used as a base image during an image build.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS
  • macOS Big Sur
• Rancher Desktop version
  • 0.6.1
• Kubernetes version
  • v1.21.5

Additional context

[rumstead]

Happy to contribute time, docs, or code just point me at where to look :).

[rumstead]

I saw the VPN tag, maybe that should be added here as well?

[jandubois]

I saw the VPN tag, maybe that should be added here as well?

Isn't this issue about proxies, which is separate from VPN?

I would agree though that we probably should have a bunch more labels, like area/dns, area/proxy, area/vpn, etc.

[rumstead]

Yea it's about proxies but I only use proxies because I am on my company's VPN haha. If it doesn't fit that label, no worries. Just thought I would highlight it.

[yassan]

@rumstead
I think the following env is unnecessary.

no_proxy=my-noproxy
NO_PROXY=my-noproxy

Perhaps the following command will work.

nerdctl build \
  --build-arg HTTP_PROXY=http://my-proxy \
  --build-arg HTTPS_PROXY=http://my-proxy \
  --build-arg NO_PROXY="localhost,127.0.0.1" \
  .

[rumstead]

Why would adding a lower case no proxy env impact pulling images?

[yassan]

Sorry, Not related to this, but "my-private-registry.example.com" was not needed for "--build-arg NO_PROXY=~".
(So I am deleting it from my comment above.)

The reason I added --build-arg NO_PROXY=~ is because your NO_PROXY / no_proxy" setting is NO_PROXY=my-noproxy`, which is not correct, and I wanted to avoid that.

[nauni77]

Can I tell nerdctl only at buildtime if it should use a proxy or not? I didn't find a flag to configure this at every call - so I can use a alias to do the job ...

HTTP-PROXY configuration?
I sometimes need the proxy and the next time I don't need the proxy. But I think the Proxy should be configured at the VirtualMachine - or does nerdctl forward the env variables at every call?

CONTAINER-PROXY configuration?
If I need the HTTP-Proxy configuration - in my case I can also use our Container proxy. But even I can't find a solution to tell nerdctl/ rancher desktop to replace "registry-1.docker.io" with our Container-Proxy-Server like "container.mycompany.com".

If someone have an idea it would be great! Thanks.

[rumstead]

#1267
#2033
#1264

Linking similar issues

[rumstead]

Same issues with docker not just nerdctl.

[rumstead]

I was able to get around my issue by setting proxies on the docker init.d and restarting the docker service.

lima-rancher-desktop:/$ cat /etc/init.d/docker
#!/sbin/openrc-run
supervisor=supervise-daemon

name="Docker Daemon"
description="Persistent process that manages docker containers"
description_reload="Reload configuration without exiting"

command="${DOCKERD_BINARY:-/usr/bin/dockerd}"
command_args="${DOCKER_OPTS}"
DOCKER_LOGFILE="${DOCKER_LOGFILE:-/var/log/${RC_SVCNAME}.log}"
DOCKER_ERRFILE="${DOCKER_ERRFILE:-${DOCKER_LOGFILE}}"
DOCKER_OUTFILE="${DOCKER_OUTFILE:-${DOCKER_LOGFILE}}"
supervise_daemon_args="--stderr \"${DOCKER_ERRFILE}\" --stdout \"${DOCKER_OUTFILE}\""

export HTTPS_PROXY=http://my-proxy
export no_proxy=my-noproxy
export NO_PROXY=my-noproxy
export https_proxy=http://my-proxy
export http_proxy=http://my-proxy
export ftp_proxy=ftp://my-proxy
export FTP_PROXY=ftp://my-proxy
export HTTP_PROXY=http://my-proxy

extra_started_commands="reload"

rc_ulimit="${DOCKER_ULIMIT:--c unlimited -n 1048576 -u unlimited}"

retry="${DOCKER_RETRY:-TERM/60/KILL/10}"

depend() {
	need sysfs cgroups
	after iptables ip6tables
}

start_pre() {
	checkpath -f -m 0644 -o root:docker "$DOCKER_ERRFILE" "$DOCKER_OUTFILE"
}

reload() {
	ebegin "Reloading configuration"
	$supervisor $RC_SVCNAME --signal HUP
	eend $?
}

@jandubois - I see this persisted even when I close rancher desktop or do a lima shutdown 0. Are these files persisted on the underlying host? Any docs I can read?

[rumstead]

A more elegant but less-than-ideal solution...

Using overrides to toggle between using a proxy and not.

Proxies:

provision:
- mode: system
  script: |
    cat <<EOF > /tmp/proxy.sh
    #!/bin/sh
    export http_proxy="http://foo.com"
    export https_proxy="http://foo.com"
    export no_proxy=kubernetes.docker.internal,127.0.0.1,127.0.0.0/8
    export ftp_proxy="ftp://foo.com/"
    export all_proxy="http://foo.com"
    export HTTP_PROXY="http://foo.com"
    export HTTPS_PROXY="http://foo.com"
    export NO_PROXY=kubernetes.docker.internal,127.0.0.1,127.0.0.0/8
    export FTP_PROXY="ftp://foo.com/"
    export ALL_PROXY="http://foo.com"
    EOF

    echo ". /tmp/proxy.sh" | sed -i -e '7r /dev/stdin' /etc/init.d/docker
    echo ". /tmp/proxy.sh" | sed -i -e '9r /dev/stdin' /etc/init.d/containerd

No Proxies

provision:
- mode: system
  script: |
    sed -i 's/. \/tmp\/proxy.sh//' /etc/init.d/docker
    sed -i 's/. \/tmp\/proxy.sh//' /etc/init.d/containerd

You have to restart rancher desktop or restart the Lima VM.

A similar approach works with windows as well except the files need to end in .start and be placed under %APPDATA%\rancher-desktop\provisioning

EDIT: Added containerd

[rumstead]

How to pass windows env to WSL2 to be used by nerdctl.

PS > $env:WSLENV="HTTP_PROXY:HTTPS_PROXY:NO_PROXY"
PS > rdctl shell env
HTTP_PROXY="http://foo.com"
HTTPS_PROXY="http://foo.com"
NO_PROXY=kubernetes.docker.internal,127.0.0.1,127.0.0.0/8