Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rancher Desktop issues when connected to company VPN #722

Closed
Tracked by #1115
ontec-xrail opened this issue Oct 4, 2021 · 25 comments
Closed
Tracked by #1115

Rancher Desktop issues when connected to company VPN #722

ontec-xrail opened this issue Oct 4, 2021 · 25 comments
Assignees
Labels
Milestone

Comments

@ontec-xrail
Copy link

Hi,

when I am connected to my company vpn (Checkpoint Endpoint Security) the rancher desktop 0.5.0 (Kubernetes) doesn't start up anymore. When not in VPN - everything works.

image

I also have a second wsl running (Ubuntu 20.04) with docker deamon - this wsl also sets the MTU of the eth0 to 1350 - thats needed that wsl can connect via VPN.

Attached you can find the k3s log file
k3s.log

Here are my network connectors:

On the host:

Ethernet-Adapter Ethernet 3:

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::e126:5894:846d:5657%10
   IPv4-Adresse  . . . . . . . . . . : 192.168.99.95
   Subnetzmaske  . . . . . . . . . . : 255.255.255.128
   Standardgateway . . . . . . . . . :

Ethernet-Adapter Ethernet 2:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix: home

Ethernet-Adapter vEthernet (Default Switch):

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::618a:99e8:a539:75e5%37
   IPv4-Adresse  . . . . . . . . . . : 172.20.176.1
   Subnetzmaske  . . . . . . . . . . : 255.255.240.0
   Standardgateway . . . . . . . . . :

Drahtlos-LAN-Adapter WLAN:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:

Drahtlos-LAN-Adapter LAN-Verbindung* 1:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:

Drahtlos-LAN-Adapter LAN-Verbindung* 2:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:

Ethernet-Adapter Ethernet:

   Verbindungsspezifisches DNS-Suffix: home
   IPv6-Adresse. . . . . . . . . . . : 2001:871:223:34d1:8df8:873e:4ca3:3fe
   Temporäre IPv6-Adresse. . . . . . : 2001:871:223:34d1:c0a5:7eab:dde1:85b7
   Verbindungslokale IPv6-Adresse  . : fe80::8df8:873e:4ca3:3fe%20
   IPv4-Adresse  . . . . . . . . . . : 10.0.0.12
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . : fe80::8e59:c3ff:fe43:971%20
                                       10.0.0.138

Ethernet-Adapter Bluetooth-Netzwerkverbindung:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:

Ethernet-Adapter vEthernet (WSL):

   Verbindungsspezifisches DNS-Suffix:
   Verbindungslokale IPv6-Adresse  . : fe80::ed2f:841d:d369:7a8b%68
   IPv4-Adresse  . . . . . . . . . . : 172.23.208.1
   Subnetzmaske  . . . . . . . . . . : 255.255.240.0
   Standardgateway . . . . . . . . . :

In the Rancher WSL:

/etc # ifconfig
cni0      Link encap:Ethernet  HWaddr 7A:19:8F:7B:E2:DE
          inet addr:10.42.0.1  Bcast:10.42.0.255  Mask:255.255.255.0
          inet6 addr: fe80::7819:8fff:fe7b:e2de/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:10678 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12099 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3219760 (3.0 MiB)  TX bytes:3712184 (3.5 MiB)

docker0   Link encap:Ethernet  HWaddr 02:42:C8:D6:DA:C1
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 00:15:5D:1A:55:01
          inet addr:172.23.220.104  Bcast:172.23.223.255  Mask:255.255.240.0
          inet6 addr: fe80::215:5dff:fe1a:5501/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1350  Metric:1
          RX packets:513 errors:0 dropped:0 overruns:0 frame:0
          TX packets:362 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:137255 (134.0 KiB)  TX bytes:30071 (29.3 KiB)

flannel.1 Link encap:Ethernet  HWaddr 62:E2:0A:7B:96:C0
          inet addr:10.42.0.0  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fe80::60e2:aff:fe7b:96c0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:30229 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30229 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11765245 (11.2 MiB)  TX bytes:11765245 (11.2 MiB)

veth12cbf751 Link encap:Ethernet  HWaddr 22:30:EA:3B:27:B3
          inet6 addr: fe80::2030:eaff:fe3b:27b3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:3542 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3606 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:243608 (237.8 KiB)  TX bytes:248328 (242.5 KiB)

veth144de2e1 Link encap:Ethernet  HWaddr 6A:AA:85:FB:02:EA
          inet6 addr: fe80::68aa:85ff:fefb:2ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:1187 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1351 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:328481 (320.7 KiB)  TX bytes:509824 (497.8 KiB)

veth2807426d Link encap:Ethernet  HWaddr 6A:3E:6D:F3:46:EE
          inet6 addr: fe80::683e:6dff:fef3:46ee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:4740 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5986 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:995009 (971.6 KiB)  TX bytes:1819226 (1.7 MiB)

veth2ca4cd5d Link encap:Ethernet  HWaddr AE:D9:61:43:63:D2
          inet6 addr: fe80::acd9:61ff:fe43:63d2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:1948 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2016 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1766258 (1.6 MiB)  TX bytes:1103435 (1.0 MiB)

vethe9b37eac Link encap:Ethernet  HWaddr 36:B0:54:30:89:2D
          inet6 addr: fe80::34b0:54ff:fe30:892d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:2851 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:287332 (280.5 KiB)  TX bytes:309737 (302.4 KiB)

Any clue whats going on in my instance? The logs have several connection failed logs - so I assume its something related to networking - any instructions how I could debug further?

@philippe-granet
Copy link

I think it's an issue with WSL2, see:
microsoft/WSL#5068
microsoft/WSL#416

@mook-as
Copy link
Contributor

mook-as commented Oct 4, 2021

That log looks… odd:

E1004 12:41:52.675598 453 network_policy_controller.go:252] Aborting sync. Failed to run iptables-restore: exit status 2 (iptables-restore v1.8.7 (legacy): Couldn't load match 'limit':No such file or directory

-A KUBE-POD-FW-ESB5TBJWUDYBQS27 -m comment --comment "rule to log dropped traffic POD name:svclb-traefik-tdxz5 namespace: kube-system" -m mark ! --mark 0x10000/0x10000 -j NFLOG --nflog-group 100 -m limit --limit 10/minute --limit-burst 10

Attempting to run iptables directly shows the same:

# iptables -A  KUBE-FORWARD -m limit --limit '10/minute' --limit-burst 10
iptables v1.8.3 (legacy): Couldn't load match 'limit':No such file or directory

Checking the config:

$ zgrep NETFILTER_XT_MATCH_LIMIT /proc/config.gz
# CONFIG_NETFILTER_XT_MATCH_LIMIT is not set

I think that means it's not built; I'm not very familiar with reading Kconfig, though.

@sebastian-hans-swm
Copy link

I've got the same problem, including the same error message from iptables on a fresh install of Rancher Desktop 0.6.0 on Windows 10 Enterprise 1909 in a corporate VPN. I have Debian running in WSL 2, too, and after setting the MTU correctly networking between WSL 2 and the host works. My k3s.log looks the same and k8s.log is full of error messages like this:

Error: connect ETIMEDOUT 172.17.87.147:6443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
  errno: -4039,
  code: 'ETIMEDOUT',
  syscall: 'connect',
  address: '172.17.87.147',
  port: 6443

@ontec-xrail
Copy link
Author

ontec-xrail commented Oct 29, 2021

Any updates here? I think in time like these a lot of people are in home office and connected via VPN - without a solution rancher desktop is unfortunately no option ...

Can I provide you with more details? Just tell me what you need (which files / commands to execute)

Thanks Max

@philippe-granet
Copy link

You can try this tool https://github.com/sakai135/wsl-vpnkit in the Rancher WSL distro.
It works in my Debian distro with Pulse Secure VPN client.

@road2programmer
Copy link

Hello,

Any updates on this? I am experiencing the same issue. Tried wsl-vpnkit but it never worked... am I the only one not being able to make it work with wsl-vpnkit?

Thank you

@gaktive gaktive added this to the v1.0.0 milestone Dec 15, 2021
@gaktive gaktive mentioned this issue Dec 15, 2021
2 tasks
@psaenz
Copy link

psaenz commented Jan 4, 2022

I'm having the same issue, basically without the VPN it works fine but with the VPN it gets stuck in the "Starting Kubernetes...." message.

As far as I saw, it is nothing to do with ip collisions, but probably with SSL certificates

I'm using Zscaler as VPN, but the issue is the same as the ones reported with other VPNs

Here is my scenario :

  1. Without the VPN - start Rancher Desktop
  2. on a cmd console run Kubectl get pods -- I get the expected messsage No resources found in default namespace
  3. open a wsl instance, in my case I have a Ubuntu distribution -> >wsl -d Ubuntu-20.04
  4. inside the wsl instance run Kubectl get pods -- I get the same expected messsage No resources found in default namespace
  5. start the VPN
  6. inside the wsl instance run again Kubectl get pods -- I get the same expected messsage No resources found in default namespace
  7. on the cmd console run again Kubectl get pods -- I get the messsage Unable to connect to the server: net/http: TLS handshake timeout
  8. Im able to ping the Rancher Desktop instance from cmd without problems
  9. This makes me thing it is nothing to do with ip collisions

Also, in my case, even when it gets stuck in the "Starting Kubernetes...." message, the k8s cluster is actually up and running, here is how I'm able to confirm that:

  1. WITH the VPN - start Rancher Desktop
  2. it gets stuck in the "Starting Kubernetes...." message
  3. open a wsl instance, in my case I have a Ubuntu distribution -> >wsl -d Ubuntu-20.04
  4. inside the wsl instance run Kubectl get pods -- I get the same expected messsage No resources found in default namespace --- I can run any kubectl command from there without issues but the "Starting Kubernetes...." message never goes away

@gaktive gaktive modified the milestones: v1.0.0-beta, v1.0.0 Jan 7, 2022
@gaktive gaktive modified the milestones: v1.0.0, v1.1.0 Jan 15, 2022
@erniedavisAA
Copy link

I'm having the same issue, basically without the VPN it works fine but with the VPN it gets stuck in the "Starting Kubernetes...." message.

As far as I saw, it is nothing to do with ip collisions, but probably with SSL certificates

I'm using Zscaler as VPN, but the issue is the same as the ones reported with other VPNs

Here is my scenario :

1. Without the VPN - start Rancher Desktop

2. on a cmd console run `Kubectl get pods` -- I get the expected messsage `No resources found in default namespace`

3. open a wsl instance, in my case I have a Ubuntu distribution -> `>wsl -d Ubuntu-20.04`

4. inside the wsl instance run `Kubectl get pods` -- I get the same expected messsage `No resources found in default namespace`

5. start the VPN

6. inside the wsl instance run again `Kubectl get pods` -- I get the same expected messsage `No resources found in default namespace`

7. on the cmd console run again `Kubectl get pods` -- I get the messsage `Unable to connect to the server: net/http: TLS handshake timeout`

8. Im able to ping the Rancher Desktop instance from cmd without problems

9. This makes me thing it is nothing to do with ip collisions

Also, in my case, even when it gets stuck in the "Starting Kubernetes...." message, the k8s cluster is actually up and running, here is how I'm able to confirm that:

1. **WITH** the VPN - start Rancher Desktop

2. it gets stuck in the "Starting Kubernetes...." message

3. open a wsl instance, in my case I have a Ubuntu distribution -> `>wsl -d Ubuntu-20.04`

4. inside the wsl instance run `Kubectl get pods` -- I get the same expected messsage `No resources found in default namespace` --- I can run any kubectl command from there without issues but the "Starting Kubernetes...." message never goes away

@psaenz - try the following:

changing your /.kube/config entry for Rancher Desktop to 'localhost' instead of the assigned IP.
download and install following directions for wsl-vpnkit here
Change your VPN Adapter Routing Metric by following the instructions here

@gaktive gaktive modified the milestones: Next, Later Apr 19, 2022
@jandubois jandubois added the kind/bug Something isn't working label Apr 28, 2022
@Nino-K
Copy link
Member

Nino-K commented May 4, 2022

@ontec-xrail can you please give #1899 (comment) feature a try?
It is available on our latest release 1.3.0.

Many thanks

@gaktive gaktive modified the milestones: Next, Later May 17, 2022
@herbat73
Copy link

herbat73 commented May 18, 2022

@Nino-K
tested with Cisco AnyConnect 4.4 by enabling experimentalHostResolver:true,
unfortunately it did not help, "waiting for services"
image

Here is a procedure how it works for my case (Win 10, Rachner Desktop 1.3, k8s 1.23.6 + dokerd (moby) + Cisco AnyConnect 4.4.x)

  1. install and run wsl-vpnkit
  2. Start Rancher Desktop with k8s enabled
  3. Enable WSL Integrations
    image
  4. Modify kube config
    located at %USERPROFILE%/.kube/config
    by replace IP address to localhost like
apiVersion: v1
kind: Config
clusters:
  - name: rancher-desktop
    cluster:
      server: https://localhost:6443
  1. Connect to VPN by Cisco Any Connect

and finally test connection to local k8s


The only problem witch I have is that the file .kube/config is recreated each time I start again Rachner Desktop... BOHICA ;-)

@tarrinho
Copy link

tarrinho commented Jul 8, 2022

Hello,
Any news regarding this topic?
Thanks

@jandubois jandubois modified the milestones: Next, Later Jul 19, 2022
@pumelo
Copy link

pumelo commented Aug 3, 2022

well, 1.5.0 seems to have made things worse (at least for me)

  • vpn without wsl-vpnkit still not working
  • vpn with wsl-vpnkit is broken (used to work in 1.4.1)
  • proxied company network with wsl-vpnkit is broken (used to work in 1.4.1)
  • proxied company network without wsl-vpnkit still not working

@slonghei
Copy link

slonghei commented Aug 3, 2022

1.4.1 obviously still working. If wsl-vpnkit is ultimately the answer it needs to be folded into a more permanent solution because a lot of large clients (and small as well) force users to be on a VPN, especially to work with internal docker registries etc... . The second issue is the problem of having to force users to reset the .kube config file to localhost every time before starting Rancher Desktop. [ **This is the cluster server address I am referring to in .kube/config file:
server: https://localhost::6443 ]. This reset is annoying at best but assume this is only because wsl-vpnkit needs it to be set this way in order to discover the appropriate server cluster IP Address. It would be great to automagically reset this every time before Rancher Desktop starts up. Presumably this could be done with custom scripting (startup folder for rancher desktop) to change the address back to localhost but this would probably only apply if the user is running under a VPN. Thoughts on folding this into newer versions of the product (assume now anything later than 1.4.1 now is broken)?

@SylChamber
Copy link

SylChamber commented Aug 25, 2022

wsl-vpnkit works for me in 1.5.1. My organization is using Check Point Mobile VPN 98.61.3510 and doesn't use a web proxy.

However, twice in 5 days I've been using wsl-vpnkit, Rancher Desktop hasn't been able to start up after logging in Windows, stuck on "Waiting for services". It took 6 minutes before I could see an error dialog, and I could not force Rancher Desktop to quit. background.log showed a 'connection refused' error. (Unfortunately I did not think to backup that file so I lost the log.)

I tried resetting Kubernetes, and I tried quitting Rancher Desktop (after getting the error) and re-launching it, but I would get the same result: Rancher Desktop stuck on "Waiting for services..." for 6 minutes.

Only rebooting fixed the problem both times.

@gaktive gaktive modified the milestones: Next, Later Aug 30, 2022
@anwesha25
Copy link

Is there any update on this issue?
we are using Cisco AnyConnect VPN and connecting to internet with VPN enabled is failing

@Nino-K
Copy link
Member

Nino-K commented Nov 14, 2022

@anwesha25 are you able to use our 1.6 release? There is a workaround that mentioned here that should allow you to use RD over Cisco AnyConnect VPN.

@DavidCorral94
Copy link

Same problems here using Windscribe VPN. If its enable, I can't reach any container.

@anwesha25
Copy link

I had to switched back to Rancher desktop 1.4.1.
this version of Rancher Desktop does not auto start on starting my laptop. And wsl-vpnkit also needs to be restarted to seamlessly work with containers when VPN is enabled.

wsl --import wsl-vpnkit $env:USERPROFILE\wsl-vpnkit wsl-vpnkit.tar.gz --version 2
wsl -d wsl-vpnkit
wsl --setdefault rancher-desktop
wsl -d wsl-vpnkit service wsl-vpnkit start

the above steps need to be executed first with VPN disabled and then you connect to VPN. And always check if wsl distributions of rancher and vpnkit is running or not.

  > wsl -l -v
  NAME                    STATE           VERSION
* rancher-desktop         Running         2
  rancher-desktop-data    Stopped         2
  wsl-vpnkit              Running         2

Followed the above steps and it's kind of working fine for me now.
Hope this helps others.

@slonghei
Copy link

I have had similar challenges and had to revert back to 1.4.1 (along with VPN Toolkit) when using Cisco AnyConnect VPN. Per instructions for running VPN-toolkit, I created some simple script written to change the .kube/config file from the IP address created when Rancher Desktop is started and changing it to localhost. This creates more pain when a user has to run some script to simply be able to run Rancher Desktop alas is the only way it will sucessfully start.

@Nino-K
Copy link
Member

Nino-K commented Apr 26, 2023

we have introduced an experimental #3810 in 1.8.1 that should fix your VPN issue. The feature will be fully baked in our next few upcoming releases. As I mentioned it is experimental and the downside is the port forwarding for all the publish ports has to be performed manually as mentioned here: #4096 (comment)

@BLZB0B
Copy link

BLZB0B commented May 26, 2023

we have introduced an experimental #3810 in 1.8.1 that should fix your VPN issue. The feature will be fully baked in our next few upcoming releases. As I mentioned it is experimental and the downside is the port forwarding for all the publish ports has to be performed manually as mentioned here: #4096 (comment)

this appears to have fixed it for me using Cisco AnyConnect VPN :)

@Nino-K
Copy link
Member

Nino-K commented Jun 20, 2023

You can now enable the new network using rdctl:

rdctl set --experimental.virtual-machine.networking-tunnel=true

This should allow Rancher Desktop to function correctly behind a VPN. I'm going to close this issue, feel free to re-open if this suggestion is not solving the issue.

@Nino-K Nino-K closed this as completed Jun 20, 2023
@patrickleet
Copy link

@Nino-K

how about on macos?

> rdctl set --experimental.virtual-machine.networking-tunnel=true
Error: option --experimental.virtual-machine.networking-tunnel is not available on macOS

@htakemoto
Copy link

@Nino-K
Same issue here as @patrickleet on macOS. Any solution I am using Zscaler. Docker failed to install rpm package.

     Step 4/10 : RUN npm install
      ---> Running in 891528f47122
     npm ERR! code UNABLE_TO_GET_ISSUER_CERT_LOCALLY
     npm ERR! errno UNABLE_TO_GET_ISSUER_CERT_LOCALLY
     npm ERR! request to https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz failed, reason: unable to get local issuer certificate

@marcusportmann
Copy link

@Nino-K
This is unfortunately still an issue on MacOS. Are there any plans to address this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests