using minikube tunnel with VMware Fusion on MacOS with MacOS firewall

[schwichtgit]

What Happened?

ENV:

macOS Monterey Version 12.1 (21C52) (x86_64)
minikube version: v1.24.0 (commit: 76b94fb) (with fuse and tcpdump)
VMware Fusion Professional Version 12.2.1 (18811640)

MacOS application firewall enabled.

$ minikube tunnel
Status:
machine: minikube
pid: 17093
route: 10.64.0.0/16 -> 192.168.144.214
minikube: Running
services: [anzograph-azg-db-lb, anzograph-azg-frontend-lb]

$ netstat -rn|grep 10.64
10.64/16 192.168.144.214 UGSc bridge100

$ cat /etc/resolver/cluster.local
nameserver 10.64.0.10
search_order 1

$ dig @10.64.0.10 host.minikube.internal +short

hangs and eventually times out.

expected result:
$ dig @10.64.0.10 host.minikube.internal +short
192.168.144.1

also .. (the more important use case) the k8s services endpoints exposed can not be connected.

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
anzograph-azg ClusterIP None 12345/TCP 15h
anzograph-azg-db-lb LoadBalancer 10.64.134.186 10.64.134.186 5600:30569/TCP,5700:31389/TCP,7000:30623/TCP 15h
anzograph-azg-frontend-lb LoadBalancer 10.64.224.23 10.64.224.23 443:30870/TCP,80:32733/TCP 15h
kubernetes ClusterIP 10.64.0.1 443/TCP 15h

$ curl -k https://10.64.224.23

hangs

expected result:

does not hang

if you use the same in the minikube VM
$ curl -k https://10.64.224.23

does not hang.

if you add the route in a seperate VMware fusion (linux) VM the LB endpoints are accessible too.

on MacOS the firewall shows

$ sudo pfctl -a "com.apple.internet-sharing/host_v4" -sr
Password:
No ALTQ support in kernel
ALTQ related functions disabled
block drop on bridge101 all
(...)
block drop on bridge100 all
pass on bridge100 inet from 192.168.144.0/24 to 192.168.144.0/24 flags any keep state
pass on bridge100 inet from 192.168.144.0/24 to 224.0.0.0/24 flags any keep state
pass on bridge100 inet from 192.168.144.0/24 to 255.255.255.255 flags any keep state
pass on bridge100 inet proto udp from 0.0.0.0 port = 68 to any port = 67 keep state

if one adds
pass on bridge100 inet from 192.168.144.0/24 to 10.64.0.0/16 flags any keep state

the MacOS firewall permits accessing the minikube k8s svc endpoints (and kube-dns) again.

In my local setup I created a file with the STDOUT content of

sudo pfctl -a "com.apple.internet-sharing/host_v4" -sr 2> /dev/null| tee rules

added the pass rule
pass on bridge100 inet from 192.168.144.0/24 to 10.64.0.0/16 flags any keep state
and
wrote it back
$ sudo pfctl -a "com.apple.internet-sharing/host_v4" -f ./rules
(please do at your own risk)

Can we please make minikube tunnel create the service CIDR rule automatically?

Attach the log file

Operating System

macOS (Default)

Driver

VMware

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[sharifelgamal]

This seems like a new feature that would needed to be added to minikube tunnel, I don't think we currently have the bandwidth to implement it, but I'd be happy to review a PR that does.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[klaases]

This seems like a new feature that would needed to be added to minikube tunnel, I don't think we currently have the bandwidth to implement it, but I'd be happy to review a PR that does.

Hi @schwichtgit, did you have bandwidth or interest in working on this issue?

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.