Minikube Ingress TCP Services Not Forwarding Traffic

[justinrcs]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version):
minikube version: v0.27.0

• OS (e.g. from /etc/os-release):
  MacOS 10.12.6 Sierra

• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
  virtualbox

• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):

minikube-v0.26.0.iso

• Install tools:
  brew

What happened:
Enabled ingress on minikube via addons manager
Created tcp-services.yaml in the kube-system namespace to forward traffic to a RabbitMQ service in the default namespace.
Verified RabbitMQ is accessible inside the minikube cluster via ClusterIP service
Verified that the nginx ingress has loaded the tcp-services.yaml ConfigMap and has a stanza in the nginx.conf file.
When attempting to telnet from host to minikube on the ingress port, get Connection Refused

What you expected to happen:
To be able to connect to a TCP-based service via the Ingress

How to reproduce it (as minimally and precisely as possible):
minikube addons enable ingress
create any tcp-based pod and appropriate service (RabbitMQ, Postgres, MySQL, etc)
add the following configmap to kube-system, substituting port, namespace and service-name

kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: kube-system data: port: "namespace/service-name:port"

Verify the TCP Services are configured with the Ingress Controller:
kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

Attempt to connect to the TCP service: telnet $(minikube ip) <port>

[seborama]

Hi,

Can you supply the following?

• your tcp-services.yaml file
• output from minikube status
• output from minikube service list
• output from minikube addons list
• output from kubectl get ingress,services -n=kube-system
• output from kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

Regards

[justinrcs]

Requested info below, thank you.

tcp-services.yaml file

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: kube-system
data:
  5672: "default/rabbitmq:5672"

output from minikube status

minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

output from minikube service list

|-------------|----------------------|--------------------------------|
|  NAMESPACE  |         NAME         |              URL               |
|-------------|----------------------|--------------------------------|
| default     | kubernetes           | No node port                   |
| default     | rabbitmq             | No node port                   |
| kube-system | default-http-backend | http://192.168.99.100:30001    |
| kube-system | kube-dns             | No node port                   |
| kube-system | kubernetes-dashboard | http://192.168.99.100:30000    |
| kube-system | tiller-deploy        | No node port                   |
|-------------|----------------------|--------------------------------|

output from minikube addons list

- addon-manager: enabled
- coredns: disabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- heapster: disabled
- ingress: enabled
- kube-dns: enabled
- metrics-server: disabled
- registry: disabled
- registry-creds: enabled
- storage-provisioner: enabled

output from kubectl get ingress,services -n=kube-system

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
svc/default-http-backend   NodePort    10.102.229.59    <none>        80:30001/TCP    6d
svc/kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   6d
svc/kubernetes-dashboard   NodePort    10.103.255.166   <none>        80:30000/TCP    6d
svc/tiller-deploy          ClusterIP   10.96.254.176    <none>        44134/TCP       6d

output from kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

printing the relevant part for the rabbit ingress:

stream {
	log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
	
	access_log /var/log/nginx/access.log log_stream;
	
	error_log  /var/log/nginx/error.log;
	
	# TCP services
	
	upstream tcp-5672-default-rabbitmq-5672 {
		
		server                  172.17.0.10:5672;
		
	}
	server {
		
		listen                  5672;
		
		listen                  [::]:5672;
		
		proxy_timeout           600s;
		proxy_pass              tcp-5672-default-rabbitmq-5672;
		
	}
	
	# UDP services
	
}

[seborama]

Hi,

You don't have an ingress defined :) The ingress controller is effectively your nginx reverse proxy. You need to define an ingress now to make use of it.

Example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress-path
  namespace: kube-system
  annotations:
    nginx.org/server-snippet: "proxy_ssl_verify off;"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - host: your.domain.name
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 80
        path: /dashboard

Then, add an entry in your DNS (or local hosts file, etc) for your.domain.name to point to your minikube ip. You should now be able to browse to http://your.domain.name/dashboard/ from your Mac. You could also define a host ingress if you prefer.

Regards.

[justinrcs]

Appreciate the response, but that's not how the TCP/UDP services work with the Ingress Controller. The Ingress sample you have above is for http (or https)-based services.

Assuming Minikube is using the same Ingress Controller via the AddOn as Kubernetes does, here is the documentation for exposing TCP/UDP:
Exposing TCP and UDP services

We have a similar setup in our cloud environments.

Thank you

[seborama]

Hi,

Struggling with it too.

On the minikube VM, after setting up the ingress controller, I can:

$ telnet 172.17.0.5 5672
GET / HTTP/1.1
AMQP	Connection closed by foreign host

Where 172.17.0.5 is the ingress controller and 172.17.0.7 is rabbitmq - so I can also:

$ telnet 172.17.0.7 5672
GET / HTTP/1.1
AMQP	Connection closed by foreign host

So the ingress controller works but I can't access rabbitmq outside the minikube VM. I think this is a limitation of the ingress in K8s. As the doc states, "Ingress does not support TCP or UDP services". The Nginx Ingress controller offers the work around based on ConfigMap but this has its limitations, I suspect. It can expose the port onto the Node but as this isn't a K8s Ingress, you cannot access the port outside the minikube VM - it isn't proxied.

The Helm package suggests these instructions (it didn't seem to work for me though):
Create a proxy server between localhost and the Kubernetes API Server:
kubectl proxy --port 8002

To Access the RabbitMQ AMQP port:

URL : amqp://127.0.0.1:8002/api/v1/proxy/namespaces/default/services/rabbitmq:5672/

To Access the RabbitMQ Management interface:

URL : http://127.0.0.1:8002/api/v1/proxy/namespaces/default/services/rabbitmq:15672/

Some other ideas (long shots):

• You might be able to install nginx to proxy amqp between the minikube vm and the nginx ingress controller
• Perhaps nc can do that too
• Find another lightweight utility that can or write your own.
• kubectl port-forward $POD_NAME 15672:15672

Please share here if you find a workaround / solution.

Best regards.

[lrvick]

Same issue here. If I connect to the nginx-ingress container I can confirm I can reach my container's tcp port through nginx, but it does not get exposed through minikube.

https://github.com/lrvick/weechat-helm/blob/master/templates/ingress.yaml

[edwardhey]

@juan131 minikube only expose port 80,443,18080 in ingress pod, so you should create another ingress yourself, then forward 5672 to your rabbitmq.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sseide]

Are there any plans to change the current behaviour and let user easily expose additional ports via builtin nginx ingress controller?

And it would be really helpful to have a link to ome ressources describing neccessary steps to deploy a new ingress controller to minikube to archive this feature requested here.

[fa1com]

I compare the ingress controller between minikube and the k8s on server.

Find out minikube's ingress without the configuration as below:

spec.template.spec.hostNetwork = true

After I add it, port can be able to forwarded to the vm host.

Hope this can help.

[tstromberg]

I don't know anything about Ingress yet, but would be happy to assist someone with getting this experience improved for all users. Help wanted!

[unixunion]

I am running into this same problem, where TCP services are not exposed on the IP the Minikube is running on. Ingress for HTTP works as expected, but not tcp-services defined as ConfigMap. My nginx config contains the relevant tcp mapping. I have also tried applying the 'hostNetwork = true' setting to the pod spec, but no success.

[sahlex]

Still not solved it seems. However, I have the same issue. Minikube's nginx ingress-controller does not do the job properly.

Please help.

[skryvets]

Update: The link to manifest file is fixed (was broken due to changes made in nginx repo)

Thanks to @MerlinPong I was able to make minikube forward TCP/UDP traffic. Here are the steps:

1. Disable default ingress addon

   minikube addons disable ingress
   

2. Deploy standalone nginx ingress controller:

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
   

   Here's a link to the official documentation.

3. Tweak the deployment of nginx-ingress-controller.
   Make sure it's running:

   $ kubectl get deploy nginx-ingress-controller -n ingress-nginx
   
   $ NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
     nginx-ingress-controller   1/1     1            1           8m39s
   

   Save the output to a file:

   kubectl get deploy nginx-ingress-controller -n ingress-nginx -o yaml>> nginx-ingress-controller.yaml
   

   Edit the file by applying spec.template.spec.hostNetwork = true and apply the configuration.

4. Use the example provided in the docs -> https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

I also wrote more detailed explanation in the article (but steps are exactly the same as here) -> Exposing TCP and UDP Services via ingress on Minikube

[Igor-lkm]

@skryvets @MerlinPong thank you, working as expected :)

@skryvets At the step Save the edited file and apply it by executing I have to use flag --force:

 kubectl apply -f nginx-ingress-controller.yaml  --force

[woodcockjosh]

This worked for me:

kubectl patch deployment nginx-ingress-controller -n kube-system --patch '{"spec":{"template":{"spec":{"hostNetwork":true}}}}'

kubectl patch configmap tcp-services -n kube-system --patch '{"data":{"3306":"default/mysql-service:3306"}}'

The main issue is that Ingress controller doesn't support tcp services. I really hope to see this supported soon. I have no idea why they ports are limited for Ingress.

This is based on the nginx hackaround https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

Good luck!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[woodcockjosh]

/assign

[woodcockjosh]

Relates to kubernetes/ingress-nginx#4624

[woodcockjosh]

Relates to
kubernetes/ingress-nginx#4586
kubernetes/ingress-nginx#4585

[woodcockjosh]

Relates to kubernetes/ingress-nginx#4626

[woodcockjosh]

Resolved by #5517

[woodcockjosh]

Relates to #5518

[woodcockjosh]

The only other thing I think we could do here other than a tutorial is add a custom patching container that watches changes on the tcp and udp configmaps and updates the ingress deployment so that the pods listen on the appropriate ports. I will leave it up to the project maintainers to determine whether or not this issue can be closed without that capability.

[bygui86]

I don't know if could help or not.
After NGINX I tried Traefik and it does not work as well. But I don't know how to fix it...

[woodcockjosh]

FYI https://minikube.sigs.k8s.io/docs/tutorials/nginx_tcp_udp_ingress/

[bygui86]

@woodcockjosh thanks a lot! NGINX ingress controller works perfectly right now.
I haven't found a fix for Traefik. Do you have an advice?

[woodcockjosh]

@bygui86 https://github.com/woodcockjosh/traefik/blob/add-minikube-config-to-crd-acme-user-guide/docs/content/user-guides/crd-acme/index.md
Example files are in same directory as index.md.

[bygui86]

Thanks @woodcockjosh I will try asap and let you know!!

[tstromberg]

@woodcockjosh mentioned that this should be closed now. Please /reopen if that is not the case. Thanks!

[Aadil-Nabi]

Hi,

I am kind of stuck in one of the issues. I am learning and developing the node.js web app using microservices. I have one section where I have to route the request from one default namespace to ingress-nginx namespace but I can't find the ingress-nginx namespace in the Kubernetes cluster.
I am using minikube on windows 7 enterprise.
Please help me where I can find the ingress-nginx load balancer. Is there anything specific we need to don in minikube. I can see we have ingress-nginx in mac and docker windows app. but with minikube I can't find any services related to Loadbalancer ingress-nginx.
In fact, I found one in Kube-system, but it is cluster Ip not load balancer type and when I tried it, it didn't work.

Thanks in advance.

[woodcockjosh]

@Aadil-Nabi I suggest you join the kubernetes slack channel for help on this. kubernetes.slack.com