Nginx Ingress Controller - the /foo redirection is not working ?

[theobolo]

Hi guys,

I work with the Ingress / Nginx Ingress Controller, but i can't get the /foo redirection working.

I deployed on a Kubernetes cluster a daemonset of Nginx ingress controller, default backend is the http-default-backend example.

This is my ingress object :

# apimobile Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: apimobile-map
spec:
  rules:
  - host: azure.coursierprive.com
    http:
      paths:
      - path: /apimobile
        backend:
          serviceName: application-apimobile
          servicePort: 80
  - host: kube-20160428-120550-loadbalancer.northeurope.cloudapp.azure.com
    http:
      paths:
      - path: /
        backend:
          serviceName: application-apimobile
          servicePort: 80

My second rule is working but not the first one, if i try to reach my api on : azure.coursierprive.com/apimobile

I face the : backend - 404

I tried with Jenkins portal instead of my API but i face the same problem.

[aledbf]

@theobolo can you post a gist with the log of the pod?

[theobolo]

@aledbf There it is : https://gist.github.com/theobolo/48724a1ebeef6296d7bfcc2b4feb9844

It's just the log of one Pod of the DaemonSet where we see the request to /apimobile returns 404

Btw i added the kubectl get pod,svc,ing output.

[aledbf]

@theobolo please execute kubectl exec nginx-ingress-lb-zg9g9 cat /etc/nginx/nginx.conf and post it in a gist

[theobolo]

@aledbf Added to the Gist : nginx.conf

[aledbf]

@theobolo please check with the latest version gcr.io/google_containers/nginx-ingress-controller:0.61

[theobolo]

@aledbf I checked out the last version but i face the same problem,

I tried with many differents frontends but my /foo or /bar redirections always return a 404 error :/

Tried with Jenkins, my API, Dashboard ...

In the Nginx Conf i can see the Location rule /apimobile and /jenkins for exemple but it's not working :/

Théo

[aledbf]

@theobolo can you post the new nginx.conf?

[theobolo]

Yep i updated the Nginx.conf on the Gist.

There is Jenkins on azureapi.coursierprive.com and on azureapi.coursierprive.com/jenkins but not working on /jenkins.

[aledbf]

@theobolo please try the image aledbf/nginx-third-party:0.20 it contains #898 that reverse the order of the locations in nginx so / is the last attempted route.

[theobolo]

oh ! thx i give you my feedback in few minutes !

[theobolo]

So i tried your image but even if the Location / is putted after the /jenkins in the nginx.conf it's not working :/

I talked with a folk and he thinks that's maybe a problem about the URL asked by Nginx to the Jenkins pod. Maybe the Nginx controller is asking jenkins on http://default-jenkins-80/jenkins ? or somehting similar ?

[theobolo]

Found something about that :

http://serverfault.com/questions/444532/reverse-proxy-remove-subdirectory

Maybe is only missing a / at the proxy-pass line in the nginx.conf :

      proxy_pass http://default-application-apimobile-80;
      proxy_pass http://default-application-apimobile-80/;

[aledbf]

http://default-jenkins-80

good point, Location / this will not redirect to /jenkins automatically. Currently is not possible to express this using Ingress rules. Please check #679

proxy_pass http://default-application-apimobile-80/;

same here. When is requires the final slash or not? This is one of the missing things in Ingress.
kubernetes/kubernetes#16891

If you have an application (not prepackaged app) you can handle this internally (the redirect) as a workaround.

Any idea or feedback is welcomed.

[theobolo]

Thanks for all that stuff, i looked all the issues and for the moment it's a very active problem ;)

I think that i'll use the application way as Grafana and the ROOT_URL variable in the YML.
Still looking at this, i'll give you my feedback if i find anything.

I think that the best way is to implement something with Ingress as the Redirect and Rewrite stuff without soliciting the backend app. But the cloud providers apparently are not allowing that :/

Thank you very much for your help Manuel,
I keep in touch

Cheers,
Théo

[aledbf]

@theobolo please check the rewrite example
(gcr.io/google_containers/nginx-ingress-controller:0.7)
Any feedback or suggestion is welcomed

[fabiormoura]

@aledbf I was having the same problem with subroutes. I just tried deploying a pod with nginx-ingress-controller:0.7 and it's working now.

[theobolo]

Hey @aledbf it's working well ! I can reach the jenkins Dashboard with the subdirectory /jenkins without changing anything on the controller so it's a good good point !

BTW jenkins is looking for example for some static ressources, even if the path is /jenkins i touch the dahsboard, but the ressources are broken (Css, JS) because it tries to look at /static and not /jenkins/static.

I think that not a problem about the rewrite stuff it's a global problem about the Nginx that not able to rewrite all the paths and just making a symbolic link from /jenkins to / : with the real path=/

If i want to touch jenkins at /jenkins without breaking all the dashboard, i still need to modify the ROOT_URL in the Jenkins controller or in my apps to tell to the CSS that the path as changed to /jenkins.

So it's very affective for an API or a compute service that not need any attached ressources like a web page.

Tell me if i'm wrong and if i missed something about the concept, i'm really thankful about your update ;)

[fabiormoura]

@theobolo I had the same issue with static assets on jenkins you described but I was able to fix this issue. Are you using the official docker jenkins image?

For the official jenkins image you can set the environment variable with JENKINS_OPTS="--prefix=/jenkins" and it won't be necessary to configure a custom rewrite annotation in the ingress specs as @aledbf suggested. Here is one example:

Deployment + Service + Ingress specs

kind: List
apiVersion: v1
items:
- kind: Deployment
  apiVersion: extensions/v1beta1
  metadata:
    name: jenkins-deployment
  spec:
    replicas: 1
    template:
      metadata:
        labels:
          app: jenkins
          version: 1.651.2-alpine
      spec:
        containers:
        - name: jenkins
          image: jenkins:1.651.2-alpine
          ports:
          - containerPort: 8080
          env:
          - name: JENKINS_OPTS
            value: "--prefix=/jenkins"
- kind: Service
  apiVersion: v1
  metadata:
    name: jenkins
    labels:
      app: jenkins
  spec:
    ports:
    - port: 80
      targetPort: 8080
    selector:
      app: jenkins
- kind: Ingress
  apiVersion: extensions/v1beta1
  metadata:
#    annotations:
#      ingress.kubernetes.io/rewrite-target: / (suggested by @aledbf but it's not necessary if using JENKINS_OPTS above)
    name: jenkins-ingress
    labels:
      app: jenkins
  spec:
    backend:
    rules:
      http:
        paths:
        - path: /jenkins
          backend:
            serviceName: jenkins
            servicePort: 80

[theobolo]

@fabiormoura Thanks for that but i'm already aware of that possibility i called that the ROOT_URL variable in my previous posts witch is the --prefix that we are using for jenkins there.

But by the way that was functional before the rewrite merging, just wondering if the rewrite function could handle that kind of stuff (static ressource rewrite too) without touching or modifying my own apps to add a prefix env variable in the YML or not ;)

But thanks for the solution anyway :)

[aledbf]

@theobolo there's an additional annotation ingress.kubernetes.io/add-base-url:"true" (I need to add an example) that will "inject" the ingress route in the response body. Sadly this do not work with jenkins because all the links are absolute (if they were relative this works OOTB)

Just as an example in your /jenkins rule the body of any page will return
<head><base http://$server_name/jenkins/">......

[aledbf]

thanks @fabiormoura for the example

[theobolo]

@aledbf thanks for that other annotation it's helpfull in my case :D

[bprashanth]

@theobolo @fabiormoura suggest contributing to the docs/examples section if you have time, we can always use more good examples

[eghobo]

@theobolo: did you get any luck with running Grafana under ingress controller? i am having trouble with '/grafana' path, 404 error at login page.

[theobolo]

@eghobo I must say that i'm not using Nginx/Ingress controller anymore since the Azure cloud provider implementation gives me PublicIP adresses for eachs applications.

Did you change into the Grafana controller the RootPath of the grafana webserver to point to /grafana ?

[eghobo]

@theobolo I figure it out (trailing slash was the issue), see examples below (in case if you are interested)

apiVersion: v1
kind: ReplicationController
metadata:
  name: grafana-v1
  labels:
    app: grafana
    version: v1
spec:
  replicas: 1
  selector:
    app: grafana
    version: v1
  template:
    metadata:
      labels:
        app: grafana
        version: v1
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:3.1.1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
              cpu: 100m
              memory: 100Mi
        env:
          - name: "GF_SERVER_ROOT_URL"
            value: "http://example.com/grafana/"
        ports:
        - containerPort: 3000
          name: ui
          protocol: TCP

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana-ingress-v1
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - example.com
    secretName: nginx-ingress-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /grafana/
        backend:
          serviceName: grafana
          servicePort: 3000

[theobolo]

@eghobo Thanks for sharing ! Well done

[jungopro]

thanks @eghobo.
With the help of your example I was able to fix my issue as well.
For me it was a bit more tricky since I'm also using grafana.ini via a configmap object. In addition, I wanted to expose it on port 80, not 3000.

My working setup (should others find it useful) is this:

• I added the env part from your example to my deployment
• I added the ingress annotation for rewrite
• I modified the path to include / in the end (/grafana/ vs /grafana)
• I modified the grafana.ini with the following values:

1. ;http_port = 80
2. ;domain = my.fully.qualified.domain
3. ;root_url = %(protocol)s://%(domain)s:%(http_port)s:/grafana/

Thank you very much for this example, really helpful

[dkirrane]

Question: why is the trailing slash needed (/grafana/ vs /grafana)?

[ratulbasak]

NEED help!!!

I'am facing the same problem.
my sample-ingress.yml
`apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-internal-ingress
namespace: default
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:

• host: stg-tenant.selise.biz
  http:
  paths:
  • backend:
    serviceName: web
    servicePort: 8080
    path: /v1`

In path /v1 it returned 404
`ubuntu@kube-master:~/scripts/ingress$ curl stg-tenant.selise.biz/v1

<title>404 Not Found</title>

404 Not Found

---

nginx/1.15.3 `

nginx.conf from the nginx-controller
` upstream upstream_balancer {
server 0.0.0.1; # placeholder

	balancer_by_lua_block {
		balancer.balance()
	}
	
	keepalive 32;
	
}

server {
	server_name stg-tenant.selise.biz ;
	
	listen 80;
	
	listen [::]:80;
	
	set $proxy_upstream_name "-";
	
	location /v1 {
		
		set $namespace      "default";
		set $ingress_name   "nginx-internal-ingress";
		set $service_name   "web";
		set $service_port   "8080";
		set $location_path  "/v1";
		
		rewrite_by_lua_block {
			
			balancer.rewrite()
			
		}
		
		log_by_lua_block {
			
			balancer.log()
			
			monitor.call()
		}
		
		port_in_redirect off;
		
		set $proxy_upstream_name "default-web-8080";
		
		client_max_body_size                    "1m";
		
		proxy_set_header Host                   $best_http_host;
		
		# Pass the extracted client certificate to the backend
		
		# Allow websocket connections
		proxy_set_header                        Upgrade           $http_upgrade;
		
		proxy_set_header                        Connection        $connection_upgrade;
		
		proxy_set_header X-Request-ID           $req_id;
		proxy_set_header X-Real-IP              $the_real_ip;
		
		proxy_set_header X-Forwarded-For        $the_real_ip;
		
		proxy_set_header X-Forwarded-Host       $best_http_host;
		proxy_set_header X-Forwarded-Port       $pass_port;
		proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
		
		proxy_set_header X-Original-URI         $request_uri;
		
		proxy_set_header X-Scheme               $pass_access_scheme;
		
		# Pass the original X-Forwarded-For
		proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
		
		# mitigate HTTPoxy Vulnerability
		# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
		proxy_set_header Proxy                  "";
		
		# Custom headers to proxied server
		
		proxy_connect_timeout                   5s;
		proxy_send_timeout                      60s;
		proxy_read_timeout                      60s;
		
		proxy_buffering                         "off";
		proxy_buffer_size                       "4k";
		proxy_buffers                           4 "4k";
		proxy_request_buffering                 "on";
		
		proxy_http_version                      1.1;
		
		proxy_cookie_domain                     off;
		proxy_cookie_path                       off;
		
		# In case of errors try the next upstream server before returning an error
		proxy_next_upstream                     error timeout;
		proxy_next_upstream_tries               3;
		
		proxy_pass http://upstream_balancer;
		
		proxy_redirect                          off;
		
	}

`