Oathkeeper behind ssl terminating balancer

[Entrio]

I have deployed ory os on various systems and I always run into the same issue when dealing with oathkeeper. In my case there is an nginx server that terminates all https requests and forwards the traffic to an oathkeeper upstream server.

Due to the nature of this setup, I always have to specify my match rules as http protocol as that is what nginx passes to oathkeeper.

Can someone point me in the right direction on solving this issue? I could not find anything in the documentation.

Thanks

[aeneasr]

Does the forwarded request include X-Forwarded-Proto: https in the header?

[Entrio]

Yes, I have X-Forwarded-Proto set to $scheme (which in this case is https)

Or must I specify https explicitly?

[aeneasr]

No, in that case it would make sense to have https as the protocl for oathkeeper to match against!

[Entrio]

This is my nginx config:

server {

        server_name             gateway.***;

        location / {
                proxy_set_header                        Host                    $host;
                proxy_set_header                        X-Forwarded-For         $remote_addr;
                proxy_set_header                        X-Forwarded-Proto       $scheme;
                proxy_set_header                        X-Real-IP               $remote_addr;
                proxy_pass                              http://oathkeeper-proxy;
        }

        access_log              /var/log/nginx/gateway.access.log main;
        error_log               /var/log/nginx/gateway.error.log;

        listen          443     ssl; # managed by Certbot
        ssl_certificate         /etc/letsencrypt/live/gateway.***/fullchain.pem; # managed by Certbot
        ssl_certificate_key     /etc/letsencrypt/live/gateway.***/privkey.pem; # managed by Certbot
        include                 /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam             /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


}

In this case $scheme that is set is https. I will test with explicit https rather than $scheme and let you know how it goes.

Edit 1: Ive set my nginx config to proxy_set_header X-Forwarded-Proto https;

EDIT 2:
This is my rule on OA

{
        "id": "oathkeeper:rules:health",
        "description": "This is a test rule that should be awesome",
        "match": {
            "methods": [
                "GET"
            ],
            "url": "https://gateway.***/status"
        },
        "authenticators": [{
        "handler": "anonymous"
    }],
        "authorizer": {
            "handler": "allow",
            "config": null
        },
        "credentials_issuer": {
            "handler": "noop",
            "config": null
        },
        "upstream": {
            "preserve_host": true,
            "strip_path": "/status",
            "url": "http://172.30.1.25:9001/health/alive"
        }
    }

And when i make a request to https://gateway.***/status
Unfortunately i get a message saying

{
    "error": {
        "code": 404,
        "status": "Not Found",
        "message": "Requested url does not match any rules"
    }
}

[Entrio]

In the log i see the following message (it is clear it is passing http)

Feb 15 16:00:35 lb1 oathkeeper[26591]: time="2019-02-15T16:00:35+06:00" level=warning msg="Access request denied" access_url="http://gateway.***/status" error="Requested url does not match any rules" granted=false

[rauanmayemir]

@Entrio did you manage to fix this? I'm having similar issues with oathkeeper api failing to match the url to the rule.

[Entrio]

No, I've had to settle down to using http in rules as it doesn't take into account X-Forwarded-Proto

[christian-roggia]

I would like to know whether there is any update on this, we managed to set up our Istio environment so that the header X-Forwarded-Proto: https is forwarded correctly to oathkeeper, but as described in this ticket it gets ignored.

This is, in my opinion, a massive security issue as the solution proposed by @Entrio:

No, I've had to settle down to using http in rules as it doesn't take into account X-Forwarded-Proto

is a very big security hole and something I would never ever recommend in a production environment.

Let's take as an example a payment API where requests get routed via oathkeeper, if the header X-Forwarded-Proto is not honored and the only viable solution is to set up the rules (e.g. match.url: <{https,http}>://animeshon.com/) and error handlers (e.g. return_to in oathkeeper and whitelisted_return_urls in kratos) so that they accept HTTP requests as they were HTTPS, then I have no way to verify that the user is actually submitting requests only via HTTPS potentially, exposing sensitive information on a non-secure communication channel.

We can, of course, enforce HTTPS via e.g. Load Balancers, Istio rules, etc., but I believe this is beyond the point. This issue is promoting really bad and potentially harmful practices such as whitelisting and allowing everything regardless of the protocol in use.

[Entrio]

@christian-roggia completely agree.

I can understand ory.sh position somewhat, they assume and recommend that oathkeeper is running in a secure environment but still...

I have deployed ory stack on multiple projects now since I initially brought this up to attention (it has been 2 years!) and I've had no issue with using http as matching proto.

Maybe that now more and more people are raising this issue, maybe @aeneasr can look into it? :)

[christian-roggia]

In our case mTLS is managed by Istio and therefore all communication intra-cluster happens via TLS, while our ingress exposes only HTTPS (we completely disallow connections via HTTP).

The fact that oathkeeper is not honoring the header is even more disruptive for us as users get wrongly redirected to the http (instead of https) address where they will be met with an error message (we are waiting for ingress-gce redirect HTTP to HTTPS feature, which is already available on the RAPID release channel of GKE).

Unfortunately, I have a lot on my plate right now, otherwise I would've taken over this task myself.

[amoshydra]

#638 Doesn't solve the issue for me. Tested with ORY Oathkeeper v0.38.9-beta.1.pre.2

I have the same setup as described in #153 (comment)

And I am still getting error="Requested url does not match any rules" when making request to e.g. https://www.my-domain.com/

access-rules.yml

Same as https://github.com/ory/kratos/blob/f180dba2207638e83e4a23ebc213cddaecb5677f/contrib/quickstart/oathkeeper/access-rules.yml

-
  id: "ory:kratos-selfservice-ui-node:protected"
  upstream:
    preserve_host: true
    url: "http://auth-client:4435"
  match:
    url: "https://www.my-domain.com/<{,debug,dashboard,settings}>"
    methods:
      - GET
  ...

github.com/ory/oathkeeper/proxy.(*RequestHandler).HandleRequest

{
  "audience": "application",
  "error": {
    "debug": "",
    "message": "Requested url does not match any rules",
    "reason": "",
    "status": "Not Found",
    "status_code": 404,
    "trace": "\ngithub.com/ory/oathkeeper/rule.(*RepositoryMemory).Match\n\t/home/ory/rule/repository_memory.go:142\ngithub.com/ory/oathkeeper/proxy.(*Proxy).Director\n\t/home/ory/proxy/proxy.go:127\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/local/go/src/net/http/httputil/reverseproxy.go:241\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/reqlog.(*Middleware).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:134\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/oathkeeper/metrics.(*Middleware).ServeHTTP\n\t/home/ory/metrics/middleware.go:88\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96\ngithub.com/rs/cors.(*Cors).Handler.func1\n\t/go/pkg/mod/github.com/rs/[email protected]/cors.go:207\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2069\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2887\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1952\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"
  },
  "file": "/home/ory/proxy/proxy.go:81",
  "func": "github.com/ory/oathkeeper/proxy.(*Proxy).RoundTrip",
  "granted": false,
  "http_host": "www.my-domain.com",
  "http_method": "GET",
  "http_url": "http://www.my-domain.com/",
  //           ^^^^
  //           ⚠ http_url shows HTTP instead of HTTPS


  "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0",
  "level": "warning",
  "msg": "Access request denied",
  "service_name": "ORY Oathkeeper",
  "service_version": "v0.38.9-beta.1.pre.2",
  "time": "2021-03-16T13:30:06Z"
}

github.com/ory/x/reqlog.(*Middleware).ServeHTTP

{
  "file": "/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:139",
  "func": "github.com/ory/x/reqlog.(*Middleware).ServeHTTP",
  "http_request": {
    "headers": {
      "x-forwarded-proto": "https", // <---- This header has no effect to the resulting scheme
      "x-forwarded-for": "REDACTED",


      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "accept-encoding": "gzip, deflate, br",
      "accept-language": "en-US,en;q=0.5",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0"
    },
    "host": "www.my-domain.com",
    "method": "GET",
    "path": "/",
    "query": null,
    "remote": "172.22.0.1:45578",

    "scheme": "http" // <---- Logger reports http isntead of https
  },
  "http_response": {
    "status": 404,
    "text_status": "Not Found",
    "took": 880600
  },
  "level": "info",
  "msg": "completed handling request",
  "time": "2021-03-16T14:02:04Z"
}

Possible solution?

I manage to get it to work by checking for X-Forwarded-Proto header inside proxy.(*Proxy) EnrichRequestedURL using the same logic in #638

I have created a PR containing this solution here #665

github.com/ory/oathkeeper/proxy.(*RequestHandler).HandleRequest

{
  "audience": "application",
  "error": {
    "debug": "",
    "message": "Access credentials are invalid",
    "reason": "",
    "status": "Unauthorized",
    "status_code": 401,
    "trace": "\ngithub.com/ory/oathkeeper/proxy.(*RequestHandler).HandleRequest\n\t/app/proxy/request_handler.go:247\ngithub.com/ory/oathkeeper/proxy.(*Proxy).Director\n\t/app/proxy/proxy.go:134\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/local/go/src/net/http/httputil/reverseproxy.go:241\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/reqlog.(*Middleware).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:134\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/oathkeeper/metrics.(*Middleware).ServeHTTP\n\t/app/metrics/middleware.go:88\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96\ngithub.com/rs/cors.(*Cors).Handler.func1\n\t/go/pkg/mod/github.com/rs/[email protected]/cors.go:207\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2069\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2887\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1952\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"
  },
  "file": "/app/proxy/request_handler.go:252",
  "func": "github.com/ory/oathkeeper/proxy.(*RequestHandler).HandleRequest",
  "granted": false,
  "http_host": "www.my-domain.com",
  "http_method": "GET",
  "http_url": "https://www.my-domain.com/",
  //           ^^^^^
  //           🌟 http_url shows HTTPS; Request get handled as expected

  "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0",
  "level": "warning",
  "msg": "No authentication handler was responsible for handling the authentication request",
  "reason_id": "authentication_handler_no_match",
  "rule_id": "ory:kratos-selfservice-ui-node:protected",
  "service_name": "ORY Oathkeeper",
  "service_version": "master",
  "time": "2021-03-16T18:58:51Z"
}

github.com/ory/x/reqlog.(*Middleware).ServeHTTP

{
  "file": "/go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:139",
  "func": "github.com/ory/x/reqlog.(*Middleware).ServeHTTP",
  "http_request": {
    "headers": {
      "x-forwarded-proto": "https",
      "x-forwarded-for": "REDACTED",


      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "accept-encoding": "gzip, deflate, br",
      "accept-language": "en-US,en;q=0.5",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0"
    },
    "host": "www.my-domain.com",
    "method": "GET",
    "path": "/",
    "query": null,
    "remote": "172.22.0.1:48628",

    "scheme": "http" // <---- Logger reports http isntead of https
  },
  "http_response": {
    "status": 302,
    "text_status": "Found",
    "took": 1542800
  },
  "level": "info",
  "msg": "completed handling request",
  "time": "2021-03-16T18:58:51Z"
}

However, github.com/ory/x/reqlog continues to report scheme as http

[christian-roggia]

Have you tried to use the latest v0.38.9-beta.1 version? It works correctly for us with Istio + Oathkeeper, both rules and redirects are happening correctly.

[amoshydra]

@christian-roggia I tried with v0.38.9-beta.1.pre.2, which already contains the change from #638.
I have also tried building from master (at f3561e6)

As you have mentioned in #638 the change "affect only the decision API".

However, in my case and I assume in @Entrio's case, we use oathkeeper as a reverse proxy. The mechanism doesn't invoke api/decision.go. Therefore #638 does not solve the issue here.

---

The request's scheme is filled in /app/proxy/proxy.go
The rule matching happened in /app/proxy/request_handler.go

Trace:

github.com/ory/oathkeeper/proxy.(*RequestHandler).HandleRequest
    /app/proxy/request_handler.go:247
github.com/ory/oathkeeper/proxy.(*Proxy).Director
    /app/proxy/proxy.go:134

net/http/httputil.(*ReverseProxy).ServeHTTP
    /usr/local/go/src/net/http/httputil/reverseproxy.go:241
github.com/urfave/negroni.Wrap.func1
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46
github.com/urfave/negroni.HandlerFunc.ServeHTTP
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29
github.com/urfave/negroni.middleware.ServeHTTP
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38
github.com/ory/x/reqlog.(*Middleware).ServeHTTP
    /go/pkg/mod/github.com/ory/[email protected]/reqlog/middleware.go:134
github.com/urfave/negroni.middleware.ServeHTTP
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38
github.com/ory/oathkeeper/metrics.(*Middleware).ServeHTTP
    /app/metrics/middleware.go:88
github.com/urfave/negroni.middleware.ServeHTTP
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38
github.com/ory/x/metricsx.(*Service).ServeHTTP
    /go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:261
github.com/urfave/negroni.middleware.ServeHTTP
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38
github.com/urfave/negroni.(*Negroni).ServeHTTP
    /go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96
github.com/rs/cors.(*Cors).Handler.func1
    /go/pkg/mod/github.com/rs/[email protected]/cors.go:207
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2069
net/http.serverHandler.ServeHTTP
    /usr/local/go/src/net/http/server.go:2887
net/http.(*conn).serve
    /usr/local/go/src/net/http/server.go:1952
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1371