Kong, OIDC and Keycloak: Authentication fails with an error "KONG Error - an unexpected error occured" from time to time.

[Pinguwien]

Situation:

We've added an API "myapi" to kong (0.10.3, not clustered, just 1 pod in openshift v3) which is secured by keycloak (3.2.1.Final) via its js client adapter and the kong oidc plugin in version 1.0.1.

Now, when calling https://mykong/myapi as a non-authenticated user, it always(!) redirects to the login and it is possible to login with valid credentials.

So far, so good, but:
In say 4 of 10 cases, the redirect works perfectly. We get access to the API. Yay!

Sadly, the other 6/10 cases (approx.) are failing, showing a Browser Error Page showing nothing more than "KONG Error - an unexpected error occured". Yes, sadly one of these "the one time it works, other time it doesn't"-kind of errors.

The Keycloak event logs showing that it successfully returns a token to kong, no matter if the 500 shows up or not, so my first guess: on this side, everything works.

This is the output of the concerned Kong logs as far as I could find some:

`2017/10/25 13:59:32 [error] 100#0: *28057 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/plugins/oidc/utils.lua:63: attempt to index local 'user' (a nil value)

stack traceback:

coroutine 0:

/usr/local/share/lua/5.1/kong/plugins/oidc/utils.lua: in function 'injectUser'

/usr/local/share/lua/5.1/kong/plugins/oidc/handler.lua:43: in function 'access'

/usr/local/share/lua/5.1/kong.lua:295: in function 'access'

access_by_lua(nginx-kong.conf:94):2: in function <access_by_lua(nginx-kong.conf:94):1>, client: 10.104.159.85, server: kong, request: "GET /cfg HTTP/1.1", host: "gateway.hub-test.ose.db.de"

10.104.159.85 - - [25/Oct/2017:13:59:32 +0000] "GET /cfg HTTP/1.1" 500 131 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"`

We've tested this with all major browsers, same outcome, so I think its no browserspecific thing. Seems that, from time to time, there's a user missing, as far as I can read the logs?! Would be great to get some help in this case!

Feel free to ask for more info, i'll be happy to help!

Best regards,
Dominik

[Trojan295]

Update the version of this plugin to the newest (1.0.4).
It looks like a problem with the Userinfo Endpoint.

[Pinguwien]

just a short update on the issue:

• after doing the update to kong 1.0.4 using all the latest versions of the dependencies, we're running into an endless loop of requests resutling in an error. I'll investigate further here on monday, could be an issue with our kong version, which isn't up to date, too. i'll update it monday and hope, the error will be gone then. if not, i'll write again.

[jeremyjpj0916]

Any updates here? Did you try on 11.x Kong with the newest 1.0.4?

[Pinguwien]

@jeremyjpj0916 Sorry for the late response. I've been ill last week, sadly. Will try this out with my colleague who is the domainowner of the api gateway today and will give further response shortly. Had to set up a whole dev environment with new components in the process, working out how to set up keycloak-usage with ssl via a kong-route (/auth), so this had been a fairly more complex matter than I thought.

May I ask one question by the way?!:

Question is: If I use this plugin, there's no need to use any Keycloak Client Adapters, because kong does all the things I need, right? or did I get something wrong? That said another question (sorry): Is there an integration into the spring security adapter to obtain the roles automatically, yet? Or do I have to write this myself (which I would do, for sure)

[dominikguhr]

so, the first error here is gone, upgrading kong to 0.11.1, keycloak to 3.4.0 Final and the oidc plugin to 1.0.4...
but now i am stuck in a redirect loop calling the api which uses the oidc plugin. seems like the token keycloak sends is wrong calling it via gateway/auth after logging in. In keycloak, a session is established. But I am getting "invalid signature" when I try looking the token up at jwt.io, seems like payload is missing, but when i call the token endpoint directly with username/pw i get a fully working token. After editing nginx.conf for the last few hours and reading through I think half the internet, To be honest, I have no clue why. I have to investigate it further tomorrow (23:59 here), but any hint would be highly appreciated :(

this is the call with the wrong token:
https://{gateway}/{appUri}/?state=314101b1550be1a837d38cdc4a6e1a8e&code=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..4GUH3HFFfRstBrr2_Q403w.y05R2HBW0dwXX_bRDFpxHUPvkw8Ks7MQFFqWqpOm5UQ3RMK0Vpmq3Qcxqh6P1zLUIKASLMvpbMkzbOXUbRjOoKgyegEC2LkSnae2BocvifiY8Pes9itt-h9ubBi20udQsz8sD3N92rz5QTo8dHKI0TzmT3JEz3IM7xDHIP6eKzVDrfQKEbiMzOIeVc3fHHV0K1sgbf-dj0wlpfqvDHz5shKJMP8Kxf8o9nuUhYUmq1sb6uZ7h-FxtwKSXlEyt-xy.FsXCn91d6F_dD2zkAC6IWw

the '..' seems to be wrong.

[Pinguwien]

oh, just seen i've posted under the wrong account. my bad.

[Pinguwien]

@Trojan295

So, my situation as is is mentioned above ("dominikguhr" being another account of me, actually, wrong mail used ;) ). I'll try to give you some important data about my environment, though I couldn't get it to work yet, sadly. I really, really want this. I hope you or anyone reading this could help me out here:

• I've upgraded kong to 0.11.1
• I've upgraded Keycloak to 3.3.0 Final
• I've upgraded the oidc plugin to 1.0.4 with the following dependencies (excerpt from my Dockerfile):

#copy rocks to directory because no access to luarocks mirrors
COPY rocks /tmp/
RUN luarocks install /tmp/lua-resty-session-2.8-1.all.rock && \
    luarocks install /tmp/lua-resty-jwt-0.1.11-0.src.rock && \
    luarocks install /tmp/lua-resty-http-0.11-0.src.rock && \
    luarocks install /tmp/lua-resty-hmac-v1.0-1.all.rock && \
    luarocks install /tmp/lua-resty-openidc-1.4.0-1.src.rock && \
    luarocks install /tmp/kong-oidc-1.0.4-0.src.rock

COPY nginx_kong.lua /usr/local/share/lua/5.1/kong/templates/nginx_kong.lua

ENV KONG_CUSTOM_PLUGINS=oidc

Following is my customized nginx_kong.lua, which I refactored according to several stackoverflow-threads to send the right headers etc. (would be great to have an example nginxkonf.lua, btw!):

return [[
charset UTF-8;

> if anonymous_reports then
${{SYSLOG_REPORTS}}
> end

error_log ${{PROXY_ERROR_LOG}} ${{LOG_LEVEL}};

> if nginx_optimizations then
>-- send_timeout 60s;          # default value
>-- keepalive_timeout 75s;     # default value
>-- client_body_timeout 60s;   # default value
>-- client_header_timeout 60s; # default value
>-- tcp_nopush on;             # disabled until benchmarked
>-- proxy_buffer_size 128k;    # disabled until benchmarked
>-- proxy_buffers 4 256k;      # disabled until benchmarked
>-- proxy_busy_buffers_size 256k; # disabled until benchmarked
>-- reset_timedout_connection on; # disabled until benchmarked
> end

client_max_body_size ${{CLIENT_MAX_BODY_SIZE}};
proxy_ssl_server_name on;
underscores_in_headers on;

lua_package_path '${{LUA_PACKAGE_PATH}};;';
lua_package_cpath '${{LUA_PACKAGE_CPATH}};;';
lua_socket_pool_size ${{LUA_SOCKET_POOL_SIZE}};
lua_max_running_timers 4096;
lua_max_pending_timers 16384;
lua_shared_dict kong                5m;
lua_shared_dict kong_cache          ${{MEM_CACHE_SIZE}};
lua_shared_dict kong_process_events 5m;
lua_shared_dict kong_cluster_events 5m;
> if database == "cassandra" then
lua_shared_dict kong_cassandra      5m;
> end
lua_socket_log_errors off;
> if lua_ssl_trusted_certificate then
lua_ssl_trusted_certificate '${{LUA_SSL_TRUSTED_CERTIFICATE}}';
lua_ssl_verify_depth ${{LUA_SSL_VERIFY_DEPTH}};
> end

init_by_lua_block {
    kong = require 'kong'
    kong.init()
}

init_worker_by_lua_block {
    kong.init_worker()
}

proxy_next_upstream_tries 999;

upstream kong_upstream {
    server 0.0.0.1;
    balancer_by_lua_block {
        kong.balancer()
    }
    keepalive ${{UPSTREAM_KEEPALIVE}};
}

server {
    server_name kong;
    listen ${{PROXY_LISTEN}}${{PROXY_PROTOCOL}};
    error_page 400 404 408 411 412 413 414 417 /kong_error_handler;
    error_page 500 502 503 504 /kong_error_handler;

    access_log ${{PROXY_ACCESS_LOG}};
    error_log ${{PROXY_ERROR_LOG}} ${{LOG_LEVEL}};

    client_body_buffer_size ${{CLIENT_BODY_BUFFER_SIZE}};

> if ssl then
    listen ${{PROXY_LISTEN_SSL}} ssl${{HTTP2}}${{PROXY_PROTOCOL}};
    add_header X-debug-ssl "if ssl-block reached" always;
    ssl_certificate ${{SSL_CERT}};
    ssl_certificate_key ${{SSL_CERT_KEY}};
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_certificate_by_lua_block {
        kong.ssl_certificate()
    }

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ${{SSL_CIPHERS}};
> end

> if client_ssl then
    proxy_ssl_certificate ${{CLIENT_SSL_CERT}};
    proxy_ssl_certificate_key ${{CLIENT_SSL_CERT_KEY}};
> end

    real_ip_header     ${{REAL_IP_HEADER}};
    real_ip_recursive  ${{REAL_IP_RECURSIVE}};
> for i = 1, #trusted_ips do
    set_real_ip_from   $(trusted_ips[i]);
> end

    location /auth {
        proxy_pass https://identity-dev.bh-infra-dev.svc.cluster.local:8443;
        proxy_set_header Host $host;

        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Url-Scheme $scheme;
    }

    location / {
        set $upstream_host               '';
        set $upstream_upgrade            '';
        set $upstream_connection         '';
        set $upstream_scheme             '';
        set $upstream_uri                '';
        set $upstream_x_forwarded_for    '';
        set $upstream_x_forwarded_proto  '';
        set $upstream_x_forwarded_host   '';
        set $upstream_x_forwarded_port   '';

        set $session_check_ssi off;
        set $session_secret hallotest123123;

        rewrite_by_lua_block {
            kong.rewrite()
        }

        access_by_lua_block {
            kong.access()
        }

        proxy_http_version 1.1;
        proxy_set_header   Host              $upstream_host;
        proxy_set_header   Upgrade           $upstream_upgrade;
        proxy_set_header   Connection        $upstream_connection;
        proxy_set_header   X-Forwarded-For   $upstream_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $upstream_x_forwarded_proto;
        proxy_set_header   X-Forwarded-Host  $upstream_x_forwarded_host;
        proxy_set_header   X-Forwarded-Port  $upstream_x_forwarded_port;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_pass_header  Server;
        proxy_pass_header  Date;
        proxy_ssl_name     $upstream_host;
        proxy_pass         $upstream_scheme://kong_upstream$upstream_uri;

        header_filter_by_lua_block {
            kong.header_filter()
        }

        body_filter_by_lua_block {
            kong.body_filter()
        }

        log_by_lua_block {
            kong.log()
        }
    }

    location = /kong_error_handler {
        internal;
        content_by_lua_block {
            kong.handle_error()
        }
    }
}

server {
    server_name kong_admin;
    listen ${{ADMIN_LISTEN}};

    access_log ${{ADMIN_ACCESS_LOG}};
    error_log ${{ADMIN_ERROR_LOG}} ${{LOG_LEVEL}};

    client_max_body_size 10m;
    client_body_buffer_size 10m;

> if admin_ssl then
    listen ${{ADMIN_LISTEN_SSL}} ssl${{ADMIN_HTTP2}};
    ssl_certificate ${{ADMIN_SSL_CERT}};
    ssl_certificate_key ${{ADMIN_SSL_CERT_KEY}};
    ssl_protocols TLSv1.1 TLSv1.2;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ${{SSL_CIPHERS}};
> end

    location / {
        default_type application/json;
        content_by_lua_block {
            kong.serve_admin_api()
        }
    }

    location /nginx_status {
        internal;
        access_log off;
        stub_status;
    }

    location /robots.txt {
        return 200 'User-agent: *\nDisallow: /';
    }
}
]]

Keycloak works using a kong-route, too, so I can call my https://gateway/auth ... to call keycloaks endpoint config etc. via 8443/https.

My Keycloak dockerfile looks like this:

FROM jboss/keycloak:3.3.0.Final

USER root

ENV KEYCLOAK_LOGLEVEL DEBUG
ENV PROXY_ADDRESS_FORWARDING true

ADD config.sh /tmp/
ADD batch.cli /tmp/

RUN bash /tmp/config.sh

#Give correct permissions when used in an OpenShift environment.
RUN chown -R jboss:0 $JBOSS_HOME/standalone && \
    chmod -R g+rw $JBOSS_HOME/standalone

USER jboss

EXPOSE 8080 8443

There's a little jboss-cli script called in config.sh which sets the following entries to my standalone.xml (checked it in container, the entries are there):

embed-server --std-out=echo

# Enable SSL on a Reverse Proxy
# First add proxy-address-forwarding and redirect-socket to the http-listener element.
# Then add a new socket-binding element to the socket-binding-group element.

batch

/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=proxy-address-forwarding,value=true)

/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=redirect-socket,value=proxy-https)

/socket-binding-group=standard-sockets/socket-binding=proxy-https:add(port=443)
/subsystem=logging/console-handler=CONSOLE:change-log-level(level=ALL)
/subsystem=logging/root-logger=ROOT/:write-attribute(name=level,value=DEBUG)
run-batch

stop-embedded-server

As I understood, this should be fairly enough low-level config to make the plugin work as rp.

After deploying these two components to my openshift platform (3.6, just 1 pod each), I have set up a confidential client in a realm named "hub" and copied the client id ("landingpage") and client secret to use them in the oidc config. On the realm config-level, I only changed the switch "Require SSL" to all requests.

Clientconfig in keycloak:
here

So then I went to kong (using the kong-dashboard gui) and I've set up the following APIs (see pictures):

/auth API

The "App" I want to secure is a simple html landingpage and I want to call it via gateway/lop, for testing purposes. The API looks like this at the moment:

/lop API

I tried around a bit with Strip Uri and Preserve Host, but didn't help.

Then I added the oidc plugin to the API. Configuration for the plugin is here (hope not too small):

https://imgur.com/a/5Ausu

So,after doing this I tried to call gateway/lop. I got redirected to the loginpage of keycloak with a url like this:https://gateway-dev.{domain}/auth/realms/hub/protocol/openid-connect/auth?response_type=code&client_id=landingpage&state=8f26c2d2c24e0c3c17ce0de0bb77266a&redirect_uri=https://gateway-dev.{domain}/lop/&prompt=&nonce=1a81556204a30065872a1f8d7d7aa48f&scope=openid- so far, so good.

Next request after adding credentials is a POST to this url:
https://gateway-dev.{domain}/auth/realms/hub/login-actions/authenticate?code=nlCRXwc5gcoqGGeX4o-Z7aYDsWSAACCL4GLVxfJO_uM&execution=96682f02-2348-4f26-b753-4fae9ab906c5&client_id=landingpage
So, after adding the (valid) credentials, I got stuck into the redirect loop you could see in the following picture:

redirectloop

Corresponding kong logs: https://pastebin.com/v8ELbsbC

So, that's the situation. The Session is established successfully in keycloak, btw. Would be absolutely great to get some help here. I guess it's something in the nginx.conf I am missing, or I am having a knot in the brain (might be a big one).

Best regards and thanks in advance, Dominik

[Pinguwien]

couldn'T solve this, yet. Is there really nothing you could help me with? even an example kong/nginx conf would be nice.

[tsyrjanen]

Hi

Just guessing ...

In our first issue, #1, I wrote

"In nginx conf-file we have added under
server {
server_name kong;

following line

set_decode_base64 $session_secret 'XX';

In XX we use some decoded default value which kong-oidc will later change."

You have session_secret in "location / "

[tsyrjanen]

Have you enabled kong_oidc plugin "globally" or for specific apis?

If I remember right we had this issue when we enabled kong_oidc plugin globally. That is why added filters into schema.lua.

So you have /auth API to access keycloak. Then you should add "config.filters=^/auth$,^/auth[^%w_%-%.~]" when you enable kong_oidc plugin.

[Pinguwien]

Hi guys, and thanks for the answers so-far!

I finally found a solution (nearly). The cause of the problem was the version of lua resty-session plugin which is 2.8-1 as stated here as minimum req: https://luarocks.org/modules/hanszandbelt/lua-resty-openidc

After upgrading it to 2.19.1 the error was gone, login worked - yay. I found this issue at the repo which led me to the right direction: bungle/lua-resty-session#35

I think so because I checked with version 2.13-1, the last before the fix in 2.15.1 => redirect loop.

Now sadly, there's one thing missing: The X-UserInfo Header which should come back after logging in.

So may it be possible that your plugin doesn't support chunked cookies? If so, could you please make this work? Would be great! If I could give you any help, just ask. Sadly I never did a thing in lua ;)

Here are the kong debug logs with version 2.19-1 if you're interested. I cannot see why there shouldn't be the userinfo header set, I can even see the call the plugin makes:
https://pastebin.com/7un2RVcR

but in the browser my requests are looking like this when logged in:

https://imgur.com/a/T5HUw

@tsyrjanen @Trojan295 Thank you very much so far, and if you need more info feel free to reach out! I hope we get this last hurdle fixed fast so I can finally use Keycloak and oidc in our scenario :)

[Pinguwien]

One thing besides: Would be very nice to exactly know which versions of the oidc plugin and its dependencies you are using together with which kc version, because maybe that I have another "wrong" version.

Here's my actualised version-list for oidc plugin:

#copy rocks to directory because no access to luarocks mirrors
COPY rocks /tmp/
RUN luarocks install /tmp/lua-resty-session-2.19-1.all.rock && \
    luarocks install /tmp/lua-resty-jwt-0.1.11-0.src.rock && \
    luarocks install /tmp/lua-resty-http-0.11-0.src.rock && \
    luarocks install /tmp/lua-resty-hmac-v1.0-1.all.rock && \
    luarocks install /tmp/lua-resty-openidc-1.4.0-1.src.rock && \
    luarocks install /tmp/kong-oidc-1.0.4-0.src.rock

This is part of my Dockerfile, as stated in another post above. Keycloak used was 3.4.0.Final, and then 3.3 for testing, but it should work with 3.2.1, 3.3 and 3.4 nevertheless.

[Pinguwien]

okay, you can close the issue. after cleaning up kc-config and nginx.conf (I removed the whole /auth route, its not necessary - just left the session_secret in server, as @tsyrjanen stated), the X-UserInfo header appeared when routing to httpbin.org/get as upstream to check the headers.

Thank you for your support.

btw: a list of dependency versions would be nice, nevertheless ;)