kubectl refresh issues

[zihaoyu]

We use dex as OIDC provider for kube-apiserver, and configure kubeconfig to use OIDC authenticator in user config. Our expiration settings look like the following:

expiry:
  signingKeys: 6h
  idTokens: 24h

However we noticed that when we come into office the next day, kubectl can't authenticate us some times:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

So we need to get new tokens and put them in kubeconfig file.

Is it because signingKeys were rotated before idTokens were re-issued, so dex didn't know how to decrypt the tokens?

[ericchiang]

@zihaoyu we've been seeing this issue crop up recently.

Dex's logic will keep old signing keys around as long as there are id tokens that it could have signed. There will no longer be used to sign, but will be kept around to verify. I don't think this is the source of the issue, but yes, we've seen this response from kubectl a bit lately.

What version of dex are you using? What logins (LDAP, GitHub, etc.)?

Anecdotally we've seen these bugs reported less with our recent release v2.2.3. Could you try that?

[zihaoyu]

@ericchiang We use GitHub connector, and quay.io/coreos/dex:v2.0.0-beta.1 image. Will try v2.2.3. Thanks.

[joshrosso]

Experiencing an identical behavior with:

Dex: 2.1.0
Connector: LDAP

We're working to validate 2.2.3 and will let you know after 24 hours 😄

[ericchiang]

I think I can see an issue https://github.com/coreos/dex/blob/3797a71ec9b5cbd254908f9d509a9dd21d0cfb96/server/rotation.go#L135

[ericchiang]

Yep, the expiry bug is causing verification keys to arbitrarily be deleted. I think this is causing the issue. Will send a fix ASAP.

[ericchiang]

Okay I was finally able to reproduce this

$ kubectl get pods
I0301 13:16:35.917132    1893 logs.go:41] RoundTripper returned a response & error; ignoring response
Unable to connect to the server: No valid id-token, and cannot refresh without refresh-token

[joshrosso]

Thanks @ericchiang.

So #829 fixes the above message?

[ericchiang]

@joshrosso From my testing, yes it looks like #829 fixes the issue. Observed behavior of v2.2.3 is: after two rotations the validation key is prematurely marked as expired (after ~12 hours by default). During this time the ID Token hasn't expired, so kubectl doesn't trigger a refresh, but the public key is no longer returned by dex, so the API server marks it as invalid.

Going to do a v2.2.4 release with #829 cc @rithujohn191

[joshrosso]

Thanks. We're happy to validate this once the v2.2.4 drops.

[zihaoyu]

Same here. Thanks for the fast turnaround!

[ericchiang]

v2.2.4 is out. Let me know if you can login in 12 hours :)

[joshrosso]

In flight, will let you know tomorrow.

[zihaoyu]

@ericchiang This morning, some of us reported successful kubectl operations without needing to update tokens manually. However some saw another error:

Unable to connect to the server: could not refresh token: server_error

Looking at dex logs:

2017-03-02T13:36:50.906120876Z time="2017-03-02T13:36:50Z" level=error msg="failed to refresh identity: no upstream access token found" 

Any ideas?

We are running v2.2.4.

[ericchiang]

@zihaoyu sounds like #767

Did you do your initial login before or after you bumped to v2.2.4?

[zihaoyu]

@ericchiang Before the bump

[zihaoyu]

@ericchiang v2.2.4 seems to be working fine. After a weekend we didn't have to get new tokens manually.

[InAnimaTe]

For anyone running Tectonic and running into this, 1.5.2-tectonic.1 is running 2.1.0 of Dex which means you are affected. Looks like 1.5.3 release upgrades Dex to 2.2.4 in which case things should work.

Thanks to everyone for fixing this!