JWKS URL changes aren't reflected immediately

[lenaing]

Hello,

I encounter the following issue:

When the JWKS URL of an JWT Auth is updated in an high availability setup, the new JWKS URL is called by Vault at configuration time for validation but the old URL is still in use until the Vault switches of master.

Setup : Non-enterprise Vault version 1.5.2 (Also tested on 1.4.2), 3 servers, PostgreSQL HA and Storage Backend

How to reproduce:

• I spun up a test Vault HA Cluster with two small Python webservers (lets call them URL A & B) with both the same JWKSet with expired JWKeys. This force Vault to request JWKS each time and, I hoped, to avoid key caching.

• I configure the JWT endpoint in Vault through the UI with URL A.

• Vault requests URL A for initial configuration validation.

• Then I start to auth with a sample JWT against this Vault like this:

vault write auth/jwt/login role=test-role jwt="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"

• Vault always requests URL A upon each authent request (so no cache ?, weird, sometime it seems more than what is specified here ).

• At this point, I change the JWT endpoint's JWKS URL to URL B.

• Vault requests URL B for configuration validation.

• When I try again to authenticate against Vault, it still requests URL A, even after 10 minutes.

• Then I issue a vault operator step-down to change the Vault master.

• Upon new authent request, Vault now calls correctly URL B.

URL A logs:

xx.xx.xx.97 - - [26/Aug/2020 10:37:26] "GET /test-jwks.json HTTP/1.1" 200 - Authent request
xx.xx.xx.97 - - [26/Aug/2020 10:39:12] "GET /test-jwks.json HTTP/1.1" 200 - Authent request
xx.xx.xx.97 - - [26/Aug/2020 10:40:57] "GET /test-jwks.json HTTP/1.1" 200 - Authent request
xx.xx.xx.97 - - [26/Aug/2020 10:42:29] "GET /test-jwks.json HTTP/1.1" 200 - Authent request after URL A configuration
xx.xx.xx.97 - - [26/Aug/2020 10:44:04] "GET /test-jwks.json HTTP/1.1" 200 - Authent request after URL A configuration

URL B logs:

$ python -m SimpleHTTPServer 8080
Serving HTTP on 0.0.0.0 port 8080 ...
xx.xx.xx.97 - - [26/Aug/2020 10:42:11] "GET /test-jwks.json HTTP/1.1" 200 - Configuration validation
xx.xx.xx.98 - - [26/Aug/2020 10:44:21] "GET /test-jwks.json HTTP/1.1" 200 - Authent request after Vault Step Down (Notice the Vault server IP changed)
xx.xx.xx.98 - - [26/Aug/2020 10:44:50] "GET /test-jwks.json HTTP/1.1" 200 - Authent request
xx.xx.xx.98 - - [26/Aug/2020 10:46:07] "GET /test-jwks.json HTTP/1.1" 200 - Authent request

Behavior I expected:

Once the JWKS URL is updated, all requests should be made against immediately to the new URL. Not after a switch of Vault master :)

Kind regards,