Kibana sometimes sends HTTP requests to Elasticsearch without credentials

[sscarduzio]

Kibana version: 5.1.1

Elasticsearch version: 5.1.1

Server OS version:OSX 10.11.6

Browser version: Chrome 54

Browser OS version: OSX 10.11.6

Original install method (e.g. download page, yum, from source, etc.): unzipped the tarball

Description of the problem including expected versus actual behavior: Kibana server sends some HTTP request without authorization header.

Steps to reproduce:

1. Have an Elasticsearch v5.1.1 instance running allowing only requests with HTTP Basic Auth credentials.
2. Configure said credentials in kibana.yml as "elasticsearch.username" and "elasticsearch.password".
3. Wait for ES to reach green status, and Kibana to find Elasticsearch.
4. Create some dummy data points

curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic bG9nc3Rhc2g6bG9nc3Rhc2g=" -H "Cache-Control: no-cache"  -d '{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:28.998Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.267Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.267Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.267Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.268Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}
{"index":{"_id":null,"_index":"logstash-2016.10.28","_type":"logs","_routing":null}}
{"message":"","@version":"1","@timestamp":"2016-10-28T09:15:29.269Z","host":"hilbert.local"}' "http://localhost:9200/_bulk"

5. Switch on Wireshark.
6. Start exploring and searching with Kibana
7. Observe in wireshark that Kibana skips sending the credentials for some requests, like in this screenshot:

See how Authorization header is not present!**

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

 error  [09:12:03.308]  Error: Authentication Exception
    at respond (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:289:15)
    at checkRespForFailure (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/src/lib/transport.js:248:7)
    at HttpConnector.<anonymous> (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:164:7)
    at IncomingMessage.wrapper (/me/tmp/kibana-5.1.1-darwin-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4994:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

[jbudz]

The .kibana/config requests without auth will be fixed by #9442 in the next release. Are you seeing this for more endpoints? Looking into it too.

[sscarduzio]

This is the set of actions I had to explicitly allow to be requested without authentication (for any index) to make Kibana work

      actions: ["indices:data/read/field_stats", "indices:admin/mappings/fields/get", "indices:admin/get", "indices:data/read/msearch"]

[sscarduzio]

@jbudz if you have Kibana builds to make me test, I have all set up on the ES side to test and I'm more than happy to help out!

[jbudz]

@sscarduzio awesome, thanks. We have snapshots here.

[sscarduzio]

Nice, is there already something I can test about this bug in the 5.1.2 snapshot?

[jbudz]

Yeah, #9446 should fix the auth requests to .kibana/config

[sscarduzio]

Hello! I noticed this is labelled as "neeeds details", do you need more details?

[epixa]

These endpoints were fixed in 5.1.2.

[sscarduzio]

@epixa Kibana with x-pack installed has this problem again, cannot load the /app/monitoring (Marvel).

I see a POST /.monitoring-data-2/_count (with no body) coming in without valid Authorization header.

[sscarduzio]

Oh and when I click on Timelion, I see GET /.kibana/config/5.2.0, again without valid Authorization. Although this is not fatal to the browser's user experience.

[epixa]

@sscarduzio Can you give some steps to reproduce the issue you're having with monitoring? Monitoring is loading without issue for me, and requests are properly authorized.

There are circumstances where Kibana will attempt to query for the config prior to authorization being available, and in those cases it will gracefully handle the authorization error that is returned from Elasticsearch, which is probably what you saw in the latter case.

[sscarduzio]

Sure, here is a step by step how to reproduce this with screenshots.

1. Download fresh ES and run it as is.
2. Download Kibana, install xpack plugin an use this configuration:

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: false
elasticsearch.username: "kibana"
elasticsearch.password: "kibanapassword"

3. Fire up wireshark
4. run Kibana, navigate to monitoring app
5. Search Wireshark for string ".monitoring-data-2/_count"
6. Follow HTTP traffic in Wireshark, note absence of Authorization header in the POST request.

7. Take another sample of the many other kibana HTTP connections you see in the same capture in Wireshark and see the presence of Authorization headers.

[gitime]

@epixa the issue still exists with elasticstack 5.2.0, why closed?

[gitime]

@sscarduzio why the elastic team taking so much time to resolve and fix this important issue?

[epixa]

@sscarduzio To clarify, the requests that you listed in your last comment that do not include an authorization header do get rejected by Elasticsearch as unauthorized, correct?

[epixa]

@gitime We're still trying to figure out what the issue even is. We're having no authorization issues in any of our testing, neither in our manual testing nor in our automated testing. As a Kibana end-user, how is this issue affecting you?

[epixa]

I think I've reproduced the issue, but I'd appreciate it if either of you could confirm on your end.

When security is enabled on Elasticsearch and disabled on Kibana, Monitoring fails to work due to access denied errors by default. The search profiler (via dev tools) also fails (as do reporting requests behind the scenes, but that's disabled for this reproduction). In both cases, the authorization headers are not being sent, which is causing the failed requests. If you go to discover or something like that, the requests work and have the authorization header.

Now this is where it gets really strange: if you go to discover and then generate a short URL via the "share" menu, you'll see that that request does properly have an authorization header, and the short url is created. Now, if you go to search profiler and execute a search, the authorization header is sent and the request works this time. Also at this time, if you manually navigate to /app/monitoring (to make sure you're bypassing the access denied page), monitoring works and all the requests are sending the authorization header.

Edit: And taking that one step further, once the authorization header starts sending for xpack requests, even refreshing the page doesn't stop the requests from working. Starting a new incognito window will get us back into a failing state though.

[sscarduzio]

@epixa I really appreciate your time for digging into this, and man, I know browsers' "magic" in handling HTTP basic auth is a source of major confusion while dealing with this kind of issues.
:)

I just reproduced the additional instance of unauthenticated request when you click on dev tools:

GET /_aliases
GET /_mapping

The way I've been detecting these issues

In hope it will be useful, I can share my initial testing setup (before inspecting with Wireshark). I believe it offers a great framework for pin-pointing rogue unauthenticated requests.

My original test involved installing readonlyrest plugin and configuring it to accept only requests carrying HTTP basic auth credentials (user=kibana, pass=kibana).
This would be my elasticsearch.yml

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false
readonlyrest:
  enable: true
  access_control_rules:
    - name: Kibana
      auth_key: kibana:kibanapassword
      type: allow

I then set up Kibana to use those credentials too

xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: false
elasticsearch.username: "kibana"
elasticsearch.password: "kibanapassword"

Then I opened my browser and gave the same credentials from the browser too.

At this stage I knew:

• if the browser fails to send credentials, I'll see a login prompt (after receiving a HTTP 401).
• If Kibana daemon fails to send credentials, I'd see an HTML error banner in the browser and/or a red line in the readonlyrest logs stating the request type, action, involved indices, headers and path.

Tell me if I can help any further.

_Simone

[epixa]

@sscarduzio Thanks for all of that info and your time so far in debugging this. We've figured it out. The problem is that certain requests in Kibana and X-Pack are not returning a 401 when Elasticsearch does. Browsers rely on that 401 error for their basic auth process, even when basic auth has already been completed once on that domain. The thing is, once basic auth credentials for a given realm are stored in the browser, the browser will automatically resend newly-authorized requests that 401 with the same realm that is stored, but browser dev tools merge these requests together and treat them as a single thing. This is convenient in most cases, but it made it particularly hard to track down an issue like this without a lower level tool like wireshark.

One thing that's worth mentioning again is that this is not a security vulnerability since the requests are being rejected as they should be, but it is definitely not the behavior we want.

To help illustrate the problem for anyone that still doesn't follow, let's talk about what is happening at the request level in the Really Strange™ behavior I described in a previous comment.

1. The initial request to /app/kibana 401s with the realm "security", and no stored credentials exist for that realm, so it prompts for a username and password, which we input. This 401 error appears in the dev console because it was specifically prompted for input. The browser will now send the authorization header on all requests to /app/*, since /app/ is the base path of this request that was just authenticated, which is why you can navigate between the various applications in Kibana without getting 401 errors.
2. Requests to monitoring result in 403 errors instead of 401 errors so the browser does not attempt to authorize them at all.
3. Attempting to use the search profiler results in a 200 request with the error information embedded into the response body, so the browser doesn't attempt to authorize it.
4. The short URL service makes a request to /shorten, which 401s with the realm "security". Since credentials for this realm were already stored, the same request is repeated with the authorization header. In the browser dev tools, this appears as a single successful request and includes the authorization headers, but a lower level tool like wireshark will show otherwise. Like in the first step, the browser will now send the authorization header to any request matching the base path of this request, which just happens to be / for this particular endpoint. This effectively means that all requests to Kibana will now include the authorization header by default, which is why everything begins to work at that point onward.

The solution here is to go through all authenticated endpoints in Kibana and X-Pack to make sure they are properly returning 401 errors for missing credentials. We're tracking this effort in an internal ticket, but I'm going to open this one so we have a public place to update on status and to which we can link pull requests.

[epixa]

/cc @spalger @ycombinator

[nan008]

@epixa any news on this problem?

[euprogramador]

Idem, receive same error

[euprogramador]

works,

added readonlyrest one rule:

• name: "public access monitoring"
  actions: ["indices:data/read/search"]
  indices: [".monitoring*"]

Kibana require auth, monitoring (x-pack), works.

wow!

[burandobata]

+1

[kobelb]

We're actively looking into a solution to this problem that will prevent us from chasing down end-points that aren't handling the WWW-Authenticate headers properly on a 401. Thanks for bringing this to our attention, and your patience. I'll update this issue as we know more.

[julienyu]

+1

[nsbk]

We just stumbled upon this. So far we are using the elasticsearch.customHeaders field on kibana.yml to make sure that auth is always present on the request.

elasticsearch.customHeaders: { Authorization: Basic base64auth }

Being base64auth base64 encoded username:password

[kobelb]

#17725 just merged, so the index patterns api will now be handling the WWW-Authenticate header properly on 401s.

A more holistic approach is explored in this POC PR: #17724

[parallelthought]

This behaviour is observed in Elastic v6.4.2 as well (especially for calls made to update monitoring index by x-pack plugin). The impact is more for use cases where an OSS plugin is used for security on top of ES and basic auth needs is enabled for Kibana.

[ygersie]

I'm using the following hack with nginx as a proxy in front of Kibana to trick the browser to store credentials for the entire hostname:

    location = / {
      # Hack to workaround https://github.com/elastic/kibana/issues/9583
      if ($http_authorization !~* "^Basic") {
        rewrite .* /app/kibana break;
      }
      proxy_pass http://localhost:5601;
    }

    location / {
      if ($http_authorization !~* "^Basic") {
        return 302 https://$host;
      }
      proxy_pass http://localhost:5601;
    }

If a client does not send an Authorization header I force a proxy_pass towards /app/kibana which will prompt for auth. After the credentials are stored with the correct uri (/) all subsequent requests will contain the authorization header.

[sscarduzio]

News about this
https://forum.readonlyrest.com/t/provide-kibana-login-page-for-ror-oss-version/1441

[legrego]

This has been resolved via #88850 (shipped in 7.11.0)