Authorization header not being sent on x-pack requests by default

[elasticmachine]

Superseded by #9583

Original comment by @epixa:

Kibana version: 5.2.1 (and probably all version through master)

Elasticsearch version: 5.2.1

Server OS version: darwin

Browser version: Chrome 56.0.2924.87

Browser OS version: darwin

Original Kibana install method (e.g. download page, yum, from source, etc.): download page

Original X-Pack install method (e.g. by name, offline install, from source, etc.): by name

Description of the problem including expected versus actual behavior:

Steps to reproduce:

1. Disable security on Kibana only
2. You can disable reporting as well to avoid network spam
3. Then follow along on these weird steps:

Monitoring fails to work due to access denied errors by default. The search profiler (via dev tools) also fails (as do reporting requests behind the scenes, but that's disabled for this reproduction). In both cases, the authorization headers are not being sent, which is causing the failed requests. If you go to discover or something like that, the requests work and have the authorization header.

Now this is where it gets really strange: if you go to discover and then generate a short URL via the "share" menu, you'll see that that request does properly have an authorization header, and the short url is created. Now, if you go to search profiler and execute a search, the authorization header is sent and the request works this time. Also at this time, if you manually navigate to /app/monitoring (to make sure you're bypassing the access denied page), monitoring works and all the requests are sending the authorization header.

Taking that one step further, once the authorization header starts sending for xpack requests, even refreshing the page doesn't stop the requests from working. Starting a new incognito window will get us back into a failing state though.

[elasticmachine]

Original comment by @epixa:

/cc @ycombinator @spalger

[elasticmachine]

Original comment by @epixa:

This was originally reported in comments on #9583

[elasticmachine]

Original comment by @ycombinator:

I'm able to reproduce this on master.

[elasticmachine]

Original comment by @ycombinator:

For clarity: the original issue LINK REDACTED) mentions using Wireshark to inspect requests between the Kibana server and Elasticsearch. This isn't strictly necessary, IMO, because the problem seems to start with Kibana client (browser) itself. Sometimes the Authorization header is being set in XHR requests and other times not.

Authorization header sent from Discover

Authorization header not sent from Monitoring

[elasticmachine]

Original comment by @ycombinator:

Also noticed that the Authorization header is NOT being sent by Console requests:

So I'm not sure this is an x-pack vs. not issue.

[elasticmachine]

Original comment by @epixa:

X-Pack server.route() calls that need to be audited to ensure they return a 401 with the appropriate realms:

graph

• ./plugins/graph/server/routes/graphExplore.js:17
• ./plugins/graph/server/routes/searchProxy.js:3

ml

• ./plugins/ml/server/routes/anomaly_detectors.js:7
• ./plugins/ml/server/routes/anomaly_detectors.js:20
• ./plugins/ml/server/routes/anomaly_detectors.js:34
• ./plugins/ml/server/routes/anomaly_detectors.js:47
• ./plugins/ml/server/routes/anomaly_detectors.js:61
• ./plugins/ml/server/routes/anomaly_detectors.js:76
• ./plugins/ml/server/routes/anomaly_detectors.js:90
• ./plugins/ml/server/routes/anomaly_detectors.js:104
• ./plugins/ml/server/routes/anomaly_detectors.js:118
• ./plugins/ml/server/routes/anomaly_detectors.js:132
• ./plugins/ml/server/routes/datafeeds.js:7
• ./plugins/ml/server/routes/datafeeds.js:20
• ./plugins/ml/server/routes/datafeeds.js:34
• ./plugins/ml/server/routes/datafeeds.js:47
• ./plugins/ml/server/routes/datafeeds.js:62
• ./plugins/ml/server/routes/datafeeds.js:76
• ./plugins/ml/server/routes/datafeeds.js:92

monitoring

• ./plugins/monitoring/server/routes/api/v1/clusters.js:105
• ./plugins/monitoring/server/routes/api/v1/clusters.js:131
• ./plugins/monitoring/server/routes/api/v1/clusters.js:157
• ./plugins/monitoring/server/routes/api/v1/clusters.js:198
• ./plugins/monitoring/server/routes/api/v1/clusters.js:221
• ./plugins/monitoring/server/routes/api/v1/indices.js:19
• ./plugins/monitoring/server/routes/api/v1/indices.js:79
• ./plugins/monitoring/server/routes/api/v1/kibana.js:27
• ./plugins/monitoring/server/routes/api/v1/kibana.js:64
• ./plugins/monitoring/server/routes/api/v1/logstash/node.js:25
• ./plugins/monitoring/server/routes/api/v1/logstash/nodes.js:25
• ./plugins/monitoring/server/routes/api/v1/logstash/overview.js:25
• ./plugins/monitoring/server/routes/api/v1/nodes.js:32
• ./plugins/monitoring/server/routes/api/v1/nodes.js:107
• ./plugins/monitoring/server/routes/api/v1/phone_home.js:9

reporting

• ./plugins/reporting/server/routes/jobs.js:18
• ./plugins/reporting/server/routes/jobs.js:34
• ./plugins/reporting/server/routes/jobs.js:50
• ./plugins/reporting/server/routes/jobs.js:63
• ./plugins/reporting/server/routes/jobs.js:81
• ./plugins/reporting/server/routes/main.js:27
• ./plugins/reporting/server/routes/main.js:39
• ./plugins/reporting/server/routes/main.js:46
• ./plugins/reporting/server/routes/main.js:53

searchprofiler

• ./plugins/searchprofiler/server/routes/profile.js:5

security

• ./plugins/security/server/routes/api/v1/authenticate.js:12
• ./plugins/security/server/routes/api/v1/authenticate.js:42
• ./plugins/security/server/routes/api/v1/authenticate.js:54
• ./plugins/security/server/routes/api/v1/indices.js:8
• ./plugins/security/server/routes/api/v1/roles.js:12
• ./plugins/security/server/routes/api/v1/roles.js:29
• ./plugins/security/server/routes/api/v1/roles.js:46
• ./plugins/security/server/routes/api/v1/roles.js:64
• ./plugins/security/server/routes/api/v1/users.js:18
• ./plugins/security/server/routes/api/v1/users.js:32
• ./plugins/security/server/routes/api/v1/users.js:49
• ./plugins/security/server/routes/api/v1/users.js:67
• ./plugins/security/server/routes/api/v1/users.js:81
• ./plugins/security/server/routes/views/login.js:8
• ./plugins/security/server/routes/views/logout.js:4

xpack_main

• ./plugins/xpack_main/server/routes/api/v1/xpack_info.js:8

[elasticmachine]

Original comment by @epixa:

Kibana OSS API routes that need to be audited to ensure they properly return 401s with the necessary realm:

cli

• ./src/cli/cluster/base_path_proxy.js:58
• ./src/cli/cluster/base_path_proxy.js:66
• ./src/cli/cluster/base_path_proxy.js:109

console

• ./src/core_plugins/console/index.js:134
• ./src/core_plugins/console/index.js:147
• ./src/core_plugins/console/index.js:155
• ./src/core_plugins/console/index.js:171

elasticsearch

• ./src/core_plugins/elasticsearch/lib/create_proxy.js:51

kibana

• ./src/core_plugins/kibana/server/routes/api/ingest/register_field_capabilities.js:5
• ./src/core_plugins/kibana/server/routes/api/scripts/register_languages.js:5
• ./src/core_plugins/kibana/server/routes/api/search/count/register_count.js:5
• ./src/core_plugins/kibana/server/routes/api/settings/register_delete.js:4
• ./src/core_plugins/kibana/server/routes/api/settings/register_get.js:4
• ./src/core_plugins/kibana/server/routes/api/settings/register_set.js:4
• ./src/core_plugins/kibana/server/routes/api/settings/register_set_many.js:4

timelion

• ./src/core_plugins/timelion/server/routes/functions.js:4
• ./src/core_plugins/timelion/server/routes/run.js:13
• ./src/core_plugins/timelion/server/routes/validate_es.js:2

server/http

• ./src/server/http/index.js:88
• ./src/server/http/index.js:99
• ./src/server/http/index.js:116
• ./src/server/http/index.js:130

server/status

• ./src/server/status/index.js:15
• ./src/server/status/index.js:45

ui

• ./src/ui/index.js:47

[elasticmachine]

Original comment by @ycombinator:

Plan after Zooming with @tsullivan and @epixa:

1. Continue auditing endpoints listed by Court in previous two comments. Each endpoint should return a Boom.unauthorized() if it gets a 401 from Elasticsearch. Two notable classes of exceptions to this rule:

   1. Monitoring is a bit of a special case because it might be talking to a production cluster and a monitoring cluster. @tsullivan is going to put up a PR for conditionally responding with a 401, if the error is coming from a production cluster.

   2. For system APIs such as Reporting's list_completed_since API that follow users around Kibana outside of a Reporting context, we don't want to kick user's out to the login page if such an API returns a 401. So in such cases, the API error handler should swallow the 401 from Elasticsearch.

[elasticmachine]

Original comment by @epixa:

I've confirmed this behavior has been around since at least 5.1, so I've removed the blocker label on this for 5.3. We should get ahead of this problem and get this fixed up, though.

[elasticmachine]

Original comment by @skearns64:

This seems likely to be related to LINK REDACTED

[elasticmachine]

Original comment by @epixa:

@skearns64 Yep, that's the same issue.

[legrego]

Performing some long overdue cleanup. This used to be an internal ticket (back when x-pack was closed source) to track the work on #9583. Now that this is no longer the case, I am closing this one in favor of #9583