apm_user role has no access to observability-annotations index

[clyfish]

Kibana version:
7.8.0

Elasticsearch version:
7.8.0

Server OS version:
CentOS 7

Browser version:
Chrome 83

Browser OS version:
Win 10

Original install method (e.g. download page, yum, from source, etc.):
helm

Describe the bug:
User with apm_user role and kibana_admin role has no access to observability-annotations index, so the /api/apm/services/xxx/annotation/search api returns 500.

Please add observability-annotations index access to apm_user role, or change xpack.observability.annotations.index's default value to apm-observability-annotations.

Steps to reproduce:

1. Create a user with apm_user role and kibana_admin role
2. Access Kibana / APM / Services / xxx / Transactions
3. The /api/apm/services/xxx/annotation/search api returns 500

Expected behavior:
The /api/apm/services/xxx/annotation/search api returns 200

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
{"type":"log","@timestamp":"2020-06-22T13:42:42Z","tags":["error","http"],"pid":6,"message":"{ Error: [security_exception] action [indices:data/read/search] is unauthorized for user [x]\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 403,\n displayName: 'AuthorizationException',\n message:\n '[security_exception] action [indices:data/read/search] is unauthorized for user [x]',\n path: '/observability-annotations/_search',\n query: {},\n body:\n { error:\n { root_cause: [Array],\n type: 'security_exception',\n reason:\n 'action [indices:data/read/search] is unauthorized for user [x]' },\n status: 403 },\n statusCode: 403,\n response:\n '{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/read/search] is unauthorized for user [x]"}],"type":"security_exception","reason":"action [indices:data/read/search] is unauthorized for user [x]"},"status":403}',\n toString: [Function],\n toJSON: [Function] }"}
{"type":"error","@timestamp":"2020-06-22T13:42:42Z","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?start=2020-06-21T13%3A38%3A39.696Z&end=2020-06-22T13%3A38%3A39.697Z","query":{"start":"2020-06-21T13:38:39.696Z","end":"2020-06-22T13:38:39.697Z"},"pathname":"/api/apm/services/xxx/annotation/search","path":"/api/apm/services/xxx/annotation/search?start=2020-06-21T13%3A38%3A39.696Z&end=2020-06-22T13%3A38%3A39.697Z","href":"/api/apm/services/xxx/annotation/search?start=2020-06-21T13%3A38%3A39.696Z&end=2020-06-22T13%3A38%3A39.697Z"},"message":"Internal Server Error"}
{"type":"response","@timestamp":"2020-06-22T13:42:42Z","tags":["access:apm"],"pid":6,"method":"get","statusCode":500,"req":{"url":"/api/apm/services/xxx/annotation/search?start=2020-06-21T13%3A38%3A39.696Z&end=2020-06-22T13%3A38%3A39.697Z","method":"get","headers":{"host":"kibana","x-request-id":"8fad8e17181e2414021e8b0218d9f37d","x-real-ip":"10.x.x.x","x-forwarded-for":"10.x.x.x","x-forwarded-host":"kibana","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,zh-TW;q=0.6"},"remoteAddress":"10.x.x.x","userAgent":"10.x.x.x"},"res":{"statusCode":500,"responseTime":183,"contentLength":9},"message":"GET /api/apm/services/xxx/annotation/search?start=2020-06-21T13%3A38%3A39.696Z&end=2020-06-22T13%3A38%3A39.697Z 500 183ms - 9.0B"}

Any additional context:

[clyfish]

Workaround:
Create a role apm_user_extra with read and view_index_metadata privileges to observability-annotations index and add user to apm_user_extra role, or set kibana config xpack.observability.annotations.index to apm-observability-annotations

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

Can confirm. Logging in with apm_read_user (custom user in APM testing process) I get the same response

Summarising my thoughts (similar to what's already been said above):
By default the observability index is observability-annotations. Since it is not a system index we should not query it with the internal user but instead as the current user. apm_user should therefore be granted access to observability-annotations.

Questions:

• Did we already discuss this and decided no? If so, why?
• This would have been caught by logging in as apm_read_user. Can we improve the testing process for new features that rely on custom indices?

[dgieselaar]

@sqren:

Since it is not a system index we should not query it with the internal user but instead as the current user.

This should already be the case - did you see something different?

WRT adding those permissions to the apm_user role: I'm a bit on the fence if we should do this, as the name is configurable, and permissions need to be set anyway when using the API (see https://www.elastic.co/guide/en/kibana/current/apm-app-api-user.html). I'm not sure if we have any documentation around using it in the UI however (if we don't, we probably should, I'll check tomorrow with Brandon).

[dgieselaar]

@clyfish have you indexed any annotations through the API by any chance? or does this always fail?

[sorenlouv]

Since it is not a system index we should not query it with the internal user but instead as the current user.

This should already be the case - did you see something different?

No, just wanted to emphasize that we don't want to use the internal user in this case like we are doing in other places (agent configuration for instance).

[dgieselaar]

FWIW, I just checked a 7.8 cluster without annotations and do not see a request failure or a toast. From what I can tell from the code, it should only fail with a 500 when the index exists AND the current user does not have permissions, which only happens when annotations have been indexed via the API (by another user) or a user has created the index manually.

[sorenlouv]

I'm a bit on the fence if we should do this, as the name is configurable,

I think this is similar to the apm-* indices which are also configurable. We grant access to the apm_user anyway to provide a better OOTB experience.

and permissions need to be set anyway when using the API

Good point. OTOH they might simply use the superuser (elastic) for data ingestion and then expect it to work for users with read access to APM.
This is also similar to how ml works: you need extra privileges to write to ml indices but the apm_user has access to read them by default.

Regardless, I don't think it's great to show a 500 error toast to users who don't have access.

[dgieselaar]

Regardless, I don't think it's great to show a 500 error toast to users who don't have access.

Agreed, this sucks, and the error message is not actionable at all.

[clyfish]

@clyfish have you indexed any annotations through the API by any chance?

@dgieselaar
No.
There is no observability-annotations index in my cluster, but there are derived annotations in apm-* indices.

or does this always fail?

Yes

[dgieselaar]

Thanks! Disregard what I wrote before about the 7.8 cluster, I have superuser credentials there 🤦‍♂️. Op di 23 jun. 2020 04:19 schreef clyfish <[email protected]>:

…

@clyfish <https://github.com/clyfish> have you indexed any annotations through the API by any chance? No. There is no observability-annotations index in my cluster, but there are derived annotations in apm-* indices. or does this always fail? Yes — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#69642 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACWDXBBY4IMYHQCBAE7FYDRYAGL3ANCNFSM4OEX4J3A> .

[bmorelli25]

I'm not sure if we have any documentation around using it in the UI however (if we don't, we probably should, I'll check tomorrow with Brandon).

Right now we only have documentation around the permissions needed for the annotations API. I can definitely add docs for the permissions required for derived annotations. Is read and view_index_metadata privileges for observability-annotations correct?

[dgieselaar]

@bmorelli25 Yeah, that would be correct. I've opened a PR to add those to the apm_user role by default, but we should at least document that those permissions would need to be added if the user changes the configured index name for observability.annotations.index.

[bmorelli25]

Docs have been updated in #70265.