kibana_system role missing read permissions for metrics-apm-* and logs-apm-* #170031

uvNikita · 2023-10-27T13:34:49Z

Kibana version:
8.10.2

Elasticsearch version:
8.10.2

Original install method (e.g. download page, yum, from source, etc.):
ECK

Describe the bug:

kibana_system role currently has read permissions for traces-apm-*, but missing for logs-apm-* and metrics-apm-*.

APM application in Kibana is configured following these steps: https://www.elastic.co/guide/en/kibana/current/apm-spaces.html

This results in the following error in "Observability -> APM -> Settings -> Agent Configuration" tab: "The list of agent configurations could not be fetched. Your user may not have the sufficient permissions"

Browser Network tab shows the following failing request:

GET https://<kibana-url>/s/<kibana-space>/api/apm/settings/agent-configuration
{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "security_exception\n\tRoot causes:\n\t\tsecurity_exception: action [indices:data/read/search] is unauthorized for service account [elastic/kibana] on indices [metrics-apm-<kibana-space>], this action is granted by the index privileges [read,all]",
  "attributes": {
    "data": {},
    "_inspect": []
  }
}

Steps to reproduce:

Configure APM following these steps: https://www.elastic.co/guide/en/kibana/current/apm-spaces.html
Navigate to "Observability -> APM -> Settings -> Agent Configuration"
See permission Error

Alternatively, just check for missing permissions by sending GET _security/role/kibana_system

Expected behavior:
Should be able to see and create APM agent configuration.

Errors in browser console (if relevant):

GET https://<kibana-url>/s/<kibana-space>/api/apm/settings/agent-configuration
{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "security_exception\n\tRoot causes:\n\t\tsecurity_exception: action [indices:data/read/search] is unauthorized for service account [elastic/kibana] on indices [metrics-apm-<kibana-space>], this action is granted by the index privileges [read,all]",
  "attributes": {
    "data": {},
    "_inspect": []
  }
}

Any additional context:
See also Case #01503117

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-10-27T15:51:25Z

Pinging @elastic/apm-ui (Team:APM)

sorenlouv · 2023-10-27T22:50:29Z

It is very surprising to me that the route GET /api/apm/settings/agent-configuration would attempt to access any other index than .apm-agent-configuration. If it does it sounds like a bug (or a side effect I haven't considered).

Either way thanks for bringing this to our attention, and thanks for opening a PR!

uvNikita · 2023-10-31T09:49:59Z

@sqren So I tried to follow the trace where access to other indexes than .apm-agent-configuration would be made and I think you are right, at it's unnecessary.

Keep in mind that it's my first look at kibana source code, so probably someone else should double check this findings.

What I found so far:

/api/apm/settings/agent-configuration is defined here:

kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration/route.ts

Line 69 in 6327ab3

const getSingleAgentConfigurationRoute = createApmServerRoute({
it then proceeds with getting all configured APM indexes:

kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration/route.ts

Line 81 in 6327ab3

const apmIndices = await resources.getApmIndices();
it then creates internalESClient with those indexes and calls findExactConfiguration with this new client
findExactConfiguration is defined here:

kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration/find_exact_configuration.ts

Line 19 in 6327ab3

export async function findExactConfiguration({
it send query with index set to APM_AGENT_CONFIGURATION_INDEX

kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration/find_exact_configuration.ts

Line 35 in 6327ab3

index: APM_AGENT_CONFIGURATION_INDEX,
this index name is statically defined here:

kibana/x-pack/plugins/apm/server/routes/settings/apm_indices/apm_system_index_constants.ts

Line 8 in 0f3382f

export const APM_AGENT_CONFIGURATION_INDEX = '.apm-agent-configuration';

So it doesn't seem to be necessary to create internalESClient with all APM configured indexes since they are not used anyway. My guess is the error will be gone if removing indexes from the call here (I haven't checked it):

kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration/route.ts

Line 83 in 6327ab3

const internalESClient = await createInternalESClientWithContext({

Looking at this code, it seems like there is also no way for us to see configuration only for services in the current kibana space (service environment). It would be a nice feature to make APM_AGENT_CONFIGURATION_INDEX configurable in the same way as other APM indexes are, but that's probably outside of the scope of this issue (?)

Let me know if there is anything else I can do. I can give it a shot at trying to remove those indexes from createInternalESClientWithContext call, but you probably would want someone with more experience with kibana code to do it anyway?

jeramysoucy · 2023-10-31T13:41:06Z

Thank you, @uvNikita! From the Kibana Security team's perspective, we would prefer resolving this issue without having to augment kibana_system index access privileges. We typically only make exceptions for these types of indices when access is needed to support telemetry purposes. In this case, it seems like access is required only as a side effect of the current implementation.

@sqren Could someone from APM investigate further and provide some guidance to @uvNikita?

sorenlouv · 2023-11-03T18:17:43Z

Thank you for the writeup @uvNikita. I think you are right, that apmIndices should be removed from createInternalESClientWithContext since it's just passed through and never used. I hope to get a PR up to fix this soon.

That being said, I couldn't reproduce the issue. Most likely because I'm using the default setup where metrics are written to metrics-apm.* and logs to metrics-apm.*. It looks like in your case, you are writing to metrics-apm-*. Is that intentional?

@elastic/apm-server Does above sound right, that by default we write to metrics-apm.* and not to metrics-apm-*?

uvNikita · 2023-11-06T13:30:30Z

@sqren Thanks for looking into it!

Yes, we are using <beat>-*-<kibana-space> template for all our indexes, including APM ones. It all works correctly except APM agent config, so would be nice to get rid of apmIndices since they are not needed anyway.

sorenlouv · 2023-11-07T23:52:38Z

@uvNikita It looks like we will need your PR and that this cannot be fixed in Kibana. In getConfigsAppliedToAgentsThroughFleet the metrics index (metrics-apm*) is queried. Since this may run in the background (and not within a request context), it'll run as the kibana system user, so this user must have access to the metric index.

kibana/x-pack/plugins/apm/server/routes/settings/agent_configuration/get_config_applied_to_agent_through_fleet.ts

Lines 13 to 18 in 34bdce9

    
           export async function getConfigsAppliedToAgentsThroughFleet( 
        
             internalESClient: APMInternalESClient 
        
           ) { 
        
             const params = { 
        
               index: internalESClient.apmIndices.metric, 
        
               size: 0,

uvNikita · 2023-11-08T09:22:44Z

@sqren Good find!

Related #170031. The internal Elasticsearch client in APM currently takes `apmIndices` as a required argument. This is due to legacy reasons and no longer needed and should therefore be removed. Co-authored-by: Katerina <[email protected]>

kpatticha · 2023-11-15T06:22:03Z

@sqren Thanks for looking into it!

Yes, we are using <beat>-*-<kibana-space> template for all our indexes, including APM ones. It all works correctly except APM agent config, so would be nice to get rid of apmIndices since they are not needed anyway.

The PR is merged #170733.

smith · 2023-12-01T15:09:45Z

Since #170733 is merged does that mean the PR to elasticsearch is still needed?

sorenlouv · 2023-12-04T11:47:09Z

Since #170733 is merged does that mean the PR to elasticsearch is still needed?

Yes. That PR turned out not to have an impact on this problem but the refactor was still an improvement. The original problem will either need to be fixed via changes to documentation, or a change to how agent configurations are retrieved.

The problem in a nut shell is that the internal kibana user is querying user configurable indices. This means that if the user changes the indices to something non-standard, we cannot the internal user will not have permissions to query these.

sorenlouv · 2023-12-11T09:01:01Z

Opened a PR for fixing this in #173001. This replaces elastic/elasticsearch#101467

…s metrics (#173001) Closes: #170031 Replaces: elastic/elasticsearch#101467 **Problem** We need to know if an agent config has been applied at the edge (by APM agents). This is determined by comparing the etag (hash) of the config, with the etag applied at the edges. Previously the agent config itself contained this information (`config.applied_by_agent`) but when running with fleet this will instead be captured in `agent_config` metric documents. Currently the internal kibana user retrieves the `agent_config` metric documents from the APM metric index (`metrics-apm-*` by default). This index is configurable by the end-user so can be changed to something the internal user doesn't have access to. This is a problem. **Solution** This PR replaces the calls made by the internal client with calls made by the authenticated end user (via `APMEventClient`). This approach works for requests made from the browser/UI but doesn't work for background tasks made by fleet. To work around this we only conditionally query the metric index if the `APMEventClient` is available. If `APMEventClient` is not available `applied_by_agent` will be `undefined`

…s metrics (elastic#173001) Closes: elastic#170031 Replaces: elastic/elasticsearch#101467 **Problem** We need to know if an agent config has been applied at the edge (by APM agents). This is determined by comparing the etag (hash) of the config, with the etag applied at the edges. Previously the agent config itself contained this information (`config.applied_by_agent`) but when running with fleet this will instead be captured in `agent_config` metric documents. Currently the internal kibana user retrieves the `agent_config` metric documents from the APM metric index (`metrics-apm-*` by default). This index is configurable by the end-user so can be changed to something the internal user doesn't have access to. This is a problem. **Solution** This PR replaces the calls made by the internal client with calls made by the authenticated end user (via `APMEventClient`). This approach works for requests made from the browser/UI but doesn't work for background tasks made by fleet. To work around this we only conditionally query the metric index if the `APMEventClient` is available. If `APMEventClient` is not available `applied_by_agent` will be `undefined` (cherry picked from commit 58c7958)

uvNikita added the bug Fixes for quality problems that affect the customer experience label Oct 27, 2023

botelastic bot added the needs-team Issues missing a team label label Oct 27, 2023

uvNikita mentioned this issue Oct 27, 2023

Add logs-apm-* and metric-apm-* read permissions for kibana_system role elastic/elasticsearch#101467

Closed

jsanz added the Team:APM All issues that need APM UI Team support label Oct 27, 2023

botelastic bot removed the needs-team Issues missing a team label label Oct 27, 2023

sorenlouv mentioned this issue Nov 7, 2023

[APM] Refactor internal es client #170733

Merged

sorenlouv mentioned this issue Dec 8, 2023

[APM] Remove usage of internal client when fetching agent config etags metrics #173001

Merged

sorenlouv closed this as completed in #173001 Dec 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kibana_system role missing read permissions for metrics-apm-* and logs-apm-* #170031

kibana_system role missing read permissions for metrics-apm-* and logs-apm-* #170031

uvNikita commented Oct 27, 2023

elasticmachine commented Oct 27, 2023

sorenlouv commented Oct 27, 2023

uvNikita commented Oct 31, 2023

jeramysoucy commented Oct 31, 2023

sorenlouv commented Nov 3, 2023 •

edited

Loading

uvNikita commented Nov 6, 2023

sorenlouv commented Nov 7, 2023 •

edited

Loading

uvNikita commented Nov 8, 2023

kpatticha commented Nov 15, 2023

smith commented Dec 1, 2023

sorenlouv commented Dec 4, 2023 •

edited

Loading

sorenlouv commented Dec 11, 2023

kibana_system role missing read permissions for metrics-apm-* and logs-apm-* #170031

kibana_system role missing read permissions for metrics-apm-* and logs-apm-* #170031

Comments

uvNikita commented Oct 27, 2023

elasticmachine commented Oct 27, 2023

sorenlouv commented Oct 27, 2023

uvNikita commented Oct 31, 2023

jeramysoucy commented Oct 31, 2023

sorenlouv commented Nov 3, 2023 • edited Loading

uvNikita commented Nov 6, 2023

sorenlouv commented Nov 7, 2023 • edited Loading

uvNikita commented Nov 8, 2023

kpatticha commented Nov 15, 2023

smith commented Dec 1, 2023

sorenlouv commented Dec 4, 2023 • edited Loading

sorenlouv commented Dec 11, 2023

sorenlouv commented Nov 3, 2023 •

edited

Loading

sorenlouv commented Nov 7, 2023 •

edited

Loading

sorenlouv commented Dec 4, 2023 •

edited

Loading