Enable Kibana Diagnostic to be Space Aware for Rules

[stefnestor]

Consolidating parallel conversations in this repository (#479, #514, #512, elastic/kibana#124382). The Kibana diagnostic is not currently space aware though Kibana (SIEM) Rules have been space delineated since at least 7.10.

This implicates that the Kibana diagnostic only captures (SIEM) Rule info from space:default & misses any/all other spaces. (You can determine this from an export because the Task Manager Health API will have scheduled Rule counts surpassing your exported Rule counts.)

SIEM appears more highly impacted because industry Security practices frequently suggest avoiding allocating users to a product's default space rather than purposely creating a new non-default space for them.

Example here: #514 (comment).

cc: @pmuellr @pickypg @CamiloSierraH @111andre111 @mikecote

[pmuellr]

Heh, was just looking at this yesterday.

Ideally, we could get the alerting connectors / rules directly from ES, and not Kibana. But I think the problem with that approach is the diagnostic tool either points to ES when getting ES diags, or Kibana when getting Kibana diags, so when running the Kibana diags, the ES url is not available.

I say "ideally", but honestly not sure - there may be some reason it would be preferable to get the data from Kibana. I think the ES versions will likely have "more stuff" than the Kibana versions, which seems like a good thing. @mikecote ?

If we want to continue down the path of just using the Kibana URL, we'll need to get all the spaces, then for each space, do what's currently being done with the default space.

At some point, connectors / rules will be shareable across spaces, so I guess if we continue down the path of using the Kibana APIs, we'd end up having dups of rules/connectors that are shared. Not a huge problem, but one we won't have if we query the rules right from ES.

[pickypg]

I say "ideally", but honestly not sure - there may be some reason it would be preferable to get the data from Kibana. I think the ES versions will likely have "more stuff" than the Kibana versions, which seems like a good thing. @mikecote ?

In general, the diagnostic has avoided reading from data indexes -- even system indexes -- to avoid scraping user data accidentally. Having an API in Kibana would be ideal for the diagnostic because then the diagnostic would not need to be changed for any unanticipated changes to the data structure (or indexes for that matter), and Kibana can be responsible for removing content we shouldn't have.

[stefnestor]

From @pickypg's update, I believe we're confirming going with @pmuellr update

If we want to continue down the path of just using the Kibana URL, we'll need to get all the spaces, then for each space, do what's currently being done with the default space.

As mini-step-forward, I've filed #580 to pull Kibana Spaces which'll at least allow introspection that non-default-space data's missing. I'm guessing some variant of @CamiloSierraH's #514 PR, we'll want to grab spaces and then iterate pulling per space:

• Find Saved Objects: Index Pattern
• Actions/Connectors per current
• Rules per current

[mikecote]

I say "ideally", but honestly not sure - there may be some reason it would be preferable to get the data from Kibana. I think the ES versions will likely have "more stuff" than the Kibana versions, which seems like a good thing. @mikecote ?

If we want to continue down the path of just using the Kibana URL, we'll need to get all the spaces, then for each space, do what's currently being done with the default space.

@pmuellr

The data returned from Elasticsearch directly would contain encrypted attributes like API keys for alerting rules and configuration / passwords for connectors (if we export connectors). Going through Kibana will remove those encrypted attributes when using the APIs.

In Kibana, there is such a way when not using authorization (by using the saved object repository) to get a list of saved-objects across all the spaces. This is how we handle the alerting health API => https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/health/get_health.ts#L37

[pmuellr]

The data returned from Elasticsearch directly would contain encrypted attributes like API keys for alerting rules and configuration / passwords for connectors (if we export connectors). Going through Kibana will remove those encrypted attributes when using the APIs.

But they are encrypted :-).

Ya, probably best to make sure these aren't exported, so if we did use ES, we'd have to filter those away.

In Kibana, there is such a way when not using authorization (by using the saved object repository) to get a list of saved-objects across all the spaces. This is how we handle the alerting health API => https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/health/get_health.ts#L37

The diagnostic tool uses HTTP endpoints to get it's data, so I don't think this will help. I'm guessing someday we'll have a role of "alerting admin" or such, that would be kinda super-user for alerting things only, and then we could make some endpoints available only for those roles or something. Guessing that will be a while though.

[mikecote]

The diagnostic tool uses HTTP endpoints to get it's data, so I don't think this will help. I'm guessing someday we'll have a role of "alerting admin" or such, that would be kinda super-user for alerting things only, and then we could make some endpoints available only for those roles or something. Guessing that will be a while though.

I think that path makes sense as a starting point, even if starts with diagnostics purposes. We could make the API internal, protected behind a certain privilege that would give space agnostic information.

[stefnestor]

@mikecote @pmuellr will you link the Kibana-side Github to create an internal API for space-agnostic data pulls so I can follow? 🙏🏼

[pmuellr]

I don't think we had an issue open for this already, so opened a new one, which should get linked back to this issue: elastic/kibana#137352

[stefnestor]

👋🏼 @mikecote @pmuellr may we review this again? Cross-posted elastic/kibana#137352 is now closed Jan3. And user-facing elastic/kibana#140449 is also complete but not (currently/yet/?) in Kibana diagnostics.

---

The current workaround, is to have the user supplement diagnostic by pulling

> GET .kibana*/_search?q=alert.enabled:true&size=2000 > rules.json
$ cat rules.json | jq '.hits.total.value'
# check if need to re-pull with higher-up-to-10000 size
$ cat rules.json | jq '.hits.hits[]|._source.alert.executionStatus.status as $s|select($s!="ok" and $s!="active")'
# shows erring Rules

It may be possible for Alerting & DetectionEngine statuses to disagree (e.g. https://github.com/elastic/sdh-security-team/issues/584) so for List Detection Rules, may also consider polling the following

> GET .kibana*/_search?q=type:"siem-detection-engine-rule-execution-info"&size=2000

[pmuellr]

It feels like PR #580 moved us a little closer, now we have a list of the spaces. So, now it feels like we should use that list, and iterate through each space, collect the rules/connectors like we currently do. We'll have to figure out some convention to name the files, maybe an index of the space in the spaces list, so would be like kibana_alerts_1_1.json for the first set of connectors in the first space listed in kibana_spaces.json?

[mikecote]

cc @XavierM for the request to list all the rules across all the spaces in the diagnostic bundle.

[111andre111]

I suppose looking to elastic/kibana#140449
a request against:
/internal/alerting/_global_execution_logs?date_start=<start>&date_end=<end>&per_page=<num>&page=<num>&namespaces=[<list from spaces>]
should fulfill the request taken here:
#578 (comment)

However given the API has prefix /internal I suppose this one has not been tested to be valid against Infosec issues like exposing credentials to the error messages?

[pmuellr]

However given the API has prefix /internal ...

I don't think that's an appropriate API to use for this - at least I'm not aware of anyone asking for execution data like this.

It should be possible to use public APIs to iterate through all the Kibana spaces, and then do the same processing we're doing for the default space, in those spaces.

[stefnestor]

Noting this will relatively soon also effect Fleet: elastic/kibana#132559, elastic/kibana#172964, elastic/kibana#175831

[lucabelluccini]

Attempt to make Kibana Diag APIs space aware #721