[ResponseOps][Alerting] Decouple rule producer/consumer settings from Kibana feature ID

[xcrzx]

Blocker for: https://github.com/elastic/security-team/issues/9533 (internal), https://github.com/elastic/security-team/issues/8799 (internal)

Summary

As part of the RBAC changes in the Security Solution (see this issue for more details), we need to extract rule management into a standalone Kibana product feature.

This involves transferring security rule types from the siem feature (current configuration here) to a new ruleManagement feature. This will require changing the producer and consumer rule settings to a new value, as feature IDs currently serve as the producer/consumer identifiers.

As we discussed previously, migrating rules is not an option (see this comment). Therefore, we need to find out how to make the Alerting RBAC work with the new feature ID while keeping the current rule producer/consumer siem value.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[marshallmain]

As we discussed previously, migrating rules is not an option.

@xcrzx Can you add more details about why migrating rules is not an option? Either a summary in this ticket or a link to where a reader can find more information.

[xcrzx]

@marshallmain The producer/consumer fields are used for RBAC, and migrating them without downtime is not possible in Serverless. The upgrade process in Serverless works in such a way that there might be several Kibanas of different versions running simultaneously, all consuming data from a single Elasticsearch cluster. When a newer version of Kibana starts up, it begins migrating saved objects, including changing the producer/consumer fields in this case. As a result, older versions of Kibana will encounter access errors when attempting to read the migrated data.

[banderror]

Hey @cnasikas, I was told today in a sync meeting with the ResponseOps team that this is going to be taken into work around the 8.16 release cycle, but without a commitment to ship in 8.16. Just double-checking here to confirm.

[cnasikas]

Hey @banderror, this is correct. I am still working on a POC (#183756) for this. We are confident that is feasible but we need to finalize some edge cases. I will post our findings on the issue soon. The next step is to make it production-ready but as you said without a commitment to ship in 8.16.

[cnasikas]

@xcrzx @banderror @kobelb @marshallmain I finished working on the POC. The idea is to change the way we define the alerting privileges when registering a feature privilege as shown in the pseudo-code below. After the schema changes the alerting authorization class changed to accommodate the new privilege schema without breaking changes. Rules created with the siem consumer and alerts produced by these rules work as before. Also, users with the old privileges cannot view or edit these rules. Users with the new privileges can.

Before

id: 'siem', <--- feature ID
alerting:['siem.queryRule'], // other rule types are omitted for brevity
privileges: {
    all: {
      alerting: {
        rule: {
          all: ['siem.queryRule'], // other rule types are omitted for brevity,
        },
        alert: {
          all: ['siem.queryRule'], // other rule types are omitted for brevity,
        },
      },
      // more privileges
    },
    read: {
      alerting: {
        rule: {
          read: ['siem.queryRule'], // other rule types are omitted for brevity,
        },
        alert: {
          all: ['siem.queryRule'], // other rule types are omitted for brevity,
        },
      },
       // more privileges
    },
  }

After

id: 'siem', <--- old feature ID
alerting:[]
privileges: {
    all: {
       // alerting privilege is removed
      // more privileges
    },
    read: {
       // alerting privilege is removed
      // more privileges
    },
  }

id: 'ruleManagement', <--- new feature ID
alerting: [{ ruleTypeId: 'siem.queryRule', consumers: ['siem'] }],
privileges: {
    all: {
      alerting: {
        rule: {
          all: [{ ruleTypeId: 'siem.queryRule', consumers: ['siem'] }],
        },
        alert: {
          all: [{ ruleTypeId: 'siem.queryRule', consumers: ['siem'] }],
        },
      },
      // more privileges
    },
    read: {
      alerting: {
        rule: {
          read: [{ ruleTypeId: 'siem.queryRule', consumers: ['siem'] }],
        },
        alert: {
          all: [{ ruleTypeId: 'siem.queryRule', consumers: ['siem'] }],
        },
      },
       // more privileges
    },
  }

Technical analysis

The Kibana security uses Elasticsearch's application privileges. This way Kibana can represent and store its own privilege models within Elasticsearch roles. To do that, Kibana security creates actions that are granted by a specific privilege. Alerting uses its own RBAC model and is built on top of the existing Kibana security model. Alerting creates its own custom actions. For example alerting:rule-type/my-feature-id/rule/get. This action means that a user with a role that grants this action can get a rule of type rule-type with consumer my-feature-id . By changing the alerting privilege schema we also change the alerting actions. The new format is alerting:alert-type/my-consumer/rule/get. The new action means that a user with a role that grants this action can get a rule of type rule-type with consumer my-consumer. This way we decouple the feature IDs from the consumers. Changing the action strings is not considered a breaking change as long as the user's permission works as before. In our case, this is true because the consumer will stay the same, and alerting:rule-type/my-feature-id/rule/get is equal to alerting:alert-type/my-consumer/rule/get. Just the actions will be associated with a different feature privilege ID.

Conclusions

We are confident that this approach will accommodate the needs of the Security Solution team without introducing breaking changes assuming the roles can be migrated. Given the impact of the new changes across all rule types and alerts and the widespread use of the consumer we would probably need to do preparation work in our codebase. The scope of the preparation work is unknown at the moment. Also, we may discover edge cases or unknowns that we would need to address. We plan to start working on 8.15+ starting with identifying all the needed steps to reach our goal.