Mirror privileges over data streams to their backing indices

[danhermann]

This PR comprises the core of data streams security integration. It changes the authorization code to extend any privileges granted on a data stream to all of its backing indices. It also introduces an includeDataStreams() flag on any requests implementing IndicesRequest so that they can indicate how data streams should be considered during the authz process for each request. For requests where includeDataStreams() is false, authz will not include any data streams in the list of authorized indices for that request. For requests where includeDataStreams() is true, authz will include any matching data streams along with their respective backing indices in the list of authorized indices.

Note that @albertzaharovits's comment below (#58381 (comment)) includes another description of the way in which the authz code expands wildcards for requests that include data streams and those that do not.

Relates to #53100

[elasticmachine]

Pinging @elastic/es-security (:Security/Authorization)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Data streams)

[danhermann]

@elasticmachine update branch

[albertzaharovits]

The core change looks good to me, but I'd like to sit on it a bit more. I'll get back to you tomorrow.

[albertzaharovits]

@danhermann and I had a productive brainstorm session yesterday.

We worked on specifying the Security behaviour when the issued request (which might or might not "support data streams") contains wildcards (which Security must replace with concrete names from the authorised set, given the user's roles) and when the roles grant access to a data stream (or a data stream wildcard) compared to explicit backing indices. As a recap, we wish that when a role grants access to a namespace that covers an existing data stream it implicitly grants access to the backing indices of that data stream as well (but not the other way around, i.e. if the role grants access to some/all of the backing indices of a data stream it does NOT implicitly grant access to the data stream). This must work for all types of "index" privileges, i.e. indices:data* (read, index, delete, write, create, create_doc) and indices:admin* (view_index_metadata, manage, monitor, maintenance). It's worth noting that this mirroring of privileges from the data stream to the backing index only happens if the data stream and the backing index exists in the first place, i.e. if the permission to create an index (or delete, or manage_follower/leader_index) is granted over some namespace it is NOT mirrored over the equivalent backing indices namespace (.ds-).

We went over the following scenarios:

• the issued request contains wildcards that cover backing indices and the roles grant access to that namespace (or a subset of it) -> this is nothing new, existing authz covers this
• the issued request contains wildcards that cover data streams and the roles grant access to that namespace (or a subset of it). This forks into two cases:
  • the request supports data streams, in which case this is another of the ordinary case that is already covered by existing authz
  • the request does NOT support data streams, in which case wildcard expansion should not expand to concrete data stream names even if those are covered by the user's roles (this is because core expands willdcards for request that don't support data streams in a similar way)
• the issued request contains wildcards that cover data streams and the roles do NOT grant access to the covered data streams, but instead grant access to those data streams' backing indices. Permissions granted over indices do not mirror back to the data stream, so this case is once again a regular one, i.e. the requested namespace and the namespace covered by the role are treated as disjoint so the request is authorised for the empty name list.
• the issued request contains wildcards covering the backing indices of some data stream, but the role grants access only to the data stream. This is the important case. We must replace the wildcard in the request with the concrete names of the backing indices, provided that the backing indices represent a data stream that is covered by the role permission, and the request will be authorised as such. This is true for requests that do or do not support wildcards data streams (@danhermann correct me if I'm wrong, I believe this diverges from what we discussed).

---

We discussed implementation as well, but I was not able to get my head around all that enough to recommend an approach.

I now believe we should change

elasticsearch/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/RBACEngine.java

Line 503 in e33a0df

static List<String> resolveAuthorizedIndicesFromRole(Role role, String action, Map<String, IndexAbstraction> aliasAndIndexLookup) {

to include the backing index names of authorised data streams. That will work in conjunction with the change proposed in this PR, i.e. have the permission group check from

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java

Line 216 in e33a0df

if (group.check(action, indexOrAlias)) {

deal with indexAbstractions that although are not covered by the permission they are backing indices of data streams covered as such. Lastly, we should also modify isIndexVisible to hide concrete data streams when the request's wildcard covers one, but data streams are not supported for the particular request.

I only have one clarifying question: does indexAbstraction.getIndices() returns the backing indices for a data stream ? We need that to propagate permission from data stream to backing indices.

[martijnvg]

does indexAbstraction.getIndices() returns the backing indices for a data stream ?

Yes, if the index abstraction is of type data stream then it returns the corresponding backing indices.

[danhermann]

@elasticmachine update branch

[danhermann]

@albertzaharovits, I've implemented the security integration as you suggested above and added unit tests. Also, I confirmed that along with the changes from #57900, this will resolve #57712.

I added a single override to SearchRequest to include data streams as that was necessary for some existing tests to pass, but I'll leave the updating of all the other appropriate request classes to another PR to keep the size of this one somewhat manageable.

If this matches what you wanted, I'll add some additional higher-level tests that validate all the new scenarios we discussed earlier.

I did notice one oddity in some manual testing that I was doing. If I create a user with a role limited to index read privileges on the pattern b*, I get the expected 404 if I attempt GET b-foo/_search for a non-existent b-foo index or data stream. However, the attempt GET .ds-b-foo-000001/_search returns a 
action [indices:data/read/search] is unauthorized for user [limited-user] error rather than the expected 404 and I haven't been able to track down the reason for that difference in behavior.

[ruflin]

@neptunian @ph As soon as this gets merged, we need to clean up the .ds-* permissions in the Ingest Manager. Lets make sure we don't forget it.

[danhermann]

@albertzaharovits, let me know if you have any test suggestions here. I tried to include unit tests targeted narrowly at the behavioral changes in the three methods that were changed as well as top-level end-to-end tests. For the latter, I didn't see a good place to add those except for the REST tests.

[danhermann]

@elasticmachine update branch

[james-elastic]

👍

[albertzaharovits]

LGTM, I think the tests you've added are minimal but good enough.

I've left a couple of test suggestions. I'm not insisting over them because the missing pieces are covered across separate test classes; the issue is we're really not methodical with tests here, and so the burden of structuring testing in this area should not befall on you.

[albertzaharovits]

I would make it an IllegalStateException.

[albertzaharovits]

We need to test that RBACEngine.resolveAuthorizedIndicesFromRole:

• returns the list containing the backing indices if the role grants access to a data stream (granting as part of a wildcard or a simple name)
• indices which are not backing indices are not returned when some data stream is covered by the role permission
• granting access to backing indices (as a wildcard or simple name) does not include the data stream in the list of authorised names

We need to repeat the tests for when the request works with data streams or not.

[albertzaharovits]

no backing indices as well.
But regular indices and aliases are visible. Same with backing indices.
(I would expand the namespace granted to by the role to ensure those entities are OK - i.e. I'm not suggesting another test, just a more complete one).

But I would also test a simple (not wildcard) data stream name.

[albertzaharovits]

Should include the backing indices as well, right?

[albertzaharovits]

Before merging, can you work on the description and the title as well?

Eg "Mirror privileges over data stream to their backing indices"

Also explain the wildcard behaviour for requests that don't work with wildcards data streams.

[danhermann]

Thank you, @albertzaharovits. I'll make the changes you requested. I'm also happy to work on additional tests in a follow-up if there are others that you would like to have.