Add basic alias support for data streams #72613

martijnvg · 2021-05-03T09:00:51Z

Aliases to data streams can be defined via the existing update aliases api.
Aliases can either only refer to data streams or to indices (not both).
Also the existing get aliases api has been modified to support returning
aliases that refer to data streams.

Aliases for data streams are stored separately from data streams and
and refer to data streams by name and not to the backing indices of
a data stream. This means that when backing indices are added or removed
from a data stream that then the data stream alias doesn't need to be
updated. When a data stream alias is used in an api then all the backing indices
of the data streams the alias currently refers to are resolved.

The authorization model for aliases that refer to data streams is the
same as for aliases the refer to indices. In security privileges can
be defined on aliases, indices and data streams. When a privilege is
granted on an alias then access is also granted on the indices that
an alias refers to (irregardless whether privileges are granted or denied
on the actual indices). The same will apply for aliases that refer
to data streams. See for more details: #66163 (comment)

martijnvg · 2021-05-03T15:17:14Z

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/security/authz/50_data_streams.yml

@@ -192,8 +192,8 @@ teardown:

  - do:
      headers: { Authorization: "Basic dGVzdF91c2VyOngtcGFjay10ZXN0LXBhc3N3b3Jk" } # test_user
-      indices.get_alias:
-        name: easy*
+      indices.get:


get_alias api now support data streams, so changed this test to use get index api to test
that apis that don't data streams do not include data streams among authorised indices.

martijnvg · 2021-05-03T15:18:16Z

x-pack/plugin/build.gradle

@@ -138,6 +138,7 @@ tasks.named("yamlRestCompatTest").configure {
    'security/authz/14_cat_indices/Test explicit request while multiple authorized indices',
    'security/authz/14_cat_indices/Test explicit request while multiple opened/closed authorized indices',
    'security/authz/14_cat_indices/Test wildcard request with multiple authorized indices',
+    'security/authz/50_data_streams/Test that requests not supporting data streams do not include data streams among authorized indices',


In 7.x, this test uses get alias api, but that api now supports data streams,
so this yaml rest combat test needs to be muted.

Hi @martijnvg I am testing in 7.14.0 with wildcard matching on the data streams indices and it does not seem to work. Not sure what I did wrong here

But not using wild card does work

Hey @hungnguyen-elastic, thanks for reporting! I checked this locally and ran into the same issue. I will work on a fix for this.

I've opened #75456 to track this bug.

Thank you @martijnvg

@mikeh-elastic FYI

Aliases to data streams can be defined via the existing update aliases api. Aliases can either only refer to data streams or to indices (not both). Also the existing get aliases api has been modified to support returning aliases that refer to data streams. Aliases for data streams are stored separately from data streams and and refer to data streams by name and not to the backing indices of a data stream. This means that when backing indices are added or removed from a data stream that then the data stream alias doesn't need to be updated. The authorization model for aliases that refer to data streams is the same as for aliases the refer to indices. In security privileges can be defined on aliases, indices and data streams. When a privilege is granted on an alias then access is also granted on the indices that an alias refers to (irregardless whether privileges are granted or denied on the actual indices). The same will apply for aliases that refer to data streams. See for more details: elastic#66163 (comment) Relates to elastic#66163

elasticmachine · 2021-05-04T09:28:07Z

Pinging @elastic/es-core-features (Team:Core/Features)

danhermann

Definitely a lot of stuff happening in this PR! I have some suggestions, but nothing blocking.

I'd suggest getting @jrodewig to look over the doc changes for consistency, wording, etc.
I believe the merge conflict in DataStreamMetadata is due to a recent change by Armin that makes that class immutable. I'd suggest doing the same thing with the new dataStreamAliases member that you're adding here.
Along the lines of the previous item, I'd suggest making DataStreamAlias immutable as its list of data streams is currently mutable.
I wonder if it makes sense to introduce a new IndexAbstraction.Type member since index aliases and data stream aliases are different. As it stands, IndexAbstraction.DataStreamAlias is the only implementation of IndexAbstraction that does not have its own type.
More broadly, and perhaps not a question for this PR specifically, but with data stream aliases limited to data streams, they won't support the use case of migrating a Kibana dashboard gradually from indices to data streams. Were we to want to support that use case in the future, it seems like it would require a lot of rework to this implementation rather than just an iterative refinement of it.

martijnvg · 2021-05-10T08:51:15Z

Thanks for reviewing @danhermann!

I believe the merge conflict in DataStreamMetadata is due to a recent change by Armin that makes that class immutable. I'd suggest doing the same thing with the new dataStreamAliases member that you're adding here.

👍

Along the lines of the previous item, I'd suggest making DataStreamAlias immutable as its list of data streams is currently mutable.

👍

I wonder if it makes sense to introduce a new IndexAbstraction.Type member since index aliases and data stream aliases are different. As it stands, IndexAbstraction.DataStreamAlias is the only implementation of IndexAbstraction that does not have its own type.

Data stream alias are a different kind of alias compared to indices aliases, but both are similar enough that the functionality that these kinds of aliases provide can abstracted behind the IndexAbstraction.Type.ALIAS enum. If we would introduce another IndexAbstraction.Type enum instance then a lot of checks in the codebase would check for IndexAbstraction.Type.ALIAS and IndexAbstraction.Type.DATA_STREAM_ALIAS, which I think in many places would be verbose.

Also for someone setting up aliases, there isn't a real notion of data stream aliases or indices aliases. The update aliases api or get alias api don't make a distinction between the two, it is just that an alias either refers to indices or data streams. It is just that certain alias options (like routing) aren't supported for aliases that refer to data streams. APIs that use aliases just resolve an alias to a set of indices (irregardless what type of alias or whether an alias refers to a data stream and that data stream refers to backing indices).

More broadly, and perhaps not a question for this PR specifically, but with data stream aliases limited to data streams, they won't support the use case of migrating a Kibana dashboard gradually from indices to data streams. Were we to want to support that use case in the future, it seems like it would require a lot of rework to this implementation rather than just an iterative refinement of it.

I don't see data stream aliases play a role in this case. If an alias is used in front of these indices then the migrate data stream api can be used. If not, then currently there isn't much we can do to help migration to data streams. We would need ES to support index renaming (atomically) in order to be of any help here.

jrodewig

Thanks for putting this together @martijnvg. I left some suggestions, but feel free to ignore them if wanted.

Some additional doc changes are required, but I'll handle those in a separate PR. I'll add you as a reviewer there.

docs/reference/indices/aliases.asciidoc

Co-authored-by: James Rodewig <[email protected]>

martijnvg · 2021-05-10T17:58:58Z

@elasticmachine update branch

martijnvg · 2021-05-10T18:00:27Z

Thanks for reviewing @jrodewig!

Backporting elastic#72613 to 7.x. Aliases to data streams can be defined via the existing update aliases api. Aliases can either only refer to data streams or to indices (not both). Also the existing get aliases api has been modified to support returning aliases that refer to data streams. Aliases for data streams are stored separately from data streams and and refer to data streams by name and not to the backing indices of a data stream. This means that when backing indices are added or removed from a data stream that then the data stream alias doesn't need to be updated. The authorization model for aliases that refer to data streams is the same as for aliases the refer to indices. In security privileges can be defined on aliases, indices and data streams. When a privilege is granted on an alias then access is also granted on the indices that an alias refers to (irregardless whether privileges are granted or denied on the actual indices). The same will apply for aliases that refer to data streams. See for more details: elastic#66163 (comment) Relates to elastic#66163

Backporting #72613 to 7.x. Aliases to data streams can be defined via the existing update aliases api. Aliases can either only refer to data streams or to indices (not both). Also the existing get aliases api has been modified to support returning aliases that refer to data streams. Aliases for data streams are stored separately from data streams and and refer to data streams by name and not to the backing indices of a data stream. This means that when backing indices are added or removed from a data stream that then the data stream alias doesn't need to be updated. The authorization model for aliases that refer to data streams is the same as for aliases the refer to indices. In security privileges can be defined on aliases, indices and data streams. When a privilege is granted on an alias then access is also granted on the indices that an alias refers to (irregardless whether privileges are granted or denied on the actual indices). The same will apply for aliases that refer to data streams. See for more details: #66163 (comment) Relates to #66163 * made code compatible with java 8

Relates to elastic#72613

Relates to #72613

#72613 adds data stream support to aliases. This adds an `alias` glossary entry and removes out the current `index alias` entry.

martijnvg force-pushed the data_stream_aliases_basics branch from fbaa9b4 to 9732815 Compare May 3, 2021 09:15

martijnvg commented May 3, 2021

View reviewed changes

martijnvg force-pushed the data_stream_aliases_basics branch from 801c2bf to 63ad677 Compare May 4, 2021 09:26

martijnvg marked this pull request as ready for review May 4, 2021 09:27

martijnvg requested a review from danhermann May 4, 2021 09:27

martijnvg added :Data Management/Data streams Data streams and their lifecycles >feature v7.14.0 v8.0.0 labels May 4, 2021

elasticmachine added the Team:Data Management Meta label for data/management team label May 4, 2021

martijnvg mentioned this pull request May 4, 2021

Introduce aliases for data streams #66163

Open

7 tasks

danhermann approved these changes May 7, 2021

View reviewed changes

martijnvg requested a review from jrodewig May 10, 2021 08:52

martijnvg added 2 commits May 10, 2021 11:05

Merge remote-tracking branch 'es/master' into data_stream_aliases_basics

3dd5400

iter

ed23ad8

jrodewig approved these changes May 10, 2021

View reviewed changes

docs/reference/indices/aliases.asciidoc Outdated Show resolved Hide resolved

docs/reference/indices/aliases.asciidoc Outdated Show resolved Hide resolved

martijnvg and others added 2 commits May 10, 2021 19:57

group unsupported options together and fix minor typos

df02c74

Co-authored-by: James Rodewig <[email protected]>

use more realistic data stream names

b1349a7

Co-authored-by: James Rodewig <[email protected]>

Merge branch 'master' into data_stream_aliases_basics

f657170

martijnvg merged commit 6689b8b into elastic:master May 11, 2021

martijnvg added the backport pending label May 11, 2021

martijnvg mentioned this pull request May 11, 2021

[7.x] Add basic alias support for data streams #72908

Merged

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request May 11, 2021

Disable bwc for backporting elastic#72613

4ba421f

martijnvg mentioned this pull request May 11, 2021

Disable bwc for backporting #72613 #72918

Merged

martijnvg added a commit that referenced this pull request May 11, 2021

Disable bwc for backporting #72613 (#72918)

5f6fa0b

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request May 11, 2021

Enable bwc tests and adjust serialization versioning.

633e11d

Relates to elastic#72613

martijnvg mentioned this pull request May 11, 2021

Enable bwc tests and adjust serialization versioning. #72919

Merged

martijnvg added a commit that referenced this pull request May 11, 2021

Enable bwc tests and adjust serialization versioning. (#72919)

2253eba

Relates to #72613

martijnvg removed the backport pending label May 12, 2021

This was referenced May 13, 2021

[DOCS] Update docs for data stream aliases #73044

Closed

[DOCS] Add alias to glossary #73065

Merged

[DOCS] Add alias guide #73062

Merged

jrodewig added a commit that referenced this pull request May 17, 2021

[DOCS] Add alias to glossary (#73065)

25421df

#72613 adds data stream support to aliases. This adds an `alias` glossary entry and removes out the current `index alias` entry.

jrodewig added a commit that referenced this pull request May 17, 2021

[DOCS] Add alias to glossary (#73065) (#73167)

d48139d

#72613 adds data stream support to aliases. This adds an `alias` glossary entry and removes out the current `index alias` entry.

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic alias support for data streams #72613

Add basic alias support for data streams #72613

martijnvg commented May 3, 2021 •

edited

Loading

martijnvg May 3, 2021

martijnvg May 3, 2021

hungnguyen-elastic Jul 15, 2021

martijnvg Jul 19, 2021

martijnvg Jul 19, 2021

hungnguyen-elastic Jul 19, 2021

hungnguyen-elastic Jul 19, 2021

elasticmachine commented May 4, 2021

danhermann left a comment

martijnvg commented May 10, 2021

jrodewig left a comment •

edited

Loading

martijnvg commented May 10, 2021

martijnvg commented May 10, 2021

Add basic alias support for data streams #72613

Add basic alias support for data streams #72613

Conversation

martijnvg commented May 3, 2021 • edited Loading

martijnvg May 3, 2021

Choose a reason for hiding this comment

martijnvg May 3, 2021

Choose a reason for hiding this comment

hungnguyen-elastic Jul 15, 2021

Choose a reason for hiding this comment

martijnvg Jul 19, 2021

Choose a reason for hiding this comment

martijnvg Jul 19, 2021

Choose a reason for hiding this comment

hungnguyen-elastic Jul 19, 2021

Choose a reason for hiding this comment

hungnguyen-elastic Jul 19, 2021

Choose a reason for hiding this comment

elasticmachine commented May 4, 2021

danhermann left a comment

Choose a reason for hiding this comment

martijnvg commented May 10, 2021

jrodewig left a comment • edited Loading

Choose a reason for hiding this comment

martijnvg commented May 10, 2021

martijnvg commented May 10, 2021

martijnvg commented May 3, 2021 •

edited

Loading

jrodewig left a comment •

edited

Loading