Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Protect newly introduced system indices fully #74186

Merged
merged 11 commits into from
Jun 24, 2021

Conversation

jaymode
Copy link
Member

@jaymode jaymode commented Jun 16, 2021

This change updates the way we handle net new system indices, which are
those that have been newly introduced and do not require any BWC
guarantees around non-system access. These indices will not be included
in wildcard expansions for user searches and operations. Direct access
to these indices will also not be allowed for user searches.

The first index of this type is the GeoIp index, which this change sets
the new flag on.

Closes #72572

This change updates the way we handle net new system indices, which are
those that have been newly introduced and do not require any BWC
guarantees around non-system access. These indices will not be included
in wildcard expansions for user searches and operations. Direct access
to these indices will also not be allowed for user searches.

The first index of this type is the GeoIp index, which this change sets
the new flag on.

Closes elastic#72572
@jaymode jaymode added >enhancement :Core/Infra/Core Core issues without another label v8.0.0 v7.14.0 labels Jun 16, 2021
@jaymode jaymode requested a review from williamrandolph June 16, 2021 14:52
@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Jun 16, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@gwbrown gwbrown self-requested a review June 16, 2021 16:09
Copy link
Contributor

@williamrandolph williamrandolph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good. What are your thoughts on index templates here? Unless you're already in progress there, I should probably handle that in a follow-up.

qa/system-indices/build.gradle Outdated Show resolved Hide resolved
@@ -414,6 +418,10 @@ public boolean isInternal() {
return allowedElasticProductOrigins;
}

public boolean isNetNew() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and elsewhere, I'd like to add javadoc that explains clearly what we mean by "net new". It wasn't an expression I was familiar with and I see some potential for confusion.

I'll be happy to add a commit with this javadoc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put a note about adding javadoc on #74271 so that we can merge this PR as soon as possible.

jaymode and others added 2 commits June 17, 2021 15:04
@jaymode
Copy link
Member Author

jaymode commented Jun 17, 2021

What are your thoughts on index templates here?

Index templates were a miss on this one. If you want to handle as a follow-up, that works for me

Copy link
Contributor

@gwbrown gwbrown left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@williamrandolph
Copy link
Contributor

@elasticmachine update branch

@williamrandolph williamrandolph merged commit d4afd6a into elastic:master Jun 24, 2021
williamrandolph pushed a commit that referenced this pull request Jun 24, 2021
This change updates the way we handle net new system indices, which are
those that have been newly introduced and do not require any BWC
guarantees around non-system access. These indices will not be included
in wildcard expansions for user searches and operations. Direct access
to these indices will also not be allowed for user searches.

The first index of this type is the GeoIp index, which this change sets
the new flag on.

Closes #72572
@williamrandolph
Copy link
Contributor

backport commit: 642e059

@mbudge
Copy link

mbudge commented Nov 16, 2021

We need to be able to apply custom mappings to the siem signals indexes.

This change broke the security event aggregation code in our SOAR when the siem signals indexes were changed to system indexes.

@williamrandolph
Copy link
Contributor

@mbudge The siem signals index (.siem-signals) is not defined as a system index, so I am not sure if you have found a bug or if you need to make some other adjustment for the version you're using. I will need more information to investigate.

I'd like to start the investigation at http://discuss.elastic.co . If we verify a bug, then we can open a new issue in this github repository.

Could you post in the elasticsearch category on the discuss forums? Please include the elasticsearch version where your code worked, as well as the elasticsearch version you are updating to, as well as any errors or stack traces you are finding. It would also help to know what security user and role you are using for your request. After that, comment here with a link to your discuss forum post and I will take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Core Core issues without another label >enhancement Team:Core/Infra Meta label for core/infra team v7.14.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Instances with small heaps and wildcard searches may run OOM
6 participants