[Security Solution] User is unable to upgrade the ES version from 1.1.1 to 1.2.0

[muskangulati-qasource]

Description:
The user is unable to upgrade the ES version from 1.1.1 to 1.2.0

Build Details:

VERSION: 7.16.0-BC1
BUILD: 45504
COMMIT: 9231d806c9384df4026977ba7435a9302dc2d4ab
ARTIFACT: https://staging.elastic.co/7.16.0-255b8273/summary-7.16.0.html

Browser Details:
All

Preconditions:

1. Kibana user should be logged in.

Steps to Reproduce:

1. Navigate to the fleet tab
2. Go to the policies tab
3. Click on security integration and observe the upgrade is available from 1.1.1 to 1.2.0

Impacted Test case:
N/A

Actual Result:
The user is unable to upgrade the ES version from 1.1.1 to 1.2.0

Expected Result:
The user should be able to upgrade the ES version from 1.1.1 to 1.2.0

What's working:
N/A

What's not working:
N/A

Screenshots:

Logs:
N/A

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

[muskangulati-qasource]

@manishgupta-qasource please review!!

[manishgupta-qasource]

Reviewed & assigned to @kevinlog

[kevinlog]

@muskangulati-qasource this should be testable now. It required a new package and it has been pushed out. You will no longer see the 1.2.0 version, there will be a 1.2.1 version which should work.

[muskangulati-qasource]

Hi @kevinlog,

We have seen that issue is still persistent.

We are unable to upgrade the ES version from 1.2.0 to 1.2.1

Screenshots:

Otherwise, we are able to get the new 1.2.1 ES version with the BC1 artifacts and are able to work on the same.

So it is not a blocker but it's still an issue.

Thanks!

[dasansol92]

Same error on my cloud instance when trying to update. This is the error:

POST /api/fleet/epm/packages/endpoint-1.2.1

"cannot rollover data stream {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:admin/rollover] is unauthorized for user [found-internal-kibana4-server] with roles [kibana_system,found-internal-kibana4-server] on indices [.logs-endpoint.diagnostic.collection-default,.ds-.logs-endpoint.diagnostic.collection-default-2021.10.26-000001], this action is granted by the index privileges [manage_follow_index,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/rollover] is unauthorized for user [found-internal-kibana4-server] with roles [kibana_system,found-internal-kibana4-server] on indices [.logs-endpoint.diagnostic.collection-default,.ds-.logs-endpoint.diagnostic.collection-default-2021.10.26-000001], this action is granted by the index privileges [manage_follow_index,manage,all]\"},\"status\":403}"

Not sure if this is related to the elasticsearch issue we had with fleet permissions since this is with the .logs and not with the .metrics indices.

[kevinlog]

@dasansol92 @muskangulati-qasource are these older cloud instances that initially installed the bad 1.2.0 package? I just tried building a fresh 7.15, then upgraded to 7.16 on cloud and the package installed correctly.

Still @dasansol92 the error message seems similar to the one we were seeing before. We should pull in @joshdover again to see if we need to add anything else to the check.

[dasansol92]

@dasansol92 @muskangulati-qasource are these older cloud instances that initially installed the bad 1.2.0 package? I just tried building a fresh 7.15, then upgraded to 7.16 on cloud and the package installed correctly.

Yes, it's an old instance with 1.2.0 package already installed

[joshdover]

We didn't include any dot-prefixed hidden indices in the Rollover permissions we added for kibana_system, so I think that is the problem. If you can provide me a list of all the dot-prefixed data stream names that the endpoint package uses, I can get try to this updated for BC3.

[kevinlog]

@joshdover I think this is everything - @pzl @joeypoon @ashokaditya please let me know if I missed anything.

.logs-endpoint.action.responses
.logs-endpoint.diagnostic.collection
.logs-endpoint.actions
.metrics-endpoint.metadata_united_default (not a data stream - a destination index of a transform)

[pzl]

For completeness:

Data Streams

• .logs-endpoint.action.responses-*
• .logs-endpoint.actions-*
• .logs-endpoint.alerts-*
• .logs-endpoint.diagnostic.collection-*
• .logs-endpoint.events.file-*
• .logs-endpoint.events.library-*
• .metrics-endpoint.metadata-*
• .metrics-endpoint.metrics-*
• .logs-endpoint.events.network-*
• .metrics-endpoint.policy-*
• .logs-endpoint.events.process-*
• .logs-endpoint.events.registry-*
• .logs-endpoint.events.security-*

Plain Indices

(e.g. transform destination indices)

• metrics-endpoint.metadata_current_default
• .metrics-endpoint.metadata_current_default (post-rename of above)
• .metrics-endpoint.metadata_united_default

[kevinlog]

@pzl aren't the majority of our logs and metrics data streams not hidden? i.e. no . prefix?

I thought this would be the list

• .logs-endpoint.action.responses-* (hidden)
• .logs-endpoint.actions-* (hidden)
• logs-endpoint.alerts-*
• .logs-endpoint.diagnostic.collection-* (hidden)
• logs-endpoint.events.file-*
• logs-endpoint.events.library-*
• metrics-endpoint.metadata-*
• metrics-endpoint.metrics-*
• logs-endpoint.events.network-*
• metrics-endpoint.policy-*
• logs-endpoint.events.process-*
• logs-endpoint.events.registry-*
• logs-endpoint.events.security-*

[joshdover]

I've posted a draft here against Elasticsearch. Could someone from this team verify the changes work? elastic/elasticsearch#80140

[pzl]

@kevinlog quite right. I forgot they are not all hidden.

Then there are only 3 data streams that are hidden.

.logs-endpoint.action.responses-*
.logs-endpoint.actions-*
.logs-endpoint.diagnostic.collection-*

and then the plain indices are still as above

[kevinlog]

@joshdover - I just checked it out and tried it. I'm seeing this error in the Kibana and ES logs:

Kibana:
[2021-11-01T13:34:35.773-04:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

ES:
│ info [o.e.x.s.a.RealmsAuthenticator] [klogan-mbp] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

I tried this on both main and your checked out branch here: elastic/elasticsearch#80140 and saw the same error. Perhaps it's a problem on my end with some configuration.

I can also have @dasansol92 try who has tested the previous changes

[joshdover]

Kibana: [2021-11-01T13:34:35.773-04:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

ES: │ info [o.e.x.s.a.RealmsAuthenticator] [klogan-mbp] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

This is related to the new "security on by default". In the ES logs you should see an enrollment token for Kibana that you can use to setup Kibana's connection to Elasticsearch with ./bin/kibana_init (I think that is name, if not poke around in ./bin and you'll find it) and then ./bin/kibana after that is complete.

If you run from main in the Kibana repo you can also use yarn es source which configures Elasticsearch's security for you using a build produced from your checkout of elasticsearch (must be a sibling to your kibana checkout).

[kevinlog]

@joshdover thanks, I was able to get around the issue.

I checked out your branch and tried out.

A fresh package install works.

In addition, adding an older package and then testing the package upgrade also worked. The packages instill correctly without any issues.

[dasansol92]

@kevinlog @joshdover I was also able to upgrade from 1.1.1 to 1.3.0-dev.0 locally.
Steps:

1- Run kibana, elasticsearch and fleet server in 7.15
2- Create policy and add Endpoint integration.
3- Stop everything and run all again in main branch (running elasticsearch from source using Josh branch).
4- Update Endpoint integration from 1.1.1 to 1.3.0-dev.0 successfully.

However, I wasn't able to reproduce the error locally using the elasticsearch snapshot on main branch.

Let me know if I missed something.

[muskangulati-qasource]

Bug Conversion

• Test-Case not required as this particular checkpoint is already covered in the following test plan:
  • !https://elastic.testrail.io/index.php?/cases/view/76947

Thanks!