-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
:x-pack:qa:full-cluster-restart fails with SecurityException: Keystore has been corrupted or tampered with on FIPS JVM #32737
Comments
Pinging @elastic/es-security |
Actually it seems that in CI the problem surfaced when calling
|
@atorok It is not clear to me how this fails. Can you please detail your reproduction? |
@albertzaharovits sorry for that.
|
It actually takes this to reproduce it : The previous command looks to be equivalent but it just runs an empty check due to the sub-projects. |
This disables the x-pack rolling upgrade tests using a fips JVM, as there are problems creating the keystore. Relates elastic#32737
) This disables the x-pack rolling upgrade tests using a fips JVM, as there are problems creating the keystore. Relates #32737
…stic#32775) This disables the x-pack rolling upgrade tests using a fips JVM, as there are problems creating the keystore. Relates elastic#32737
) This disables the x-pack rolling upgrade tests using a fips JVM, as there are problems creating the keystore. Relates #32737
) This disables the x-pack rolling upgrade tests using a fips JVM, as there are problems creating the keystore. Relates #32737
The
The code above makes the assumption that the
There is an issue ( unresolved ) upstream for Java 8 : https://bugs.openjdk.java.net/browse/JDK-8202893. |
Nice catch @jkakavas ! 💯 |
The actual issue causing the failure is discussed in #28515 (comment). This was identified as part of the effort for FIPS 140-2 compliance and the relevant change was introduced in #28515. It was not backported to 6.3 as it wasn't deemed necessary at the time.
|
@jkakavas What would a user do ? do we expect upgrades from non-fips to fips ? AFAIK we don't have any other tests swap out the JDK during an upgrade, but FIPS is indeed a bit different. We should probably test what we would tell our users to do, either remove 6.3 from bwc tests when running on fips - if we tell users you would need to do something like upgrade to 6.4 then enable fips, or extend the cluster formation tests to know about FIPS and be able to go from a FIPS run-time java version to a non fips one and run with that. @nik9000 any thoughts ? |
fips and non-fips in this context applies to the JVM only, not Elasticsearch. This, in addition to the fact that we do not support FIPS 140 JVMs for anything < 6.4 , means that one cannot first switch to a FIPS 140 JVM and then upgrade. Since this is the only upgrade path, I will be muting 6.3 bwc tests when running on fips. |
@jkakavas I don't disagree, but I'd like us to be crystal clear on what we do support. If someone is on 6.3 (or earlier) they will need to do a rolling upgrade to 6.4 on a non-FIPS JVM. Do we have a documented upgrade to a FIPS JVM process, or are we currently targeting new installs? |
You are right that
was not very clear. This is probably not the right venue for the detailed discussion about what is supported, but I haven't completed the docs to point to yet so:
If they are not on a trial license (that is they're already on platinum) they could do a rolling upgrade where each node starts after upgrade in a FIPS 140 JVM ( provided the necessary configuration changes are performed on the node before starting it, elasticsearch keystore manually recreated, jks keystrores replaced, etc ) .
Yes.
I'm in the process of writing it. TBC, the problem we're seeing in these CI failures is because a 6.3.3 node is running in a FIPS 140 JVM |
If my understanding is correct, removing 6.3 from bwc testing solves the problem of the failing test, but we will support users to do the rolling upgrade and enable FIPS at the same time, which would not be covered by any tests. I think we wouldn't want to mix things too much, so removing 6.3 from bwc tests when running on FIPS jvm seems like the right thing to do, this will test rolling upgrade between fips enabled versions, i.e 6.4 -> 6.5 but it also sounds to me that we would need a new FIPS specific rolling upgrade test that does include 6.3 and older to assert that any version running on non fips can be rolling upgraded to a fips enabled JVM. |
If we do this (and we should), I'd like it to include testing moving in either direction to/from FIPS on the same ES version. I suspect those will be the most likely scenario for customers. |
Conceptually this seems very similar to new JDK versions, and the BWC tests must handle that already. |
Yes, this is exactly what we do. |
IIUC this sets the JAVA_VERSION so that building the distribution won't fail. What we want here is that I'll give |
Sorry @jkakavas, I pointed you to the wrong place. Take a look at elasticsearch/buildSrc/src/main/groovy/org/elasticsearch/gradle/test/NodeInfo.groovy Lines 244 to 247 in 7c0fc20
and elasticsearch/buildSrc/src/main/groovy/org/elasticsearch/gradle/test/NodeInfo.groovy Lines 176 to 180 in 7c0fc20
This is how we ensure the right versions for Java home for the BWC nodes. We will need slightly different logic to pick up that the older nodes should run with a non-FIPS JVM. |
This reverts commit e64bb48.
Relevant log
The test is calling
elasticsearch-keystore add xpack.watcher.encryption_key -x
CI logs:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/25/console
The text was updated successfully, but these errors were encountered: