Test adjustments for FIPS 140

[jkakavas]

This change replaces #52211. Instead of explicitly setting
xpack.security.ssl.diagnose.trust to false in all or our tests,
we set xpack.security.fips_mode.enabled to true. This
implicitly disables the diagnostic trust manager and at the
same time mimics more closely the settings that we
expect users to set when running in a FIPS 140 JVM.
We need to disable the diagnostic trust manager
because we run our tests in Java 8 with the SunJSSE
in FIPS 140 mode and it explicitly disallows wrapping
the X509TrustManager.

The change seems large but it mostly consists of two
change categories:

• Set xpack.security.fips_mode.enabled to true when we
  run tests in FIPS 140 mode.
• Set the password hashing algorithm to PBKDF2 when
  we run tests in FIPS 140 mode (disallowing bcrypt where
  we already set it explicitly or setting it explicitly now so that
  the default ( bcrypt ) is not used.

The major problematic case in this PR is that the settings
above need to be set every time we create an
SSLService object with some settings, otherwise the
diagnostic trust manager will be enabled and the tests will
fail in Java 8. This is currently done by setting the setting
manually in test methods or providing a getSettingsBuilder
method for test classes where we create a Settings object
more than once. This covers the case now, but is in no
way future proof. Nothing stops one of us from adding
a test that needs a SSLServive object and creating a
Settings object manually without setting fips_mode.enabled
to true. This will pass in the PR CI as we don't run in FIPS
mode but it will fail for the next periodic FIPS job and it
will need to be triaged and fixed.

Alternatives I considered:

• Adding the Settings.builder() to the forbiddent APIs for the
  security plugin. This seems like using a sledgehammer to
  drive a nail and it will be very confusing and annoying for
  developers, for little cost.
• Adding a Base class for the security plugin test classes
  offering the getSettingsBuilder() method. That could be
  useful but it doesn't forbid anyone from adding a new
  test suite with class that doesn't extend that base class
• Stop running our tests with the SunJSSE provider in
  Java 8 and use BCJSSE as we do with Java 11 onwards.
  I refrained from doing so I as I know that SunJSSE with
  BC as the crypto provider in FIPS mode is pretty popular
  and I would expect many organizations to run with that
  in Java 8. I could be swayed if we think the current
  solution is too brittle/ugly.
• Adding a runtime check to disable the Diagnostic Trust
  Manager in FIPS mode in Java 8 when SunJSSE is used. This
  could be done but I'm not positive on how reliably we can
  check this and what's more important I don't feel
  comfortable adding production code for handling CI
  issues.

I don't necessarily love the approach taken here but it's
no use keeping the PR in my drawer for longer, so
I open it up for discussion and I;m really keen on
feedback/suggestions or ideas.

[elasticmachine]

Pinging @elastic/es-security (:Security/Security)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[tvernum]

Sorry, I saw you discussed this at a meeting I didn't attend, and I planned to circle back, but forgot.
Did we consider a system property to disable the DiagnosticTrustManager? It's not a great option, but it's not completely without precedent.

[jkakavas]

Did we consider a system property to disable the DiagnosticTrustManager? It's not a great option, but it's not completely without precedent.

No, not really. Can you elaborate ? Do you mean like a system property that would force the DiagnosticTrustManager to be false even if it is configured to be true in the settings ? Or a system property that would set xpack.security.ssl.diagnose.trust to false ?

I haven't seen any precedent for the former, can you point me to it ? It does feel a bit strange ( but the existing solution is not great either ). If the latter, we do have a system property for our CI that we use ( tests.fips.enabled ) and this is pretty much what #52211 attempted to do.

[ywangd]

IIUC, Tim's suggestion is to have something like the follows:

private static final Setting<Boolean> DIAGNOSE_TRUST_EXCEPTIONS_SETTING =
    Setting.boolSetting(
        "xpack.security.ssl.diagnose.trust", 
        Boolean.parseBoolean(System.getProperty("xpack.security.ssl.diagnose.trust", "true")),
        Setting.Property.NodeScope);

It's still a change to the production code, but much smaller footprint than dynamically detecting FIPS and SunJSSE. I cannot find an example of this approach in the existing code (master branch). But it feels reasonable to me.

[pugnascotia]

Not essential, but would it be worth putting the choice of hashing algorithm in its own utility method?

[jkakavas]

Closing this for now in favor of #56526 ( which is much much cleaner )