Remove HTTPS check for API Keys & Service Accounts (elastic#76801)

This commit removes the checks that prevented the use of API Keys and
Service Account (Service Tokens) on nodes without HTTPS
(xpack.security.http.ssl.enabled)

As a consequence of removing this check, the API Key service is now
automatically enabled, but can be explicitly disabled with

     xpack.security.authc.api_key.enabled: false

docs/reference/settings/security-settings.asciidoc

@@ -178,9 +178,7 @@ You can set the following API key service settings in
 
 `xpack.security.authc.api_key.enabled`::
 (<<static-cluster-setting,Static>>)
-Set to `false` to disable the built-in API key service. Defaults to `true` unless
- `xpack.security.http.ssl.enabled` is `false`. This prevents sniffing the API key
-  from a connection over plain http.
+Set to `false` to disable the built-in API key service. Defaults to `true`.
 
 `xpack.security.authc.api_key.hashing.algorithm`::
 (<<static-cluster-setting,Static>>)

x-pack/docs/en/rest-api/security/create-api-keys.asciidoc

@@ -26,11 +26,8 @@ See the note under <<api-key-role-descriptors,`role_descriptors`>>.
 [[security-api-create-api-key-desc]]
 ==== {api-description-title}
 
-The API keys are created by the {es} API key service, which is automatically enabled
-when you <<encrypt-http-communication,configure TLS on the HTTP interface>>. Alternatively,
-you can explicitly enable the `xpack.security.authc.api_key.enabled` setting. When
-you are running in production mode, a bootstrap check prevents you from enabling
-the API key service unless you also enable TLS on the HTTP interface.
+The API keys are created by the {es} API key service, which is automatically enabled.
+For instructions on disabling the API key service, see <<api-key-service-settings>>.
 
 A successful request returns a JSON structure that contains the
 API key, its unique id, and its name. If applicable, it also returns expiration

x-pack/docs/en/rest-api/security/create-service-token.asciidoc

@@ -25,7 +25,6 @@ authentication.
 
 [[security-api-create-service-token-desc]]
 ==== {api-description-title}
-include::../../security/authentication/service-accounts.asciidoc[tag=service-accounts-tls]
 
 A successful create service account token API call returns a JSON structure
 that contains the service account token, its name, and its secret value.

x-pack/docs/en/rest-api/security/delete-service-token.asciidoc

@@ -21,7 +21,6 @@ specified `namespace`.
 
 [[security-api-delete-service-token-desc]]
 ==== {api-description-title}
-include::../../security/authentication/service-accounts.asciidoc[tag=service-accounts-tls]
 
 The API response indicates whether the specified service account token is found
 and deleted or it is not found.

x-pack/docs/en/rest-api/security/get-service-accounts.asciidoc

@@ -28,7 +28,7 @@ NOTE: Currently, only the `elastic/fleet-server` service account is available.
 [[security-api-get-service-accounts-desc]]
 ==== {api-description-title}
 
-include::../../security/authentication/service-accounts.asciidoc[tag=service-accounts-tls]
+This API returns a list of service accounts that match the provided path parameter(s).
 
 [[security-api-get-service-accounts-path-params]]
 ==== {api-path-parms-title}

x-pack/docs/en/rest-api/security/get-service-credentials.asciidoc

@@ -21,7 +21,6 @@ Retrieves all service credentials for a  <<service-accounts,service account>>.
 
 [[security-api-get-service-credentials-desc]]
 ==== {api-description-title}
-include::../../security/authentication/service-accounts.asciidoc[tag=service-accounts-tls]
 
 Use this API to retrieve a list of credentials for a service account.
 The response includes service account tokens that were created with the

x-pack/docs/en/rest-api/security/grant-api-keys.asciidoc

@@ -23,15 +23,16 @@ Creates an API key on behalf of another user.
 This API is similar to <<security-api-create-api-key>>, however it creates the
 API key for a user that is different than the user that runs the API.
 
-The caller must have authentication credentials (either an access token, or a username and password) for the user on whose behalf the API key will be created. It is not possible to use this API to create an API key without that user's credentials.
-
-This API is intended be used by applications that need to create and manage API keys for end users, but cannot guarantee that those users have permission to create API keys on their own behalf (see <<security-api-create-api-key-prereqs>>).
+The caller must have authentication credentials (either an access token,
+or a username and password) for the user on whose behalf the API key will be
+created. It is not possible to use this API to create an API key without that
+user's credentials.
+
+This API is intended be used by applications that need to create and manage
+API keys for end users, but cannot guarantee that those users have permission
+to create API keys on their own behalf (see <<security-api-create-api-key-prereqs>>).
 The API keys are created by the {es} API key service, which is automatically
-enabled when you configure TLS on the HTTP interface. See <<encrypt-http-communication>>.
-Alternatively, you can explicitly enable the
-`xpack.security.authc.api_key.enabled` setting. When you are running in
-production mode, a bootstrap check prevents you from enabling the API key
-service unless you also enable TLS on the HTTP interface.
+enabled.
 
 A successful grant API key API call returns a JSON structure that contains the
 API key, its unique id, and its name. If applicable, it also returns expiration

x-pack/docs/en/security/authentication/overview.asciidoc

@@ -25,10 +25,12 @@ you must attach your user credentials to the requests sent to {es}. For example,
 when using realms that support usernames and passwords you can simply attach
 {wikipedia}/Basic_access_authentication[basic auth] header to the requests.
 
-The {security-features} provide two services: the token service and the api key
+The {security-features} provide two services: the token service and the API key
 service. You can use these services to exchange the current authentication for
 a token or key. This token or key can then be used as credentials for authenticating
-new requests. These services are enabled by default when TLS/SSL is enabled for HTTP.
+new requests.
+The API key service is enabled by default.
+The token service is enabled by default when TLS/SSL is enabled for HTTP.
 
 include::built-in-users.asciidoc[][]
 include::service-accounts.asciidoc[]

x-pack/docs/en/security/authentication/service-accounts.asciidoc

@@ -57,13 +57,6 @@ users. Service accounts can only be authenticated with service tokens, which are
 not applicable to regular users.
 // end::service-accounts-usage[]
 
-// tag::service-accounts-tls[]
-In <<dev-vs-prod-mode,production mode>>, service accounts require TLS on the
-HTTP interface. A runtime check prevents you from invoking any related APIs or
-authenticating with a service account token unless TLS is enabled on the HTTP
-interface. See <<encrypt-http-communication,encrypt HTTP client communications for {es}>>.
-// end::service-accounts-tls[]
-
 [discrete]
 [[service-accounts-tokens]]
 === Service account tokens

x-pack/docs/en/security/securing-communications/security-basic-setup-https.asciidoc

@@ -4,10 +4,9 @@
 <titleabbrev>Set up basic security plus HTTPS</titleabbrev>
 ++++
 
-In a production environment, some {es} features such as tokens and
-API keys will be disabled unless you enable TLS on the HTTP layer. This
-additional layer of security ensures that all communications to and from your
-cluster are secured.
+When you enable TLS on the HTTP layer it provides an additional layer of
+security to ensure that all communications to and from your
+cluster are encrypted.
 
 When you run the `elasticsearch-certutil` tool in `http` mode, the tool asks
 several questions about how you want to generate certificates. While there are

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSettings.java

@@ -93,9 +93,9 @@ private XPackSettings() {
     public static final Setting<Boolean> TOKEN_SERVICE_ENABLED_SETTING =
         Setting.boolSetting("xpack.security.authc.token.enabled", XPackSettings.HTTP_SSL_ENABLED, Setting.Property.NodeScope);
 
-    /** Setting for enabling or disabling the api key service. Defaults to the value of https being enabled */
+    /** Setting for enabling or disabling the api key service. Defaults to true */
     public static final Setting<Boolean> API_KEY_SERVICE_ENABLED_SETTING =
-        Setting.boolSetting("xpack.security.authc.api_key.enabled", XPackSettings.HTTP_SSL_ENABLED, Setting.Property.NodeScope);
+        Setting.boolSetting("xpack.security.authc.api_key.enabled", true, Setting.Property.NodeScope);
 
     /** Setting for enabling or disabling FIPS mode. Defaults to false */
     public static final Setting<Boolean> FIPS_MODE_ENABLED =

x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/license/LicensingTests.java

@@ -20,6 +20,7 @@
 import org.elasticsearch.client.Request;
 import org.elasticsearch.client.RequestOptions;
 import org.elasticsearch.client.ResponseException;
+import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.settings.SecureString;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.util.CollectionUtils;
@@ -47,7 +48,9 @@
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
+import java.util.Comparator;
 import java.util.List;
+import java.util.Locale;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 
@@ -56,6 +59,9 @@
 import static org.elasticsearch.license.LicenseService.LICENSE_EXPIRATION_WARNING_PERIOD;
 import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertNoFailures;
 import static org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
+import static org.hamcrest.Matchers.arrayWithSize;
+import static org.hamcrest.Matchers.containsStringIgnoringCase;
+import static org.hamcrest.Matchers.equalToIgnoringCase;
 import static org.hamcrest.Matchers.greaterThanOrEqualTo;
 import static org.hamcrest.Matchers.hasItem;
 import static org.hamcrest.Matchers.is;
@@ -224,10 +230,19 @@ public void testNoWarningHeaderWhenAuthenticationFailed() throws Exception {
             getRestClient().performRequest(request);
         } catch (ResponseException e) {
             headers = e.getResponse().getHeaders();
-            List<String> afterWarningHeaders= getWarningHeaders(e.getResponse().getHeaders());
+            List<String> afterWarningHeaders = getWarningHeaders(headers);
             assertThat(afterWarningHeaders, Matchers.hasSize(0));
         }
-        assertThat(headers != null && headers.length == 3, is(true));
+        assertThat(headers, notNullValue());
+        assertThat(Strings.arrayToCommaDelimitedString(headers), headers, arrayWithSize(4));
+
+        Arrays.sort(headers, Comparator.comparing((Header h) -> h.getName().toLowerCase(Locale.ROOT)).thenComparing(Header::getValue));
+        assertThat(headers[0].getName(), equalToIgnoringCase("content-length"));
+        assertThat(headers[1].getName(), equalToIgnoringCase("content-type"));
+        assertThat(headers[2].getName(), equalToIgnoringCase("WWW-Authenticate"));
+        assertThat(headers[2].getValue(), containsStringIgnoringCase("ApiKey"));
+        assertThat(headers[3].getName(), equalToIgnoringCase("WWW-Authenticate"));
+        assertThat(headers[3].getValue(), containsStringIgnoringCase("Basic"));
     }
 
     private static void assertElasticsearchSecurityException(ThrowingRunnable runnable) {

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java

@@ -227,7 +227,6 @@
 import org.elasticsearch.xpack.security.authc.service.FileServiceAccountTokenStore;
 import org.elasticsearch.xpack.security.authc.service.IndexServiceAccountTokenStore;
 import org.elasticsearch.xpack.security.authc.service.ServiceAccountService;
-import org.elasticsearch.xpack.security.authc.support.HttpTlsRuntimeCheck;
 import org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator;
 import org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore;
 import org.elasticsearch.xpack.security.authz.AuthorizationService;
@@ -459,7 +458,6 @@ Collection<Object> createComponents(Client client, ThreadPool threadPool, Cluste
         // If we wait until #getBoostrapChecks the secure settings will have been cleared/closed.
         final List<BootstrapCheck> checks = new ArrayList<>();
         checks.addAll(Arrays.asList(
-            new ApiKeySSLBootstrapCheck(),
             new TokenSSLBootstrapCheck(),
             new PkiRealmBootstrapCheck(getSslService()),
             new TLSLicenseBootstrapCheck()));
@@ -552,9 +550,6 @@ Collection<Object> createComponents(Client client, ThreadPool threadPool, Cluste
             clusterService, cacheInvalidatorRegistry, threadPool);
         components.add(apiKeyService);
 
-        final HttpTlsRuntimeCheck httpTlsRuntimeCheck = new HttpTlsRuntimeCheck(settings, transportReference);
-        components.add(httpTlsRuntimeCheck);
-
         final IndexServiceAccountTokenStore indexServiceAccountTokenStore = new IndexServiceAccountTokenStore(
             settings, threadPool, getClock(), client, securityIndex.get(), clusterService, cacheInvalidatorRegistry);
         components.add(indexServiceAccountTokenStore);
@@ -563,8 +558,11 @@ Collection<Object> createComponents(Client client, ThreadPool threadPool, Cluste
             new FileServiceAccountTokenStore(environment, resourceWatcherService, threadPool, clusterService, cacheInvalidatorRegistry);
         components.add(fileServiceAccountTokenStore);
 
-        final ServiceAccountService serviceAccountService = new ServiceAccountService(client,
-            fileServiceAccountTokenStore, indexServiceAccountTokenStore, httpTlsRuntimeCheck);
+        final ServiceAccountService serviceAccountService = new ServiceAccountService(
+            client,
+            fileServiceAccountTokenStore,
+            indexServiceAccountTokenStore
+        );
         components.add(serviceAccountService);
 
         final CompositeRolesStore allRolesStore = new CompositeRolesStore(settings, fileRolesStore, nativeRolesStore, reservedRolesStore,

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/service/TransportGetServiceAccountAction.java

@@ -19,38 +19,31 @@
 import org.elasticsearch.xpack.core.security.action.service.ServiceAccountInfo;
 import org.elasticsearch.xpack.security.authc.service.ServiceAccount;
 import org.elasticsearch.xpack.security.authc.service.ServiceAccountService;
-import org.elasticsearch.xpack.security.authc.support.HttpTlsRuntimeCheck;
 
 import java.util.function.Predicate;
 
 public class TransportGetServiceAccountAction extends HandledTransportAction<GetServiceAccountRequest, GetServiceAccountResponse> {
 
-    private final HttpTlsRuntimeCheck httpTlsRuntimeCheck;
-
     @Inject
-    public TransportGetServiceAccountAction(TransportService transportService, ActionFilters actionFilters,
-                                            HttpTlsRuntimeCheck httpTlsRuntimeCheck) {
+    public TransportGetServiceAccountAction(TransportService transportService, ActionFilters actionFilters) {
         super(GetServiceAccountAction.NAME, transportService, actionFilters, GetServiceAccountRequest::new);
-        this.httpTlsRuntimeCheck = httpTlsRuntimeCheck;
     }
 
     @Override
     protected void doExecute(Task task, GetServiceAccountRequest request, ActionListener<GetServiceAccountResponse> listener) {
-        httpTlsRuntimeCheck.checkTlsThenExecute(listener::onFailure, "get service accounts", () -> {
-            Predicate<ServiceAccount> filter = v -> true;
-            if (request.getNamespace() != null) {
-                filter = filter.and( v -> v.id().namespace().equals(request.getNamespace()) );
-            } 
-            if (request.getServiceName() != null) {
-                filter = filter.and( v -> v.id().serviceName().equals(request.getServiceName()) );
-            }
-            final ServiceAccountInfo[] serviceAccountInfos = ServiceAccountService.getServiceAccounts()
-                .values()
-                .stream()
-                .filter(filter)
-                .map(v -> new ServiceAccountInfo(v.id().asPrincipal(), v.roleDescriptor()))
-                .toArray(ServiceAccountInfo[]::new);
-            listener.onResponse(new GetServiceAccountResponse(serviceAccountInfos));
-        });
+        Predicate<ServiceAccount> filter = v -> true;
+        if (request.getNamespace() != null) {
+            filter = filter.and(v -> v.id().namespace().equals(request.getNamespace()));
+        }
+        if (request.getServiceName() != null) {
+            filter = filter.and(v -> v.id().serviceName().equals(request.getServiceName()));
+        }
+        final ServiceAccountInfo[] serviceAccountInfos = ServiceAccountService.getServiceAccounts()
+            .values()
+            .stream()
+            .filter(filter)
+            .map(v -> new ServiceAccountInfo(v.id().asPrincipal(), v.roleDescriptor()))
+            .toArray(ServiceAccountInfo[]::new);
+        listener.onResponse(new GetServiceAccountResponse(serviceAccountInfos));
     }
 }