From 5b80065dfea4d5655b59dc1b8a43db80a49fc6f1 Mon Sep 17 00:00:00 2001 From: Jake Landis Date: Thu, 15 Feb 2024 09:23:48 -0600 Subject: [PATCH 01/45] Update FIPS documentation for 8.x (#105041) This commit updates the documentation for FIPS support. In addition to the changes for 8.x it also provides more details for how to setup/configure FIPS mode. --- .../security/fips-140-compliance.asciidoc | 165 ++++++++++++++---- docs/reference/security/fips-java17.asciidoc | 13 +- 2 files changed, 132 insertions(+), 46 deletions(-) diff --git a/docs/reference/security/fips-140-compliance.asciidoc b/docs/reference/security/fips-140-compliance.asciidoc index 785c720dba407..bf880213c2073 100644 --- a/docs/reference/security/fips-140-compliance.asciidoc +++ b/docs/reference/security/fips-140-compliance.asciidoc @@ -8,59 +8,75 @@ government computer security standard used to approve cryptographic modules. {es} offers a FIPS 140-2 compliant mode and as such can run in a FIPS 140-2 configured JVM. -IMPORTANT: The JVM bundled with {es} is not configured for FIPS 140-2. You must +IMPORTANT: The JVM bundled with {es} is not configured for FIPS 140-2. You must configure an external JDK with a FIPS 140-2 certified Java Security Provider. Refer to the {es} https://www.elastic.co/support/matrix#matrix_jvm[JVM support matrix] for -supported JVM configurations. +supported JVM configurations. See https://www.elastic.co/subscriptions[subscriptions] for required licensing. -After configuring your JVM for FIPS 140-2, you can run {es} in FIPS 140-2 mode by -setting the `xpack.security.fips_mode.enabled` to `true` in `elasticsearch.yml`. +Compliance with FIPS 140-2 requires using only FIPS approved / NIST recommended cryptographic algorithms. Generally this can be done by the following: -For {es}, adherence to FIPS 140-2 is ensured by: - -- Using FIPS approved / NIST recommended cryptographic algorithms. -- Delegating the implementation of these cryptographic algorithms to a NIST - validated cryptographic module (available via the Java Security Provider - in use in the JVM). -- Allowing the configuration of {es} in a FIPS 140-2 compliant manner, as - documented below. +- Installation and configuration of a FIPS certified Java security provider. +- Ensuring the configuration of {es} is FIPS 140-2 compliant as documented below. +- Setting `xpack.security.fips_mode.enabled` to `true` in `elasticsearch.yml`. Note - this setting alone is not sufficient to be compliant +with FIPS 140-2. [discrete] -=== Upgrade considerations +=== Configuring {es} for FIPS 140-2 -[IMPORTANT] -==== -include::fips-java17.asciidoc[] -==== +Detailed instructions for the configuration required for FIPS 140-2 compliance is beyond the scope of this document. It is the responsibility +of the user to ensure compliance with FIPS 140-2. {es} has been tested with a specific configuration described below. However, there are +other configurations possible to achieve compliance. +The following is a high-level overview of the required configuration: -If you plan to upgrade your existing cluster to a version that can be run in -a FIPS 140-2 configured JVM, we recommend to first perform a rolling -upgrade to the new version in your existing JVM and perform all necessary -configuration changes in preparation for running in FIPS 140-2 mode. You can then -perform a rolling restart of the nodes, starting each node in a FIPS 140-2 JVM. -During the restart, {es}: +* Use an externally installed Java installation. The JVM bundled with {es} is not configured for FIPS 140-2. +* Install a FIPS certified security provider .jar file(s) in {es}'s `lib` directory. +* Configure Java to use a FIPS certified security provider (xref:java-security-provider[see below]). +* Configure {es}'s security manager to allow use of the FIPS certified provider (xref:java-security-manager[see below]). +* Ensure the keystore and truststore are configured correctly (xref:keystore-fips-password[see below]). +* Ensure the TLS settings are configured correctly (xref:fips-tls[see below]). +* Ensure the password hashing settings are configured correctly (xref:fips-stored-password-hashing[see below]). +* Ensure the cached password hashing settings are configured correctly (xref:fips-cached-password-hashing[see below]). +* Configure `elasticsearch.yml` to use FIPS 140-2 mode, see (xref:configuring-es-yml[below]). +* Verify the security provider is installed and configured correctly (xref:verify-security-provider[see below]). +* Review the upgrade considerations (xref:fips-upgrade-considerations[see below]) and limitations (xref:fips-limitations[see below]). -- Upgrades <> to the latest, compliant format. - A FIPS 140-2 JVM cannot load previous format versions. If your keystore is - not password-protected, you must manually set a password. See - <>. -- Upgrades self-generated trial licenses to the latest FIPS 140-2 compliant format. -If your {subscriptions}[subscription] already supports FIPS 140-2 mode, you -can elect to perform a rolling upgrade while at the same time running each -upgraded node in a FIPS 140-2 JVM. In this case, you would need to also manually -regenerate your `elasticsearch.keystore` and migrate all secure settings to it, -in addition to the necessary configuration changes outlined below, before -starting each node. +[discrete] +[[java-security-provider]] +==== Java security provider + +Detailed instructions for installation and configuration of a FIPS certified Java security provider is beyond the scope of this document. +Specifically, a FIPS certified +https://docs.oracle.com/en/java/javase/17/security/java-cryptography-architecture-jca-reference-guide.html[JCA] and +https://docs.oracle.com/en/java/javase/17/security/java-secure-socket-extension-jsse-reference-guide.html[JSSE] implementation is required +so that the JVM uses FIPS validated implementations of NIST recommended cryptographic algorithms. + +Elasticsearch has been tested with Bouncy Castle's https://repo1.maven.org/maven2/org/bouncycastle/bc-fips/1.0.2.4/bc-fips-1.0.2.4.jar[bc-fips 1.0.2.4] +and https://repo1.maven.org/maven2/org/bouncycastle/bctls-fips/1.0.17/bctls-fips-1.0.17.jar[bctls-fips 1.0.17]. +Please refer to the [Support Matrix] for details on which combinations of JVM and security provider are supported in FIPS mode. Elasticsearch does not ship with a FIPS certified provider. It is the responsibility of the user +to install and configure the security provider to ensure compliance with FIPS 140-2. Using a FIPS certified provider will ensure that only +approved cryptographic algorithms are used. + +To configure {es} to use additional security provider(s) configure {es}'s <> `java.security.properties` to point to a file +(https://raw.githubusercontent.com/elastic/elasticsearch/main/build-tools-internal/src/main/resources/fips_java.security[example]) in {es}'s +`config` directory. Ensure the FIPS certified security provider is configured with the lowest order. This file should contain the necessary +configuration to instruct Java to use the FIPS certified security provider. [discrete] -=== Configuring {es} for FIPS 140-2 +[[java-security-manager]] +==== Java security manager + +All code running in {es} is subject to the security restrictions enforced by the Java security manager. +The security provider you have installed and configured may require additional permissions in order to function correctly. You can grant these permissions by providing your own +https://docs.oracle.com/javase/8/docs/technotes/guides/security/PolicyFiles.html#FileSyntax[Java security policy] + +To configure {es}'s security manager configure the JVM property `java.security.policy` to point a file +(https://raw.githubusercontent.com/elastic/elasticsearch/main/build-tools-internal/src/main/resources/fips_java.policy[example])in {es}'s +`config` directory with the desired permissions. This file should contain the necessary configuration for the Java security manager +to grant the required permissions needed by the security provider. -Apart from setting `xpack.security.fips_mode.enabled`, a number of security -related settings need to be configured accordingly in order to be compliant -and able to run {es} successfully in a FIPS 140-2 configured JVM. [discrete] [[keystore-fips-password]] @@ -78,6 +94,7 @@ Note that when the keystore is password-protected, you must supply the password Elasticsearch starts. [discrete] +[[fips-tls]] ==== TLS SSLv2 and SSLv3 are not allowed by FIPS 140-2, so `SSLv2Hello` and `SSLv3` cannot @@ -172,6 +189,78 @@ hashes using non-compliant algorithms will be discarded and the new ones will be created using the algorithm you have selected. [discrete] +[[configuring-es-yml]] +==== Configure {es} elasticsearch.yml + +* Set `xpack.security.fips_mode.enabled` to `true` in `elasticsearch.yml`. This setting is used to ensure to configure some internal +configuration to be FIPS 140-2 compliant and provides some additional verification. + +* Set `xpack.security.autoconfiguration.enabled` to `false`. This will disable the automatic configuration of the security settings. +Users must ensure that the security settings are configured correctly for FIPS-140-2 compliance. This is only applicable for new installations. + +* Set `xpack.security.authc.password_hashing.algorithm` appropriately see xref:fips-stored-password-hashing[above]. + +* Other relevant security settings. For example, TLS for the transport and HTTP interfaces. (not explicitly covered here or in the example below) + +* Optional: Set `xpack.security.fips_mode.required_providers` in `elasticsearch.yml` to ensure the required security providers (8.13+). +see xref:verify-security-provider[below]. + +[source,yaml] +-------------------------------------------------- +xpack.security.fips_mode.enabled: true +xpack.security.autoconfiguration.enabled: false +xpack.security.fips_mode.required_providers: ["BCFIPS", "BCJSSE"] +xpack.security.authc.password_hashing.algorithm: "pbkdf2_stretch" +-------------------------------------------------- + +[discrete] +[[verify-security-provider]] +==== Verify the security provider is installed + +To verify that the security provider is installed and in use, you can use any of the following steps: + +* Verify the required security providers are configured with the lowest order in the file pointed to by `java.security.properties`. +For example, `security.provider.1` is a lower order than `security.provider.2` + +* Set `xpack.security.fips_mode.required_providers` in `elasticsearch.yml` to the list of required security providers. +This setting is used to ensure that the correct security provider is installed and configured. (8.13+) +If the security provider is not installed correctly, {es} will fail to start. `["BCFIPS", "BCJSSE"]` are the values to +use for Bouncy Castle's FIPS JCE and JSSE certified provider. + +[discrete] +[[fips-upgrade-considerations]] +=== Upgrade considerations +include::fips-java17.asciidoc[] + +[IMPORTANT] +==== +Some encryption algorithms may no longer be available by default in updated FIPS 140-2 security providers. +Notably, Triple DES and PKCS1.5 RSA are now discouraged and https://www.bouncycastle.org/fips-java[Bouncy Castle] now +requires explicit configuration to continue using these algorithms. +==== + +If you plan to upgrade your existing cluster to a version that can be run in +a FIPS 140-2 configured JVM, we recommend to first perform a rolling +upgrade to the new version in your existing JVM and perform all necessary +configuration changes in preparation for running in FIPS 140-2 mode. You can then +perform a rolling restart of the nodes, starting each node in a FIPS 140-2 JVM. +During the restart, {es}: + +- Upgrades <> to the latest, compliant format. +A FIPS 140-2 JVM cannot load previous format versions. If your keystore is +not password-protected, you must manually set a password. See +<>. +- Upgrades self-generated trial licenses to the latest FIPS 140-2 compliant format. + +If your {subscriptions}[subscription] already supports FIPS 140-2 mode, you +can elect to perform a rolling upgrade while at the same time running each +upgraded node in a FIPS 140-2 JVM. In this case, you would need to also manually +regenerate your `elasticsearch.keystore` and migrate all secure settings to it, +in addition to the necessary configuration changes outlined below, before +starting each node. + +[discrete] +[[fips-limitations]] === Limitations Due to the limitations that FIPS 140-2 compliance enforces, a small number of diff --git a/docs/reference/security/fips-java17.asciidoc b/docs/reference/security/fips-java17.asciidoc index ee1c9bf15eba0..dc46a3ec628f5 100644 --- a/docs/reference/security/fips-java17.asciidoc +++ b/docs/reference/security/fips-java17.asciidoc @@ -1,10 +1,7 @@ -{es} {version} requires Java 17 or later. -There is not yet a FIPS-certified security module for Java 17 -that you can use when running {es} {version} in FIPS 140-2 mode. -If you run in FIPS 140-2 mode, you will either need to request -an exception from your security organization to upgrade to {es} {version}, -or remain on {es} 7.x until Java 17 is certified. -ifeval::["{release-state}"=="released"] +{es} 8.0+ requires Java 17 or later. {es} 8.13+ has been tested with https://www.bouncycastle.org/java.html[Bouncy Castle]'s Java 17 +https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4616[certified] FIPS implementation and is the +recommended Java security provider when running {es} in FIPS 140-2 mode. +Note - {es} does not ship with a FIPS certified security provider and requires explicit installation and configuration. + Alternatively, consider using {ess} in the https://www.elastic.co/industries/public-sector/fedramp[FedRAMP-certified GovCloud region]. -endif::[] \ No newline at end of file From 309adbfc54ddf14406fb6290a6686b8d7a922ba6 Mon Sep 17 00:00:00 2001 From: Kathleen DeRusso Date: Thu, 15 Feb 2024 10:25:24 -0500 Subject: [PATCH 02/45] Fix bug in rule_query where text_expansion errored because it was not rewritten (#105365) --- docs/changelog/105365.yaml | 6 + x-pack/plugin/ent-search/qa/rest/build.gradle | 2 +- .../test/entsearch/260_rule_query_search.yml | 150 ++++++++++++++++++ .../application/rules/RuleQueryBuilder.java | 66 ++++---- .../rules/RuleQueryBuilderTests.java | 3 +- 5 files changed, 190 insertions(+), 37 deletions(-) create mode 100644 docs/changelog/105365.yaml diff --git a/docs/changelog/105365.yaml b/docs/changelog/105365.yaml new file mode 100644 index 0000000000000..265e6dccc3915 --- /dev/null +++ b/docs/changelog/105365.yaml @@ -0,0 +1,6 @@ +pr: 105365 +summary: Fix bug in `rule_query` where `text_expansion` errored because it was not + rewritten +area: Application +type: bug +issues: [] diff --git a/x-pack/plugin/ent-search/qa/rest/build.gradle b/x-pack/plugin/ent-search/qa/rest/build.gradle index c9b1557d74a9c..37f1d8f13c850 100644 --- a/x-pack/plugin/ent-search/qa/rest/build.gradle +++ b/x-pack/plugin/ent-search/qa/rest/build.gradle @@ -7,7 +7,7 @@ dependencies { restResources { restApi { - include '_common', 'bulk', 'cluster', 'connector', 'nodes', 'indices', 'index', 'query_ruleset', 'search_application', 'xpack', 'security', 'search' + include '_common', 'bulk', 'cluster', 'connector', 'nodes', 'indices', 'index', 'query_ruleset', 'search_application', 'xpack', 'security', 'search', 'ml' } } diff --git a/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml b/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml index c287209da5bed..40cdb7839c9ed 100644 --- a/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml +++ b/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml @@ -236,4 +236,154 @@ setup: - match: { hits.total.value: 1 } - match: { hits.hits.0._id: 'doc1' } +--- +"Perform a rule query with an organic query that must be rewritten to another query type": + - skip: + version: " - 8.13.99" + reason: Bugfix that was broken in previous versions + + - do: + indices.create: + index: test-index-with-sparse-vector + body: + mappings: + properties: + source_text: + type: keyword + ml.tokens: + type: sparse_vector + + - do: + ml.put_trained_model: + model_id: "text_expansion_model" + body: > + { + "description": "simple model for testing", + "model_type": "pytorch", + "inference_config": { + "text_expansion": { + "tokenization": { + "bert": { + "with_special_tokens": false + } + } + } + } + } + - do: + ml.put_trained_model_vocabulary: + model_id: "text_expansion_model" + body: > + { "vocabulary": ["[PAD]", "[UNK]", "these", "are", "my", "words", "the", "washing", "machine", "is", "leaking", "octopus", "comforter", "smells"] } + - do: + ml.put_trained_model_definition_part: + model_id: "text_expansion_model" + part: 0 + body: > + { + "total_definition_length":2078, + "definition": "UEsDBAAACAgAAAAAAAAAAAAAAAAAAAAAAAAUAA4Ac2ltcGxlbW9kZWwvZGF0YS5wa2xGQgoAWlpaWlpaWlpaWoACY19fdG9yY2hfXwpUaW55VGV4dEV4cGFuc2lvbgpxACmBfShYCAAAAHRyYWluaW5ncQGJWBYAAABfaXNfZnVsbF9iYWNrd2FyZF9ob29rcQJOdWJxAy5QSwcIITmbsFgAAABYAAAAUEsDBBQACAgIAAAAAAAAAAAAAAAAAAAAAAAdAB0Ac2ltcGxlbW9kZWwvY29kZS9fX3RvcmNoX18ucHlGQhkAWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWoWRT4+cMAzF7/spfASJomF3e0Ga3nrrn8vcELIyxAzRhAQlpjvbT19DWDrdquqBA/bvPT87nVUxwsm41xPd+PNtUi4a77KvXs+W8voBAHFSQY3EFCIiHKFp1+p57vs/ShyUccZdoIaz93aBTMR+thbPqru+qKBx8P4q/e8TyxRlmwVctJp66H1YmCyS7WsZwD50A2L5V7pCBADGTTOj0bGGE7noQyqzv5JDfp0o9fZRCWqP37yjhE4+mqX5X3AdFZHGM/2TzOHDpy1IvQWR+OWo3KwsRiKdpcqg4pBFDtm+QJ7nqwIPckrlnGfFJG0uNhOl38Sjut3pCqg26QuZy8BR9In7ScHHrKkKMW0TIucFrGQXCMpdaDO05O6DpOiy8e4kr0Ed/2YKOIhplW8gPr4ntygrd9ixpx3j9UZZVRagl2c6+imWUzBjuf5m+Ch7afphuvvW+r/0dsfn+2N9MZGb9+/SFtCYdhd83CMYp+mGy0LiKNs8y/eUuEA8B/d2z4dfUEsHCFSE3IaCAQAAIAMAAFBLAwQUAAgICAAAAAAAAAAAAAAAAAAAAAAAJwApAHNpbXBsZW1vZGVsL2NvZGUvX190b3JjaF9fLnB5LmRlYnVnX3BrbEZCJQBaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpahZHLbtNAFIZtp03rSVIuLRKXjdk5ojitKJsiFq24lem0KKSqpRIZt55gE9/GM+lNLFgx4i1Ys2aHhIBXgAVICNggHgNm6rqJN2BZGv36/v/MOWeea/Z5RVHurLfRUsfZXOnccx522itrd53O0vLqbaKYtsAKUe1pcege7hm9JNtzM8+kOOzNApIX0A3xBXE6YE7g0UWjg2OaZAJXbKvALOnj2GEHKc496ykLktgNt3Jz17hprCUxFqExe7YIpQkNpO1/kfHhPUdtUAdH2/gfmeYiIFW7IkM6IBP2wrDNbMe3Mjf2ksiK3Hjghg7F2DN9l/omZZl5Mmez2QRk0q4WUUB0+1oh9nDwxGdUXJdXPMRZQs352eGaRPV9s2lcMeZFGWBfKJJiw0YgbCMLBaRmXyy4flx6a667Fch55q05QOq2Jg2ANOyZwplhNsjiohVApo7aa21QnNGW5+4GXv8gxK1beBeHSRrhmLXWVh+0aBhErZ7bx1ejxMOhlR6QU4ycNqGyk8/yNGCWkwY7/RCD7UEQek4QszCgDJAzZtfErA0VqHBy9ugQP9pUfUmgCjVYgWNwHFbhBJyEOgSwBuuwARWZmoI6J9PwLfzEocpRpPrT8DP8wqHG0b4UX+E3DiscvRglXIoi81KKPwioHI5x9EooNKWiy0KOc/T6WF4SssrRuzJ9L2VNRXUhJzj6UKYfS4W/q/5wuh/l4M9R9qsU+y2dpoo2hJzkaEET8r6KRONicnRdK9EbUi6raFVIwNGjsrlbpk6ZPi7TbS3fv3LyNjPiEKzG0aG0tvNb6xw90/whe6ONjnJcUxobHDUqQ8bIOW79BVBLBwhfSmPKdAIAAE4EAABQSwMEAAAICAAAAAAAAAAAAAAAAAAAAAAAABkABQBzaW1wbGVtb2RlbC9jb25zdGFudHMucGtsRkIBAFqAAikuUEsHCG0vCVcEAAAABAAAAFBLAwQAAAgIAAAAAAAAAAAAAAAAAAAAAAAAEwA7AHNpbXBsZW1vZGVsL3ZlcnNpb25GQjcAWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWjMKUEsHCNGeZ1UCAAAAAgAAAFBLAQIAAAAACAgAAAAAAAAhOZuwWAAAAFgAAAAUAAAAAAAAAAAAAAAAAAAAAABzaW1wbGVtb2RlbC9kYXRhLnBrbFBLAQIAABQACAgIAAAAAABUhNyGggEAACADAAAdAAAAAAAAAAAAAAAAAKgAAABzaW1wbGVtb2RlbC9jb2RlL19fdG9yY2hfXy5weVBLAQIAABQACAgIAAAAAABfSmPKdAIAAE4EAAAnAAAAAAAAAAAAAAAAAJICAABzaW1wbGVtb2RlbC9jb2RlL19fdG9yY2hfXy5weS5kZWJ1Z19wa2xQSwECAAAAAAgIAAAAAAAAbS8JVwQAAAAEAAAAGQAAAAAAAAAAAAAAAACEBQAAc2ltcGxlbW9kZWwvY29uc3RhbnRzLnBrbFBLAQIAAAAACAgAAAAAAADRnmdVAgAAAAIAAAATAAAAAAAAAAAAAAAAANQFAABzaW1wbGVtb2RlbC92ZXJzaW9uUEsGBiwAAAAAAAAAHgMtAAAAAAAAAAAABQAAAAAAAAAFAAAAAAAAAGoBAAAAAAAAUgYAAAAAAABQSwYHAAAAALwHAAAAAAAAAQAAAFBLBQYAAAAABQAFAGoBAABSBgAAAAA=", + "total_parts": 1 + } + - do: + bulk: + index: test-index-with-sparse-vector + refresh: true + body: | + {"index": {}} + {"source_text": "my words comforter", "ml.tokens":{"my":1.0, "words":1.0,"comforter":1.0}} + {"index": {}} + {"source_text": "the machine is leaking", "ml.tokens":{"the":1.0,"machine":1.0,"is":1.0,"leaking":1.0}} + {"index": {}} + {"source_text": "these are my words", "ml.tokens":{"these":1.0,"are":1.0,"my":1.0,"words":1.0}} + {"index": {}} + {"source_text": "the octopus comforter smells", "ml.tokens":{"the":1.0,"octopus":1.0,"comforter":1.0,"smells":1.0}} + {"index": {}} + {"source_text": "the octopus comforter is leaking", "ml.tokens":{"the":1.0,"octopus":1.0,"comforter":1.0,"is":1.0,"leaking":1.0}} + {"index": {}} + {"source_text": "washing machine smells", "ml.tokens":{"washing":1.0,"machine":1.0,"smells":1.0}} + {"index": { "_id": "pinned_doc1" }} + {"source_text": "unrelated pinned doc", "ml.tokens":{"unrelated":1.0,"pinned":1.0,"doc":1.0}} + {"index": { "_id": "pinned_doc2" }} + {"source_text": "another unrelated pinned doc", "ml.tokens":{"another":1.0, "unrelated":1.0,"pinned":1.0,"doc":1.0}} + + - do: + ml.start_trained_model_deployment: + model_id: text_expansion_model + wait_for: started + + - do: + query_ruleset.put: + ruleset_id: combined-ruleset + body: + rules: + - rule_id: rule1 + type: pinned + criteria: + - type: exact + metadata: foo + values: [ bar ] + actions: + ids: + - 'pinned_doc1' + - rule_id: rule2 + type: pinned + criteria: + - type: exact + metadata: foo + values: [ baz ] + actions: + docs: + - '_index': 'test-index-with-sparse-vector' + '_id': 'pinned_doc2' + - do: + search: + body: + query: + rule_query: + organic: + text_expansion: + ml.tokens: + model_id: text_expansion_model + model_text: "octopus comforter smells" + match_criteria: + foo: bar + ruleset_id: combined-ruleset + + - match: { hits.total.value: 5 } + - match: { hits.hits.0._id: 'pinned_doc1' } + + - do: + search: + body: + query: + rule_query: + organic: + text_expansion: + ml.tokens: + model_id: text_expansion_model + model_text: "octopus comforter smells" + match_criteria: + foo: baz + ruleset_id: combined-ruleset + + - match: { hits.total.value: 5 } + - match: { hits.hits.0._id: 'pinned_doc2' } + + - do: + search: + body: + query: + rule_query: + organic: + text_expansion: + ml.tokens: + model_id: text_expansion_model + model_text: "octopus comforter smells" + match_criteria: + foo: puggle + ruleset_id: combined-ruleset + + - match: { hits.total.value: 4 } + diff --git a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilder.java b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilder.java index 3882b6c61bb2c..11d2945a97354 100644 --- a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilder.java +++ b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilder.java @@ -111,18 +111,6 @@ private RuleQueryBuilder( throw new IllegalArgumentException("rulesetId must not be null or empty"); } - // PinnedQueryBuilder will return an error if we attempt to return more than the maximum number of - // pinned hits. Here, we truncate matching rules rather than return an error. - if (pinnedIds != null && pinnedIds.size() > MAX_NUM_PINNED_HITS) { - HeaderWarning.addWarning("Truncating query rule pinned hits to " + MAX_NUM_PINNED_HITS + " documents"); - pinnedIds = pinnedIds.subList(0, MAX_NUM_PINNED_HITS); - } - - if (pinnedDocs != null && pinnedDocs.size() > MAX_NUM_PINNED_HITS) { - HeaderWarning.addWarning("Truncating query rule pinned hits to " + MAX_NUM_PINNED_HITS + " documents"); - pinnedDocs = pinnedDocs.subList(0, MAX_NUM_PINNED_HITS); - } - this.organicQuery = organicQuery; this.matchCriteria = matchCriteria; this.rulesetId = rulesetId; @@ -174,6 +162,9 @@ protected void doXContent(XContentBuilder builder, Params params) throws IOExcep @Override protected Query doToQuery(SearchExecutionContext context) throws IOException { + // NOTE: this is old query logic, as in 8.12.2+ and 8.13.0+ we will always rewrite this query + // into a pinned query or the organic query. This logic remains here for backwards compatibility + // with coordinator nodes running versions 8.10.0 - 8.12.1. if ((pinnedIds != null && pinnedIds.isEmpty() == false) && (pinnedDocs != null && pinnedDocs.isEmpty() == false)) { throw new IllegalArgumentException("applied rules contain both pinned ids and pinned docs, only one of ids or docs is allowed"); } @@ -187,21 +178,25 @@ protected Query doToQuery(SearchExecutionContext context) throws IOException { } else { return organicQuery.toQuery(context); } - } - @SuppressWarnings("unchecked") @Override - protected QueryBuilder doRewrite(QueryRewriteContext queryRewriteContext) throws IOException { - if (pinnedIds != null || pinnedDocs != null) { - return this; - } else if (pinnedIdsSupplier != null || pinnedDocsSupplier != null) { - List identifiedPinnedIds = pinnedIdsSupplier != null ? pinnedIdsSupplier.get() : null; - List identifiedPinnedDocs = pinnedDocsSupplier != null ? pinnedDocsSupplier.get() : null; - if (identifiedPinnedIds == null && identifiedPinnedDocs == null) { - return this; // not executed yet + protected QueryBuilder doRewrite(QueryRewriteContext queryRewriteContext) { + if (pinnedIdsSupplier != null && pinnedDocsSupplier != null) { + List identifiedPinnedIds = pinnedIdsSupplier.get(); + List identifiedPinnedDocs = pinnedDocsSupplier.get(); + if (identifiedPinnedIds == null || identifiedPinnedDocs == null) { + return this; // Not executed yet + } else if (identifiedPinnedIds.isEmpty() && identifiedPinnedDocs.isEmpty()) { + return organicQuery; // Nothing to pin here + } else if (identifiedPinnedIds.isEmpty() == false && identifiedPinnedDocs.isEmpty() == false) { + throw new IllegalArgumentException( + "applied rules contain both pinned ids and pinned docs, only one of ids or docs is allowed" + ); + } else if (identifiedPinnedIds.isEmpty() == false) { + return new PinnedQueryBuilder(organicQuery, truncateList(identifiedPinnedIds).toArray(new String[0])); } else { - return new RuleQueryBuilder(organicQuery, matchCriteria, rulesetId, identifiedPinnedIds, identifiedPinnedDocs, null, null); + return new PinnedQueryBuilder(organicQuery, truncateList(identifiedPinnedDocs).toArray(new Item[0])); } } @@ -244,18 +239,19 @@ public void onFailure(Exception e) { }); }); - QueryBuilder newOrganicQuery = organicQuery.rewrite(queryRewriteContext); - RuleQueryBuilder rewritten = new RuleQueryBuilder( - newOrganicQuery, - matchCriteria, - this.rulesetId, - null, - null, - pinnedIdsSetOnce::get, - pinnedDocsSetOnce::get - ); - rewritten.boost(this.boost); - return rewritten; + return new RuleQueryBuilder(organicQuery, matchCriteria, this.rulesetId, null, null, pinnedIdsSetOnce::get, pinnedDocsSetOnce::get) + .boost(this.boost) + .queryName(this.queryName); + } + + private List truncateList(List input) { + // PinnedQueryBuilder will return an error if we attempt to return more than the maximum number of + // pinned hits. Here, we truncate matching rules rather than return an error. + if (input.size() > MAX_NUM_PINNED_HITS) { + HeaderWarning.addWarning("Truncating query rule pinned hits to " + MAX_NUM_PINNED_HITS + " documents"); + return input.subList(0, MAX_NUM_PINNED_HITS); + } + return input; } @Override diff --git a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilderTests.java b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilderTests.java index 1d19011fdd4a9..d4ab8d8f8e6e8 100644 --- a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilderTests.java +++ b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/rules/RuleQueryBuilderTests.java @@ -31,6 +31,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xpack.application.LocalStateEnterpriseSearch; +import org.elasticsearch.xpack.searchbusinessrules.SearchBusinessRules; import org.hamcrest.Matchers; import java.io.IOException; @@ -62,7 +63,7 @@ protected void doAssertLuceneQuery(RuleQueryBuilder queryBuilder, Query query, S @Override protected Collection> getPlugins() { - return Collections.singletonList(LocalStateEnterpriseSearch.class); + return List.of(LocalStateEnterpriseSearch.class, SearchBusinessRules.class); } public void testIllegalArguments() { From 2e5875de8b489360ab2b61c4cab7b387988c4178 Mon Sep 17 00:00:00 2001 From: Tim Grein Date: Thu, 15 Feb 2024 16:32:01 +0100 Subject: [PATCH 03/45] [Connectors Secrets API] Move null checks from constructor to validate --- .../secrets/ConnectorSecretsConstants.java | 14 ++++++++++++++ .../action/DeleteConnectorSecretRequest.java | 8 ++++++-- .../secrets/action/GetConnectorSecretRequest.java | 6 +++++- .../action/DeleteConnectorSecretActionTests.java | 12 +++++++++++- .../action/GetConnectorSecretActionTests.java | 11 ++++++++++- 5 files changed, 46 insertions(+), 5 deletions(-) create mode 100644 x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/ConnectorSecretsConstants.java diff --git a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/ConnectorSecretsConstants.java b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/ConnectorSecretsConstants.java new file mode 100644 index 0000000000000..9c7f192c19248 --- /dev/null +++ b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/ConnectorSecretsConstants.java @@ -0,0 +1,14 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.application.connector.secrets; + +public class ConnectorSecretsConstants { + + public static final String CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE = "[id] of the connector secret cannot be null or empty."; + +} diff --git a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretRequest.java b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretRequest.java index 183362f64ea8f..efc8b2b6c660d 100644 --- a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretRequest.java +++ b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretRequest.java @@ -12,6 +12,7 @@ import org.elasticsearch.common.Strings; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.xpack.application.connector.secrets.ConnectorSecretsConstants; import java.io.IOException; import java.util.Objects; @@ -23,7 +24,7 @@ public class DeleteConnectorSecretRequest extends ActionRequest { private final String id; public DeleteConnectorSecretRequest(String id) { - this.id = Objects.requireNonNull(id); + this.id = id; } public DeleteConnectorSecretRequest(StreamInput in) throws IOException { @@ -46,7 +47,10 @@ public ActionRequestValidationException validate() { ActionRequestValidationException validationException = null; if (Strings.isNullOrEmpty(id)) { - validationException = addValidationError("id missing", validationException); + validationException = addValidationError( + ConnectorSecretsConstants.CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE, + validationException + ); } return validationException; diff --git a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretRequest.java b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretRequest.java index cf1cc0f563eba..e8792dd3755bd 100644 --- a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretRequest.java +++ b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretRequest.java @@ -12,6 +12,7 @@ import org.elasticsearch.common.Strings; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.xpack.application.connector.secrets.ConnectorSecretsConstants; import java.io.IOException; import java.util.Objects; @@ -46,7 +47,10 @@ public ActionRequestValidationException validate() { ActionRequestValidationException validationException = null; if (Strings.isNullOrEmpty(id)) { - validationException = addValidationError("id missing", validationException); + validationException = addValidationError( + ConnectorSecretsConstants.CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE, + validationException + ); } return validationException; diff --git a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretActionTests.java b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretActionTests.java index 5d9127527fc3a..503aa0a6304cd 100644 --- a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretActionTests.java +++ b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/DeleteConnectorSecretActionTests.java @@ -9,8 +9,10 @@ import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.xpack.application.connector.secrets.ConnectorSecretsConstants; import org.elasticsearch.xpack.application.connector.secrets.ConnectorSecretsTestUtils; +import static org.elasticsearch.xpack.application.connector.ConnectorTestUtils.NULL_STRING; import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.notNullValue; import static org.hamcrest.Matchers.nullValue; @@ -29,6 +31,14 @@ public void testValidate_WhenConnectorSecretIdIsEmpty_ExpectValidationError() { ActionRequestValidationException exception = requestWithMissingConnectorId.validate(); assertThat(exception, notNullValue()); - assertThat(exception.getMessage(), containsString("id missing")); + assertThat(exception.getMessage(), containsString(ConnectorSecretsConstants.CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE)); + } + + public void testValidate_WhenConnectorSecretIdIsNull_ExpectValidationError() { + DeleteConnectorSecretRequest requestWithMissingConnectorId = new DeleteConnectorSecretRequest(NULL_STRING); + ActionRequestValidationException exception = requestWithMissingConnectorId.validate(); + + assertThat(exception, notNullValue()); + assertThat(exception.getMessage(), containsString(ConnectorSecretsConstants.CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE)); } } diff --git a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretActionTests.java b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretActionTests.java index 9fc01e56ee5a0..c60aeffa2911a 100644 --- a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretActionTests.java +++ b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/secrets/action/GetConnectorSecretActionTests.java @@ -9,6 +9,7 @@ import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.xpack.application.connector.secrets.ConnectorSecretsConstants; import org.elasticsearch.xpack.application.connector.secrets.ConnectorSecretsTestUtils; import static org.hamcrest.Matchers.containsString; @@ -29,6 +30,14 @@ public void testValidate_WhenConnectorSecretIdIsEmpty_ExpectValidationError() { ActionRequestValidationException exception = requestWithMissingConnectorId.validate(); assertThat(exception, notNullValue()); - assertThat(exception.getMessage(), containsString("id missing")); + assertThat(exception.getMessage(), containsString(ConnectorSecretsConstants.CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE)); + } + + public void testValidate_WhenConnectorSecretIdIsNull_ExpectValidationError() { + GetConnectorSecretRequest requestWithMissingConnectorId = new GetConnectorSecretRequest(""); + ActionRequestValidationException exception = requestWithMissingConnectorId.validate(); + + assertThat(exception, notNullValue()); + assertThat(exception.getMessage(), containsString(ConnectorSecretsConstants.CONNECTOR_SECRET_ID_NULL_OR_EMPTY_MESSAGE)); } } From 2e58cb91a575550d6b8fc4b09a1f3cb60ec81ed2 Mon Sep 17 00:00:00 2001 From: Ignacio Vera Date: Thu, 15 Feb 2024 16:39:22 +0100 Subject: [PATCH 04/45] Reduce InternalBinaryRange and InternalRandomSampler in a streaming fashion (#105541) --- .../bucket/range/InternalBinaryRange.java | 60 +++++++------------ .../sampler/random/InternalRandomSampler.java | 16 +++-- 2 files changed, 31 insertions(+), 45 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalBinaryRange.java b/server/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalBinaryRange.java index f81a5ffbcaf18..2b5bcd9931f6e 100644 --- a/server/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalBinaryRange.java +++ b/server/src/main/java/org/elasticsearch/search/aggregations/bucket/range/InternalBinaryRange.java @@ -12,18 +12,18 @@ import org.elasticsearch.TransportVersions; import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.core.Releasables; import org.elasticsearch.search.DocValueFormat; import org.elasticsearch.search.aggregations.AggregationReduceContext; import org.elasticsearch.search.aggregations.AggregatorReducer; import org.elasticsearch.search.aggregations.InternalAggregation; import org.elasticsearch.search.aggregations.InternalAggregations; import org.elasticsearch.search.aggregations.InternalMultiBucketAggregation; +import org.elasticsearch.search.aggregations.bucket.FixedMultiBucketAggregatorsReducer; import org.elasticsearch.search.aggregations.support.SamplingContext; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; -import java.util.ArrayList; -import java.util.Arrays; import java.util.List; import java.util.Map; import java.util.Objects; @@ -248,50 +248,32 @@ protected AggregatorReducer getLeaderReducer(AggregationReduceContext reduceCont return new AggregatorReducer() { - final List aggregations = new ArrayList<>(size); + final FixedMultiBucketAggregatorsReducer reducer = new FixedMultiBucketAggregatorsReducer<>( + reduceContext, + size, + getBuckets() + ) { + + @Override + protected Bucket createBucket(Bucket proto, long docCount, InternalAggregations aggregations) { + return new Bucket(proto.format, proto.keyed, proto.key, proto.from, proto.to, docCount, aggregations); + } + }; @Override public void accept(InternalAggregation aggregation) { - aggregations.add((InternalBinaryRange) aggregation); + InternalBinaryRange binaryRange = (InternalBinaryRange) aggregation; + reducer.accept(binaryRange.getBuckets()); } @Override public InternalAggregation get() { - reduceContext.consumeBucketsAndMaybeBreak(buckets.size()); - long[] docCounts = new long[buckets.size()]; - InternalAggregations[][] aggs = new InternalAggregations[buckets.size()][]; - for (int i = 0; i < aggs.length; ++i) { - aggs[i] = new InternalAggregations[aggregations.size()]; - } - for (int i = 0; i < aggregations.size(); ++i) { - InternalBinaryRange range = aggregations.get(i); - if (range.buckets.size() != buckets.size()) { - throw new IllegalStateException( - "Expected [" + buckets.size() + "] buckets, but got [" + range.buckets.size() + "]" - ); - } - for (int j = 0; j < buckets.size(); ++j) { - Bucket bucket = range.buckets.get(j); - docCounts[j] += bucket.docCount; - aggs[j][i] = bucket.aggregations; - } - } - List buckets = new ArrayList<>(getBuckets().size()); - for (int i = 0; i < getBuckets().size(); ++i) { - Bucket b = getBuckets().get(i); - buckets.add( - new Bucket( - format, - keyed, - b.key, - b.from, - b.to, - docCounts[i], - InternalAggregations.reduce(Arrays.asList(aggs[i]), reduceContext) - ) - ); - } - return new InternalBinaryRange(name, format, keyed, buckets, metadata); + return new InternalBinaryRange(name, format, keyed, reducer.get(), metadata); + } + + @Override + public void close() { + Releasables.close(reducer); } }; } diff --git a/server/src/main/java/org/elasticsearch/search/aggregations/bucket/sampler/random/InternalRandomSampler.java b/server/src/main/java/org/elasticsearch/search/aggregations/bucket/sampler/random/InternalRandomSampler.java index c17056cecc053..4dde9cc67b975 100644 --- a/server/src/main/java/org/elasticsearch/search/aggregations/bucket/sampler/random/InternalRandomSampler.java +++ b/server/src/main/java/org/elasticsearch/search/aggregations/bucket/sampler/random/InternalRandomSampler.java @@ -10,8 +10,10 @@ import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.core.Releasables; import org.elasticsearch.search.aggregations.AggregationReduceContext; import org.elasticsearch.search.aggregations.AggregatorReducer; +import org.elasticsearch.search.aggregations.AggregatorsReducer; import org.elasticsearch.search.aggregations.InternalAggregation; import org.elasticsearch.search.aggregations.InternalAggregations; import org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation; @@ -20,8 +22,6 @@ import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; -import java.util.ArrayList; -import java.util.List; import java.util.Map; public class InternalRandomSampler extends InternalSingleBucketAggregation implements Sampler { @@ -79,24 +79,28 @@ protected InternalSingleBucketAggregation newAggregation(String name, long docCo protected AggregatorReducer getLeaderReducer(AggregationReduceContext reduceContext, int size) { return new AggregatorReducer() { long docCount = 0L; - final List subAggregationsList = new ArrayList<>(size); + final AggregatorsReducer subAggregatorReducer = new AggregatorsReducer(reduceContext, size); @Override public void accept(InternalAggregation aggregation) { docCount += ((InternalSingleBucketAggregation) aggregation).getDocCount(); - subAggregationsList.add(((InternalSingleBucketAggregation) aggregation).getAggregations()); + subAggregatorReducer.accept(((InternalSingleBucketAggregation) aggregation).getAggregations()); } @Override public InternalAggregation get() { - InternalAggregations aggs = InternalAggregations.reduce(subAggregationsList, reduceContext); + InternalAggregations aggs = subAggregatorReducer.get(); if (reduceContext.isFinalReduce() && aggs != null) { SamplingContext context = buildContext(); aggs = InternalAggregations.from(aggs.asList().stream().map(agg -> agg.finalizeSampling(context)).toList()); } - return newAggregation(getName(), docCount, aggs); } + + @Override + public void close() { + Releasables.close(subAggregatorReducer); + } }; } From a8d2722c5969b19cb442b23fbf4c09436bfc20d5 Mon Sep 17 00:00:00 2001 From: Mark Tozzi Date: Thu, 15 Feb 2024 10:39:47 -0500 Subject: [PATCH 05/45] [ESQL] Inverse trig function argument descriptions (#105525) --- .../qa/testFixtures/src/main/resources/show.csv-spec | 10 +++++----- .../esql/expression/function/scalar/math/Acos.java | 5 ++++- .../esql/expression/function/scalar/math/Asin.java | 5 ++++- .../esql/expression/function/scalar/math/Atan.java | 5 ++++- .../esql/expression/function/scalar/math/Atan2.java | 4 ++-- 5 files changed, 19 insertions(+), 10 deletions(-) diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/show.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/show.csv-spec index 8d86563f78938..6887a1bbe9069 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/show.csv-spec +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/show.csv-spec @@ -6,15 +6,15 @@ v:long ; # TODO: switch this test to ``&format=csv&delimiter=|` output -showFunctions#[skip:-8.12.99] +showFunctions#[skip:-8.13.99] show functions; name:keyword | synopsis:keyword | argNames:keyword | argTypes:keyword | argDescriptions:keyword |returnType:keyword | description:keyword | optionalArgs:boolean | variadic:boolean | isAggregation:boolean abs |"double|integer|long|unsigned_long abs(n:double|integer|long|unsigned_long)" |n |"double|integer|long|unsigned_long" | "" |"double|integer|long|unsigned_long" | "Returns the absolute value." | false | false | false -acos |"double acos(n:double|integer|long|unsigned_long)" |n |"double|integer|long|unsigned_long" | "" |double | "The arccosine of an angle, expressed in radians." | false | false | false -asin |"double asin(n:double|integer|long|unsigned_long)"|n |"double|integer|long|unsigned_long" | "" |double | "Inverse sine trigonometric function." | false | false | false -atan |"double atan(n:double|integer|long|unsigned_long)" |n |"double|integer|long|unsigned_long" | "" |double | "Inverse tangent trigonometric function." | false | false | false -atan2 |"double atan2(y:double|integer|long|unsigned_long, x:double|integer|long|unsigned_long)" |[y, x] |["double|integer|long|unsigned_long", "double|integer|long|unsigned_long"] |["", ""] |double | "The angle between the positive x-axis and the ray from the origin to the point (x , y) in the Cartesian plane." | [false, false] | false | false +acos |"double acos(n:double|integer|long|unsigned_long)" |n |"double|integer|long|unsigned_long" | "Number between -1 and 1" |double | "The arccosine of an angle, expressed in radians." | false | false | false +asin |"double asin(n:double|integer|long|unsigned_long)"|n |"double|integer|long|unsigned_long" | "Number between -1 and 1" |double | "Inverse sine trigonometric function." | false | false | false +atan |"double atan(n:double|integer|long|unsigned_long)" |n |"double|integer|long|unsigned_long" | "A number" |double | "Inverse tangent trigonometric function." | false | false | false +atan2 |"double atan2(y:double|integer|long|unsigned_long, x:double|integer|long|unsigned_long)" |[y, x] |["double|integer|long|unsigned_long", "double|integer|long|unsigned_long"] |["y coordinate", "x coordinate"] |double | "The angle between the positive x-axis and the ray from the origin to the point (x , y) in the Cartesian plane." | [false, false] | false | false auto_bucket |"double|date auto_bucket(field:integer|long|double|date, buckets:integer, from:integer|long|double|date|string, to:integer|long|double|date|string)" |[field, buckets, from, to] |["integer|long|double|date", "integer", "integer|long|double|date|string", "integer|long|double|date|string"] |["", "", "", ""] | "double|date" | "Creates human-friendly buckets and returns a datetime value for each row that corresponds to the resulting bucket the row falls into." | [false, false, false, false] | false | false avg |"double avg(field:double|integer|long)" |field |"double|integer|long" | "" |double | "The average of a numeric field." | false | false | true case |"boolean|cartesian_point|date|double|geo_point|integer|ip|keyword|long|text|unsigned_long|version case(condition:boolean, rest...:boolean|cartesian_point|date|double|geo_point|integer|ip|keyword|long|text|unsigned_long|version)" |[condition, rest] |["boolean", "boolean|cartesian_point|date|double|geo_point|integer|ip|keyword|long|text|unsigned_long|version"] |["", ""] |"boolean|cartesian_point|date|double|geo_point|integer|ip|keyword|long|text|unsigned_long|version" | "Accepts pairs of conditions and values. The function returns the value that belongs to the first condition that evaluates to true." | [false, false] | true | false diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Acos.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Acos.java index 603ef86af6c64..d2e0e8f025665 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Acos.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Acos.java @@ -22,7 +22,10 @@ */ public class Acos extends AbstractTrigonometricFunction { @FunctionInfo(returnType = "double", description = "The arccosine of an angle, expressed in radians.") - public Acos(Source source, @Param(name = "n", type = { "double", "integer", "long", "unsigned_long" }) Expression n) { + public Acos( + Source source, + @Param(name = "n", type = { "double", "integer", "long", "unsigned_long" }, description = "Number between -1 and 1") Expression n + ) { super(source, n); } diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Asin.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Asin.java index f66409921ad2f..38b70cea0350c 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Asin.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Asin.java @@ -22,7 +22,10 @@ */ public class Asin extends AbstractTrigonometricFunction { @FunctionInfo(returnType = "double", description = "Inverse sine trigonometric function.") - public Asin(Source source, @Param(name = "n", type = { "double", "integer", "long", "unsigned_long" }) Expression n) { + public Asin( + Source source, + @Param(name = "n", type = { "double", "integer", "long", "unsigned_long" }, description = "Number between -1 and 1") Expression n + ) { super(source, n); } diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan.java index 8f0ad96f96e8c..071379820922a 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan.java @@ -22,7 +22,10 @@ */ public class Atan extends AbstractTrigonometricFunction { @FunctionInfo(returnType = "double", description = "Inverse tangent trigonometric function.") - public Atan(Source source, @Param(name = "n", type = { "double", "integer", "long", "unsigned_long" }) Expression n) { + public Atan( + Source source, + @Param(name = "n", type = { "double", "integer", "long", "unsigned_long" }, description = "A number") Expression n + ) { super(source, n); } diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan2.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan2.java index 5b0a795996440..b69a536c2df84 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan2.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/math/Atan2.java @@ -38,8 +38,8 @@ public class Atan2 extends EsqlScalarFunction { ) public Atan2( Source source, - @Param(name = "y", type = { "double", "integer", "long", "unsigned_long" }) Expression y, - @Param(name = "x", type = { "double", "integer", "long", "unsigned_long" }) Expression x + @Param(name = "y", type = { "double", "integer", "long", "unsigned_long" }, description = "y coordinate") Expression y, + @Param(name = "x", type = { "double", "integer", "long", "unsigned_long" }, description = "x coordinate") Expression x ) { super(source, List.of(y, x)); this.y = y; From 15877f559320f7dcc5c655e24375d009643cd0ec Mon Sep 17 00:00:00 2001 From: Brian Seeders Date: Thu, 15 Feb 2024 10:42:37 -0500 Subject: [PATCH 06/45] Renew samba fixture expired test certificates (#105561) --- .../transport/ssl/certs/simple/samba4.crt | 36 ++++++------- .../security/authc/ldap/support/ADtrust.jks | Bin 1922 -> 1923 bytes .../security/authc/ldap/support/smb_ca.crt | 36 ++++++------- .../security/authc/ldap/support/smb_cert.crt | 36 ++++++------- .../src/main/resources/smb/certs/ca.key | 50 +++++++++--------- .../src/main/resources/smb/certs/ca.pem | 36 ++++++------- .../src/main/resources/smb/certs/cert.pem | 36 ++++++------- .../src/main/resources/smb/certs/key.pem | 50 +++++++++--------- 8 files changed, 140 insertions(+), 140 deletions(-) diff --git a/x-pack/plugin/core/src/test/resources/org/elasticsearch/xpack/security/transport/ssl/certs/simple/samba4.crt b/x-pack/plugin/core/src/test/resources/org/elasticsearch/xpack/security/transport/ssl/certs/simple/samba4.crt index a1f78231fa3c4..aba0948092581 100644 --- a/x-pack/plugin/core/src/test/resources/org/elasticsearch/xpack/security/transport/ssl/certs/simple/samba4.crt +++ b/x-pack/plugin/core/src/test/resources/org/elasticsearch/xpack/security/transport/ssl/certs/simple/samba4.crt @@ -1,20 +1,20 @@ -----BEGIN CERTIFICATE----- -MIIDTzCCAjegAwIBAgIVAKswcWOE2Y8orNolWoM5EJbBVQKAMA0GCSqGSIb3DQEB -CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu -ZXJhdGVkIENBMB4XDTIxMDIxNTExNDE1NVoXDTI0MDIxNTExNDE1NVowETEPMA0G -A1UEAxMGc2FtYmE0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoNwa -9+QyLYF7q8jFOmd/1WNnV50KAnfBXti9WShk1rTso1YK3QflWtqcn0BsJFfUmRJ1 -79tTTtUmzcqcXXR/ygc8UAal/Cl19h1G4k3Mwf8tlUAjRbH0D+mM6HrZ4+QrAqwb -tDezz8TBAEuVJmTcbdrI6iA9y+I4PReqF//EyQwHeUbKvNHBF0vqYFHqErrBaKsC -VAFp9QSKjF9uRLk4e7PLJDZm9BbVTbtQ76o2SHNMni0UcriqYbi6npIWCLs5xFg5 -dSJ3SwrgBbm/Hg994miPt5CN7l69e7h3ul6/Af3UfMTM02YQ9T3mQ4evW667kAoW -Gxu2Z1cXcsq9+Qj/TwIDAQABo3sweTAdBgNVHQ4EFgQUN818rPYPElgO2oFSXM0c -GXvD4cAwHwYDVR0jBBgwFoAUdiZF7wZYdLDjDO6NuDRebJFFFnIwLAYDVR0RBCUw -I4IJbG9jYWxob3N0hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAkGA1UdEwQCMAAw -DQYJKoZIhvcNAQELBQADggEBAGZQ+NnsLNLbS5dvU2XQDrstcgq7JQcoajH+Us86 -/arjs9C1T7GID1wCXSlyR2uuFTYBH89UpX3oChLohnJByy1Mhpi8l+R2odhosgym -psnt7uejh9DWQrPeiL00ohSjXy3dha5VUESAKoyT/rga1YNl4qeY1J7RPM4NkP7l -nJFzJkTpdcfgcDj42OEOKRjSxaGKTQu5OP6/EXmpxxUdXpfoRfP0IiyCHSNl2gfJ -zxFvVlB2cy8O0EnJ4DdpRpzeTW8pMXGZ7flUSvOhT2rn2K1eqRL9q9LRJn/G/5DP -0u2HxjrZSfn40AS1jcasy6Lcvc7Mwz4W0cfc/tSA67WrFzw= +MIIDTjCCAjagAwIBAgIUJFzoNwmi+322wZTgemjEV35/yrIwDQYJKoZIhvcNAQEL +BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l +cmF0ZWQgQ0EwHhcNMjQwMjE1MTQ0ODM3WhcNMjcwMjE0MTQ0ODM3WjARMQ8wDQYD +VQQDEwZzYW1iYTQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFrD2n +loHuTCCbcX1pqmRnVj7SXcXsBjY9McXxvpEQ4kCdUBlDp3wC9bkGnt6jxlAvEaXn +ZXHg3DqJbD2GU9vRMapcVR4ptAvZmgpEJBU8FA8E1T1EeMhhIoWsYq2crOf/SLQY +g23tO1vmV5+kPD23KiTzj8errYF1Si/6b0IH7XuRUAUJAJYm4gN3YvNqIUYxO/WG +cOlxp/ssIC7Xj6E8Mcdpcfgm+AzLR/x2edl524GAbLqzuKM2bUoMroeNVwVRYPz5 +9iCi+rRxAlI9M2eIm7Gl+qjXUXN1BQAEpkXcxBxDWaIEomd7rR9yudHWvBl8J3+V +jJBgqhecOq/Z+bqbAgMBAAGjezB5MB0GA1UdDgQWBBSbPXW27/QMTZuLD13NUSmQ +0kxaCDAfBgNVHSMEGDAWgBTZMPxziaiEFpaGeNGBxTr+sWGeXTAsBgNVHREEJTAj +hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABgglsb2NhbGhvc3QwCQYDVR0TBAIwADAN +BgkqhkiG9w0BAQsFAAOCAQEAkbnWUXSooivT8FW2PfdTLYDBi18ssdH5g2KU1fNi +E0z+85X1IZaWikGZi3fyj28+GPvRr588EmpPP+IIW9WM5rx3/Gh7elknpwTBYwPo +whuLrKPofEXeM1nUi6nC9QfKDoLWC4B2ztXLD+gcqVHfO/3Q9cs88z9IK4fMAlHu +2dof6SlqVI7AAsqCHaIIuiABW3U81URdtwCJqKt2SUQmpRh6PyAT8rqP7f//gBFF +KyLGOhlEikm+DpFvLNtBHFLL/u9P6u9zV5xyqI4K3wGMAvrTbpicXLJQC6bdIEJR +gFSUL8f6cal5l0ncN6dT3RnpAWwt88xzg8OSMDPWYAyHbw== -----END CERTIFICATE----- diff --git a/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/ADtrust.jks b/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/ADtrust.jks index ab828467407526b0e43fb2e265643beafde020e9..b497bdbd0585c87cd8903bbbc722da79429fd63d 100644 GIT binary patch delta 717 zcmV;;0y6!A4}%Yo8#IlrK3Mty1z0XMFgXAK15Pl414=N00y3Zj0s#U76#(mH6@=ac zg)S`xbvsy*CupQ#IkuX|syJ}S~pB8NFO*2R-{YWi=dXpE&#>kDxvyZ4!G07ywwXz#LuT%KUc1Zw z&4kU~^(K>vcjgKAp!F8ow1;7gR68;MNJ5i+bOBtY7H*l7GMg2@$66LGZ1^?ytfkP`=YmJt z?|4@zNF`9e)INC!G3R`Xh2?cl^~u_bs%)u2SbF5VtUaQJgre$!SA!d#dVdB>YR>`# z0RRD`Q!r659R>qc9S#H*1QgjY{Bwz@gcg>Dc+r8yI{vX?o?S2>1_MP3&x+X-ypPMfE(BeEKZ`m^Y$+)XmexE z-?1>)H{97ulw#Lra9i;!Md1>h8w7k#P?VT4V}4M&8iJjirLgS5nO%#s26qsy*|NXQ zh>@~&%*KW)u$F_UzXzb@d;?vTiK-O|23~5d4rhk zZo4`EI9f!Ggc^gYql`{BpS5}dJfX9A1NH+Ht^Cb|iBQg;AC4iFB8Kr&z9B+@@OMf4 delta 716 zcmZqXZ{nXIZCt)|e!d|CYlNPufh7Y2v#&uDv!_85li>npCPpSEkyOj}hVv1c^=dn= zoxjnhv_D(E>dZucFKI&qBSTX|Lt_gAvnX+1BNHU<#5nVbuNCUg7O%)w%VuAfJl$s* zzl!e7w@eKeu4_&BIsJE{_ScI-dnErhE3b{2rWkwEeEFB<U*x- zCwP3}yw*C_4_3U5ZPR8RP>E)L@~qY2saDG5SG=!F<5pCi3hOgqX+Ci(tNAr^7Jq>b z>r#vA1CKluUc}vuI2&_Fquz?YV@L8+v&n^9wX|PLY9A@ymUH#en?63>#IL(^R3_}+ z)BQb5ZduRyZ#%@r=L_wX-_aeh?#wMRa{`Vx2Pf$ zGb01z;$VY716elaP+2|}F&2?BHP`oS5hWWQ^StZbVG@@!(N(O-KprHm%pzeR)__&P zXx z+LG~9Lzv)-S!vZ<9(W!-H1AcDw1X7)M57mP!`d<~PTy8PJ=dpHiQ90S#n}t7$GT31 zNd-Nut-kCY9B5nge7W|w%*-!U1#?wha`volFE}0UwInri?kk-Ocj{g~RC@Z`*r@u6 z&06Ez2OQknj6&JxM%wk3T->pgMOs_yhSdS~19PH}{#+|~XpTpgY3VH4Wmg|u43bv( w82O>yHemA(clGYG|C#zVH&-%$V-{KXc=w{7dBSgQ{0y~ZUHbK4&v&;#07ip77ytkO diff --git a/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_ca.crt b/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_ca.crt index df7b319d1e313..a3281c92af704 100644 --- a/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_ca.crt +++ b/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_ca.crt @@ -1,20 +1,20 @@ -----BEGIN CERTIFICATE----- -MIIDSTCCAjGgAwIBAgIUZTmHgM9YKX8muNbP2IYiv2sfeswwDQYJKoZIhvcNAQEL -BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l -cmF0ZWQgQ0EwHhcNMjEwMjE1MTEzODA2WhcNMjQwMjE1MTEzODA2WjA0MTIwMAYD -VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC -ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM1zqGsmaweuY5dMpg8kLdnt -AoDQ1yqQ+Zf7YSv10RK8Gf2DI61cliFd2Ten9Kd3RBg5RKtEXxHDiQkvSam+Eceh -noV+BfA6DYGGlpnAJFsH5OaFQOUqZJPqDet1Xqh6ylaOMASDyMpqg+sDag9wLAWl -OHvA4kgg6F7ZWM1cwig6D4i4Y+U2k3G1KivrGSvEc7Zs1dLsjg4tYfW7bCSQu7yL -92oepozP9rgXF58SvR+4i1iu7P1r769R1WUMCHJ98f9rXDHHdsS1WMfTsuTP9h2t -TJSBGkrQKtZ0v+7oe2yKArFGF+U9FuZVD1D2Fc4dTDMxpCALF6TeIK6FFxC7FSEC -AwEAAaNTMFEwHQYDVR0OBBYEFHYmRe8GWHSw4wzujbg0XmyRRRZyMB8GA1UdIwQY -MBaAFHYmRe8GWHSw4wzujbg0XmyRRRZyMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI -hvcNAQELBQADggEBADZ5ddmAODL7QeZHkUrUKtGkMWK0WpdlQKx3XkxDod4DM8cI -OaKVUhJznt5ShPCMYZ3+EbTAZhaJUnxdUrulTh8UW0ezDFcnninkzVJ1K2tlYurF -CgrOpr+ni1T08pWrOQHKgFYRqJpme7TgScXCnuqCG0AaC5Ey6O1WhmjRl7aXbUx1 -IgsxtjjN0F3GispWGlLhfXvTR1NRPXLnpyv2aWn0enCdJURsvKuHcMtXSqRlWZ3q -LNDcfunhIuX7MzJ75DytM9vAQEeGMlUHnVk+jXTRuKUEGysq2DrAB8CcW8X5rRHC -nEhqNXWaHabV4NFSGyDxWfCHPVCzuEcni83/jyk= +MIIDSjCCAjKgAwIBAgIVAOtlFYTeA4VeumO8mgMu/nMuJRxRMA0GCSqGSIb3DQEB +CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu +ZXJhdGVkIENBMB4XDTI0MDIxNTE0NDc0NFoXDTI3MDIxNDE0NDc0NFowNDEyMDAG +A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew +ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDXFQRa1KKGfLNaRbiwHeK2 +kTBab7tqVS6fFmzsTTNIKf1ILYF6kxfHxsiN1jM8tVU5n0Kirqe8eRVAQn/4Hj09 +I8dhVLmvYI/BfioZWU0MXLk0aA2nUm5dk9eVkLaZSaVDZu57XrvL/s2Ezd71JpOJ +d+l9zQLNxrhvhTgw3sV53KURQip8UoisSXjbyBG5eq3Ck1DfdZ0lrXoTM/cgGp3i +yMk/oPUW2rSHYYxUOzH/SEKTfXQBXKUWbpmTMpsVv8daFi1s+DX2rKXQ1+eCR9rv +eFcoSCVQv9Q+eQcx53yLheV1TvXJ2oqqbKlBWHrkvKw9ooaEouqBV4Mbnnp/BmrP +AgMBAAGjUzBRMB0GA1UdDgQWBBTZMPxziaiEFpaGeNGBxTr+sWGeXTAfBgNVHSME +GDAWgBTZMPxziaiEFpaGeNGBxTr+sWGeXTAPBgNVHRMBAf8EBTADAQH/MA0GCSqG +SIb3DQEBCwUAA4IBAQARiXwCC8afr98gtGGAG9gJLE6ivfP2Lypoc2PO37Ew1zfc +2UqUYtdmcFvxK0XhEpwbBHxOUJSYMWN+ULoagp2cpbDswZldi7MGdxCu2bK/zYiR +snXMxoYqsJaDqL8HoOV8A12ViaoVCQZeaq0OZ4/NrTRCcPzx+Yru9ZWfgG+4gDbC +hxnDJBY81ju8fhfAcCGuxApPE9fkwFwwVr5nwCbAl7Qotliol/h1eV35hPocyUVz +AxOW2Foc7ehru6NynnEAqOhjcTwMlZCKt0b4kPz2kd+9vL1ZKc/hjNX6Q6sW1N6V +z9Gu6uSsIiM8brs5/zhaRI6EGoOqo4xON5+1ejyh -----END CERTIFICATE----- diff --git a/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_cert.crt b/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_cert.crt index a1f78231fa3c4..aba0948092581 100644 --- a/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_cert.crt +++ b/x-pack/plugin/security/src/test/resources/org/elasticsearch/xpack/security/authc/ldap/support/smb_cert.crt @@ -1,20 +1,20 @@ -----BEGIN CERTIFICATE----- -MIIDTzCCAjegAwIBAgIVAKswcWOE2Y8orNolWoM5EJbBVQKAMA0GCSqGSIb3DQEB -CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu -ZXJhdGVkIENBMB4XDTIxMDIxNTExNDE1NVoXDTI0MDIxNTExNDE1NVowETEPMA0G -A1UEAxMGc2FtYmE0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoNwa -9+QyLYF7q8jFOmd/1WNnV50KAnfBXti9WShk1rTso1YK3QflWtqcn0BsJFfUmRJ1 -79tTTtUmzcqcXXR/ygc8UAal/Cl19h1G4k3Mwf8tlUAjRbH0D+mM6HrZ4+QrAqwb -tDezz8TBAEuVJmTcbdrI6iA9y+I4PReqF//EyQwHeUbKvNHBF0vqYFHqErrBaKsC -VAFp9QSKjF9uRLk4e7PLJDZm9BbVTbtQ76o2SHNMni0UcriqYbi6npIWCLs5xFg5 -dSJ3SwrgBbm/Hg994miPt5CN7l69e7h3ul6/Af3UfMTM02YQ9T3mQ4evW667kAoW -Gxu2Z1cXcsq9+Qj/TwIDAQABo3sweTAdBgNVHQ4EFgQUN818rPYPElgO2oFSXM0c -GXvD4cAwHwYDVR0jBBgwFoAUdiZF7wZYdLDjDO6NuDRebJFFFnIwLAYDVR0RBCUw -I4IJbG9jYWxob3N0hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAkGA1UdEwQCMAAw -DQYJKoZIhvcNAQELBQADggEBAGZQ+NnsLNLbS5dvU2XQDrstcgq7JQcoajH+Us86 -/arjs9C1T7GID1wCXSlyR2uuFTYBH89UpX3oChLohnJByy1Mhpi8l+R2odhosgym -psnt7uejh9DWQrPeiL00ohSjXy3dha5VUESAKoyT/rga1YNl4qeY1J7RPM4NkP7l -nJFzJkTpdcfgcDj42OEOKRjSxaGKTQu5OP6/EXmpxxUdXpfoRfP0IiyCHSNl2gfJ -zxFvVlB2cy8O0EnJ4DdpRpzeTW8pMXGZ7flUSvOhT2rn2K1eqRL9q9LRJn/G/5DP -0u2HxjrZSfn40AS1jcasy6Lcvc7Mwz4W0cfc/tSA67WrFzw= +MIIDTjCCAjagAwIBAgIUJFzoNwmi+322wZTgemjEV35/yrIwDQYJKoZIhvcNAQEL +BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l +cmF0ZWQgQ0EwHhcNMjQwMjE1MTQ0ODM3WhcNMjcwMjE0MTQ0ODM3WjARMQ8wDQYD +VQQDEwZzYW1iYTQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFrD2n +loHuTCCbcX1pqmRnVj7SXcXsBjY9McXxvpEQ4kCdUBlDp3wC9bkGnt6jxlAvEaXn +ZXHg3DqJbD2GU9vRMapcVR4ptAvZmgpEJBU8FA8E1T1EeMhhIoWsYq2crOf/SLQY +g23tO1vmV5+kPD23KiTzj8errYF1Si/6b0IH7XuRUAUJAJYm4gN3YvNqIUYxO/WG +cOlxp/ssIC7Xj6E8Mcdpcfgm+AzLR/x2edl524GAbLqzuKM2bUoMroeNVwVRYPz5 +9iCi+rRxAlI9M2eIm7Gl+qjXUXN1BQAEpkXcxBxDWaIEomd7rR9yudHWvBl8J3+V +jJBgqhecOq/Z+bqbAgMBAAGjezB5MB0GA1UdDgQWBBSbPXW27/QMTZuLD13NUSmQ +0kxaCDAfBgNVHSMEGDAWgBTZMPxziaiEFpaGeNGBxTr+sWGeXTAsBgNVHREEJTAj +hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABgglsb2NhbGhvc3QwCQYDVR0TBAIwADAN +BgkqhkiG9w0BAQsFAAOCAQEAkbnWUXSooivT8FW2PfdTLYDBi18ssdH5g2KU1fNi +E0z+85X1IZaWikGZi3fyj28+GPvRr588EmpPP+IIW9WM5rx3/Gh7elknpwTBYwPo +whuLrKPofEXeM1nUi6nC9QfKDoLWC4B2ztXLD+gcqVHfO/3Q9cs88z9IK4fMAlHu +2dof6SlqVI7AAsqCHaIIuiABW3U81URdtwCJqKt2SUQmpRh6PyAT8rqP7f//gBFF +KyLGOhlEikm+DpFvLNtBHFLL/u9P6u9zV5xyqI4K3wGMAvrTbpicXLJQC6bdIEJR +gFSUL8f6cal5l0ncN6dT3RnpAWwt88xzg8OSMDPWYAyHbw== -----END CERTIFICATE----- diff --git a/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.key b/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.key index c490e0477f8f5..1f359f3bd9c71 100644 --- a/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.key +++ b/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.key @@ -1,27 +1,27 @@ -----BEGIN RSA PRIVATE KEY----- -MIIEpAIBAAKCAQEAzXOoayZrB65jl0ymDyQt2e0CgNDXKpD5l/thK/XRErwZ/YMj -rVyWIV3ZN6f0p3dEGDlEq0RfEcOJCS9Jqb4Rx6GehX4F8DoNgYaWmcAkWwfk5oVA -5Spkk+oN63VeqHrKVo4wBIPIymqD6wNqD3AsBaU4e8DiSCDoXtlYzVzCKDoPiLhj -5TaTcbUqK+sZK8RztmzV0uyODi1h9btsJJC7vIv3ah6mjM/2uBcXnxK9H7iLWK7s -/Wvvr1HVZQwIcn3x/2tcMcd2xLVYx9Oy5M/2Ha1MlIEaStAq1nS/7uh7bIoCsUYX -5T0W5lUPUPYVzh1MMzGkIAsXpN4groUXELsVIQIDAQABAoIBAQC95K7ACsxWVysZ -xvCdghUXHed4ZI7bexAXF3OjDBtoM4/aL4GkVLU2teV1ebm5p5S6xwPfQNyWMKLS -aHCvgUwxtgIs5GRcu2uMUneUOHHh6ZP6NSPcLKi1xlmDTgJlusiV4+oh4iSOYpD3 -gTpgdo9Z0HI6f/cmL7RXJpDbj2atggKZVeSln0BYYRkNcuskPfCK3mPzOYJF5V9T -YbtnaTij72d9v7w181JRHRJQzL6jN0FiIl8BoUmepws+FMXW3lyX/jZ7unf8idTm -9d1cT7OfKThMzY+OzapJHZLl85rCAq4B617ovDHVx1lbNviBiBoTNQIjSNmgkgXk -WUZ2m0KBAoGBAP1WBkAjNuF1/HdGAgge8bPWU6SqJe1Gy25ZkRLkKxp6701EvC+p -a6OuuFonDZLTuTPDgL/v78HQMwQR+QULerAyfwCJDAXIHMFNSvAP/jzEbg4sKRp/ -0ZVUvjedxtADeHIs3SbNEt1P4A+g5U1Is5XGdeKe16zaxfXfy0VA8onZAoGBAM+c -uuNX9FxRrVDUiEIJJFADD/VePXR7YSbjUoLvLtlms4ObOUrnJZJ0IsWxkY4AnDv/ -F0PVA0uMZtVK28BZt1SsYLrKu4KbGPqJy9ArKCAeWuk3DK19XJ8heUkrprEKmQPq -kbIy+LU9MaUKg5pThh1T+uG3QA7liFixxW4KB9CJAoGBAPCWZMM+bh06LrSbMMzD -jmlqzu6fg/tN815uAx98vw6b1217LHjbHdVJ1dwQIIzjM4xcS4Z8eCaI8hoYc3R+ -DVsN6Zz5ighnnh9ZpyRLG/hb1+TvvW1kHAcEfs1Usn3T/ev4fWIe+Z5h//j3pSx7 -Mcm4uzWoAk0vSzcQ/PtdDbkpAoGAf9k+ZV32hxttJYeb2T6T9AnOvAUqxx5rd20p -lKQCL7LE/ViLcYriYkvOVfvBbLcHfxLZmtYET3PIp7SPmuYktanpb14FFqq4OSC+ -OBU7gnvu9AsIbZXzgbM1Y0/UONYT4IuE6T3mVoW2mrHc1R52Sn060+DrO8Exs5zV -vavDoDECgYBN2KF83fdixXCIDizrlJb3mbkhvLoOX+Q3iuJcOLvVrnFo9mGuXhXi -+2K9kVxihqgupRBoR8H60iTB/N3KGvqs4nffDu/lTO4nM9bA13CAVB1JzJG2uh1Y -jFc9zdB4obKShS41VjdTtCh5xi1icX+n8/CGb5TCF8W4ORO8mn9TNg== +MIIEoQIBAAKCAQEA1xUEWtSihnyzWkW4sB3itpEwWm+7alUunxZs7E0zSCn9SC2B +epMXx8bIjdYzPLVVOZ9Coq6nvHkVQEJ/+B49PSPHYVS5r2CPwX4qGVlNDFy5NGgN +p1JuXZPXlZC2mUmlQ2bue167y/7NhM3e9SaTiXfpfc0Czca4b4U4MN7FedylEUIq +fFKIrEl428gRuXqtwpNQ33WdJa16EzP3IBqd4sjJP6D1Ftq0h2GMVDsx/0hCk310 +AVylFm6ZkzKbFb/HWhYtbPg19qyl0Nfngkfa73hXKEglUL/UPnkHMed8i4XldU71 +ydqKqmypQVh65LysPaKGhKLqgVeDG556fwZqzwIDAQABAoIBABE76o4Up7Pls2wH +q4Ar2jS4IYsS6jjfpngHkKbGm4CsIf1x7VloW+lyxjMx910P8p+cC/eDzdujoCOh +WyZYgJ5biuhY+kqmjsj5q7SS8UI3qSdyhwA3R1ym3LQWqmr8pF9oP10R/t5JBn1z +uZUkiCyQYoc6sX987YkSFankz9IEdGRg2xJx9i2Iao5oDHmLUqDMzBS2TcKTv7LN +/pZQEPo/Dsvkvo5DDD6a6aBlLBCs5YcoBbtA2HeknSJRPFRay196sIh8yRcQ0WuK +zLeJvh57OK/J/Jm8jUrpYcDKEFDocFHeopRcXCbQlV1bXnuEJDNfAT/KLo5fkT3N +Xa2nrFECgYEA+ipKdR2vJjNXLSJmLO7tl63W+yESg1bLklXqMKf7WKvBEbrpeJgi +3o+HhxBrKKqBz/Y4O9x+xHhYlbWtm/6Wc9K72wmWEMf0hGevjRxM3eqqWB+7qSJ9 +qYXtwANKR/SMongos0Gavts7XLZK6exkE7RypMrU28P1jb+BQsg56n0CgYEA3Bk/ +sIdf+4Fus0KyUemZIp6xPfpUjOlccrasWA2GGfmi8xOTQn++s3ht0V04kMPHm5gH +AQ9S48+pCczDn9JFtjeH0h6dYVi+7of9NqHa3OXt++WL2T2jvum/iaK2PcW0NHqd +3owSJD+rrx4cXImW2LbKUo73L3CcxOapxYaA4DsCgYBPOaN48ZyteWbrWVCIfGZs +Odayk2e8hnlT77eKDzjvfP1Y8xvLYErytvvRz2ZQa6dOyAhJFOxkpkRPrUi89WSK +a7uog6Gt0NVkNT4Ib2T8hrvJysrwpoarcEm6HJCitxTuwyUImAc82Es1clnJOV78 +SpJgFAhTTPzwFi0GjEijNQKBgQC8xKrrLDAV9RyMgleOCVtdZd192oVJlZvEhwep +PXAWNxSahd922Tklk9QcDGfHQSKhP/JB5nKhEClaTlQ5bo57iYTjoX45T3PyAJAb +mxWq/0jtEiKvXz5hLvkngnXq5PV5TPC5PkkQ7crBloGcnCTUGXHM/PDjryHFfk99 +Ka6+oQJ/ZUPIxj8Lvu7oTVEFPczkM0iTgO1YolY/Z/E+a5suS4FaT5U5Ar9W8E5m +7FLh65QXVMsjxnP+DNSqqNuSpqsyTWFMGtwFIueGRbCnCMN+GBy07fMKC7JLnXmB +Vxf3WwwQ5Crncqn87o0m/mhf/ovsXbGFqqRSJrA0vJifi5D36Q== -----END RSA PRIVATE KEY----- diff --git a/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.pem b/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.pem index df7b319d1e313..a3281c92af704 100644 --- a/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.pem +++ b/x-pack/test/smb-fixture/src/main/resources/smb/certs/ca.pem @@ -1,20 +1,20 @@ -----BEGIN CERTIFICATE----- -MIIDSTCCAjGgAwIBAgIUZTmHgM9YKX8muNbP2IYiv2sfeswwDQYJKoZIhvcNAQEL -BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l -cmF0ZWQgQ0EwHhcNMjEwMjE1MTEzODA2WhcNMjQwMjE1MTEzODA2WjA0MTIwMAYD -VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC -ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM1zqGsmaweuY5dMpg8kLdnt -AoDQ1yqQ+Zf7YSv10RK8Gf2DI61cliFd2Ten9Kd3RBg5RKtEXxHDiQkvSam+Eceh -noV+BfA6DYGGlpnAJFsH5OaFQOUqZJPqDet1Xqh6ylaOMASDyMpqg+sDag9wLAWl -OHvA4kgg6F7ZWM1cwig6D4i4Y+U2k3G1KivrGSvEc7Zs1dLsjg4tYfW7bCSQu7yL -92oepozP9rgXF58SvR+4i1iu7P1r769R1WUMCHJ98f9rXDHHdsS1WMfTsuTP9h2t -TJSBGkrQKtZ0v+7oe2yKArFGF+U9FuZVD1D2Fc4dTDMxpCALF6TeIK6FFxC7FSEC -AwEAAaNTMFEwHQYDVR0OBBYEFHYmRe8GWHSw4wzujbg0XmyRRRZyMB8GA1UdIwQY -MBaAFHYmRe8GWHSw4wzujbg0XmyRRRZyMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI -hvcNAQELBQADggEBADZ5ddmAODL7QeZHkUrUKtGkMWK0WpdlQKx3XkxDod4DM8cI -OaKVUhJznt5ShPCMYZ3+EbTAZhaJUnxdUrulTh8UW0ezDFcnninkzVJ1K2tlYurF -CgrOpr+ni1T08pWrOQHKgFYRqJpme7TgScXCnuqCG0AaC5Ey6O1WhmjRl7aXbUx1 -IgsxtjjN0F3GispWGlLhfXvTR1NRPXLnpyv2aWn0enCdJURsvKuHcMtXSqRlWZ3q -LNDcfunhIuX7MzJ75DytM9vAQEeGMlUHnVk+jXTRuKUEGysq2DrAB8CcW8X5rRHC -nEhqNXWaHabV4NFSGyDxWfCHPVCzuEcni83/jyk= +MIIDSjCCAjKgAwIBAgIVAOtlFYTeA4VeumO8mgMu/nMuJRxRMA0GCSqGSIb3DQEB +CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu +ZXJhdGVkIENBMB4XDTI0MDIxNTE0NDc0NFoXDTI3MDIxNDE0NDc0NFowNDEyMDAG +A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew +ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDXFQRa1KKGfLNaRbiwHeK2 +kTBab7tqVS6fFmzsTTNIKf1ILYF6kxfHxsiN1jM8tVU5n0Kirqe8eRVAQn/4Hj09 +I8dhVLmvYI/BfioZWU0MXLk0aA2nUm5dk9eVkLaZSaVDZu57XrvL/s2Ezd71JpOJ +d+l9zQLNxrhvhTgw3sV53KURQip8UoisSXjbyBG5eq3Ck1DfdZ0lrXoTM/cgGp3i +yMk/oPUW2rSHYYxUOzH/SEKTfXQBXKUWbpmTMpsVv8daFi1s+DX2rKXQ1+eCR9rv +eFcoSCVQv9Q+eQcx53yLheV1TvXJ2oqqbKlBWHrkvKw9ooaEouqBV4Mbnnp/BmrP +AgMBAAGjUzBRMB0GA1UdDgQWBBTZMPxziaiEFpaGeNGBxTr+sWGeXTAfBgNVHSME +GDAWgBTZMPxziaiEFpaGeNGBxTr+sWGeXTAPBgNVHRMBAf8EBTADAQH/MA0GCSqG +SIb3DQEBCwUAA4IBAQARiXwCC8afr98gtGGAG9gJLE6ivfP2Lypoc2PO37Ew1zfc +2UqUYtdmcFvxK0XhEpwbBHxOUJSYMWN+ULoagp2cpbDswZldi7MGdxCu2bK/zYiR +snXMxoYqsJaDqL8HoOV8A12ViaoVCQZeaq0OZ4/NrTRCcPzx+Yru9ZWfgG+4gDbC +hxnDJBY81ju8fhfAcCGuxApPE9fkwFwwVr5nwCbAl7Qotliol/h1eV35hPocyUVz +AxOW2Foc7ehru6NynnEAqOhjcTwMlZCKt0b4kPz2kd+9vL1ZKc/hjNX6Q6sW1N6V +z9Gu6uSsIiM8brs5/zhaRI6EGoOqo4xON5+1ejyh -----END CERTIFICATE----- diff --git a/x-pack/test/smb-fixture/src/main/resources/smb/certs/cert.pem b/x-pack/test/smb-fixture/src/main/resources/smb/certs/cert.pem index a1f78231fa3c4..aba0948092581 100644 --- a/x-pack/test/smb-fixture/src/main/resources/smb/certs/cert.pem +++ b/x-pack/test/smb-fixture/src/main/resources/smb/certs/cert.pem @@ -1,20 +1,20 @@ -----BEGIN CERTIFICATE----- -MIIDTzCCAjegAwIBAgIVAKswcWOE2Y8orNolWoM5EJbBVQKAMA0GCSqGSIb3DQEB -CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu -ZXJhdGVkIENBMB4XDTIxMDIxNTExNDE1NVoXDTI0MDIxNTExNDE1NVowETEPMA0G -A1UEAxMGc2FtYmE0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoNwa -9+QyLYF7q8jFOmd/1WNnV50KAnfBXti9WShk1rTso1YK3QflWtqcn0BsJFfUmRJ1 -79tTTtUmzcqcXXR/ygc8UAal/Cl19h1G4k3Mwf8tlUAjRbH0D+mM6HrZ4+QrAqwb -tDezz8TBAEuVJmTcbdrI6iA9y+I4PReqF//EyQwHeUbKvNHBF0vqYFHqErrBaKsC -VAFp9QSKjF9uRLk4e7PLJDZm9BbVTbtQ76o2SHNMni0UcriqYbi6npIWCLs5xFg5 -dSJ3SwrgBbm/Hg994miPt5CN7l69e7h3ul6/Af3UfMTM02YQ9T3mQ4evW667kAoW -Gxu2Z1cXcsq9+Qj/TwIDAQABo3sweTAdBgNVHQ4EFgQUN818rPYPElgO2oFSXM0c -GXvD4cAwHwYDVR0jBBgwFoAUdiZF7wZYdLDjDO6NuDRebJFFFnIwLAYDVR0RBCUw -I4IJbG9jYWxob3N0hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAkGA1UdEwQCMAAw -DQYJKoZIhvcNAQELBQADggEBAGZQ+NnsLNLbS5dvU2XQDrstcgq7JQcoajH+Us86 -/arjs9C1T7GID1wCXSlyR2uuFTYBH89UpX3oChLohnJByy1Mhpi8l+R2odhosgym -psnt7uejh9DWQrPeiL00ohSjXy3dha5VUESAKoyT/rga1YNl4qeY1J7RPM4NkP7l -nJFzJkTpdcfgcDj42OEOKRjSxaGKTQu5OP6/EXmpxxUdXpfoRfP0IiyCHSNl2gfJ -zxFvVlB2cy8O0EnJ4DdpRpzeTW8pMXGZ7flUSvOhT2rn2K1eqRL9q9LRJn/G/5DP -0u2HxjrZSfn40AS1jcasy6Lcvc7Mwz4W0cfc/tSA67WrFzw= +MIIDTjCCAjagAwIBAgIUJFzoNwmi+322wZTgemjEV35/yrIwDQYJKoZIhvcNAQEL +BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l +cmF0ZWQgQ0EwHhcNMjQwMjE1MTQ0ODM3WhcNMjcwMjE0MTQ0ODM3WjARMQ8wDQYD +VQQDEwZzYW1iYTQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFrD2n +loHuTCCbcX1pqmRnVj7SXcXsBjY9McXxvpEQ4kCdUBlDp3wC9bkGnt6jxlAvEaXn +ZXHg3DqJbD2GU9vRMapcVR4ptAvZmgpEJBU8FA8E1T1EeMhhIoWsYq2crOf/SLQY +g23tO1vmV5+kPD23KiTzj8errYF1Si/6b0IH7XuRUAUJAJYm4gN3YvNqIUYxO/WG +cOlxp/ssIC7Xj6E8Mcdpcfgm+AzLR/x2edl524GAbLqzuKM2bUoMroeNVwVRYPz5 +9iCi+rRxAlI9M2eIm7Gl+qjXUXN1BQAEpkXcxBxDWaIEomd7rR9yudHWvBl8J3+V +jJBgqhecOq/Z+bqbAgMBAAGjezB5MB0GA1UdDgQWBBSbPXW27/QMTZuLD13NUSmQ +0kxaCDAfBgNVHSMEGDAWgBTZMPxziaiEFpaGeNGBxTr+sWGeXTAsBgNVHREEJTAj +hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABgglsb2NhbGhvc3QwCQYDVR0TBAIwADAN +BgkqhkiG9w0BAQsFAAOCAQEAkbnWUXSooivT8FW2PfdTLYDBi18ssdH5g2KU1fNi +E0z+85X1IZaWikGZi3fyj28+GPvRr588EmpPP+IIW9WM5rx3/Gh7elknpwTBYwPo +whuLrKPofEXeM1nUi6nC9QfKDoLWC4B2ztXLD+gcqVHfO/3Q9cs88z9IK4fMAlHu +2dof6SlqVI7AAsqCHaIIuiABW3U81URdtwCJqKt2SUQmpRh6PyAT8rqP7f//gBFF +KyLGOhlEikm+DpFvLNtBHFLL/u9P6u9zV5xyqI4K3wGMAvrTbpicXLJQC6bdIEJR +gFSUL8f6cal5l0ncN6dT3RnpAWwt88xzg8OSMDPWYAyHbw== -----END CERTIFICATE----- diff --git a/x-pack/test/smb-fixture/src/main/resources/smb/certs/key.pem b/x-pack/test/smb-fixture/src/main/resources/smb/certs/key.pem index 2def63ef63914..fbdbb5a48e1c2 100644 --- a/x-pack/test/smb-fixture/src/main/resources/smb/certs/key.pem +++ b/x-pack/test/smb-fixture/src/main/resources/smb/certs/key.pem @@ -1,27 +1,27 @@ -----BEGIN RSA PRIVATE KEY----- -MIIEogIBAAKCAQEAoNwa9+QyLYF7q8jFOmd/1WNnV50KAnfBXti9WShk1rTso1YK -3QflWtqcn0BsJFfUmRJ179tTTtUmzcqcXXR/ygc8UAal/Cl19h1G4k3Mwf8tlUAj -RbH0D+mM6HrZ4+QrAqwbtDezz8TBAEuVJmTcbdrI6iA9y+I4PReqF//EyQwHeUbK -vNHBF0vqYFHqErrBaKsCVAFp9QSKjF9uRLk4e7PLJDZm9BbVTbtQ76o2SHNMni0U -criqYbi6npIWCLs5xFg5dSJ3SwrgBbm/Hg994miPt5CN7l69e7h3ul6/Af3UfMTM -02YQ9T3mQ4evW667kAoWGxu2Z1cXcsq9+Qj/TwIDAQABAoIBAFyLRNio6FVV0Rib -gRyAvwK9FY9KQZ/51b4DY6wPmTQNErdFoSpYiJMkgkb8gTrHbQBDpqY/wEXyS0jJ -7/u0MnDAiOphaM0R7VOStF0t9UcGz+q145UNNCSTcQWu5/w8IKKA8c9U3JYD1CWf -Vkeob0ikPkK0GdlaZJCBNNJpy4T+BOR5D3U7o0Lz3aDExR4ExHWI1+U6xwnOFYxC -VtdKFyzYuT2bOk5E2Y0oi9EhEmJBlIvQ6p3rDPppXmbuTcCNszKal1EyYZePeke/ -iovJZ7+pQS1QsQoXK5el5N2UFM9oAwfARnM8N+yyW5lEV629wVbrYp1WYYefs12e -zGvDqEECgYEA1Vqszj8+RP7FdpCY1jFrvCIRSuIH6KKTD+6mEVcm2GXIZZlI5UGB -U6ZQ+VNxTL6wKPOqqK39jp46HPyvjIwLLwTHNSSLpgAAhBlgR6ZYTW1ImscIsJ5q -dageElmZgRLAmEEF4kx2iJNtTO/YGBqvpf6Q4sbWufO1ARbPQsg7RGMCgYEAwQNL -JPgKtYTzFYnamyq7zWvhfOdWXBVNCT0l0I9FCYKcNtc2yDEme2TOtwkKxcY+1EVj -CTCDabob3EJA6OCAG7GDXrpxbY1o3BB1XtbMnu7KPM/RlcG/qY1qE6kPpCMUevjb -qZfrRkVkH8sTsTQj+Ok4ECBVuUYpJ9BDHNK5fyUCgYAtq8xkFhuxT0xb1hYxe8DR -NAW5nusMfIi4l2CLQ7m4Bwm/3fFByiTyEB8zUA5n3EX/bjGxDBXECtDr1ZeKoYvf -U8mE8b7HGScDIB+BFvW+FU++ei69CBxH9WYCjZWTkL0Tmo+04qNZFx4Foy4B8ux4 -vyaqtN/QTIAJrKVPaWduewKBgC8goL58GhFMTxZZPJlai9SSnNIkoj+Fq/OvjIYq -FU9HJuF1Fxk2dxD2AktK1+iGiVzHPHFH+S5dlOPpAXRbLKyWYV9F4uA/APWKxz3K -8Nd1sse6bpBEaIn7z4TRaNJJBn0oOmpkf7v+wX3J1hsUghwKxfeaDZRZfz8LaPem -tEhJAoGAYikg/Vvw6VUtnDfl0bmC9GRUYrrXpdiz7FlbK6VPe9rs42a5HyrQqVBC -P24wBGfzwFqaTzD/ngodNfnyjNkywChNC18tV3IVbEW5udhX7IJV2Lz52aZBGryK -qt/MXIOtlgjx1hMqfsM3QoMxxyAvwY3tC8vevm3k8yzGfe7Vxb8= +MIIEpAIBAAKCAQEAxaw9p5aB7kwgm3F9aapkZ1Y+0l3F7AY2PTHF8b6REOJAnVAZ +Q6d8AvW5Bp7eo8ZQLxGl52Vx4Nw6iWw9hlPb0TGqXFUeKbQL2ZoKRCQVPBQPBNU9 +RHjIYSKFrGKtnKzn/0i0GINt7Ttb5lefpDw9tyok84/Hq62BdUov+m9CB+17kVAF +CQCWJuIDd2LzaiFGMTv1hnDpcaf7LCAu14+hPDHHaXH4JvgMy0f8dnnZeduBgGy6 +s7ijNm1KDK6HjVcFUWD8+fYgovq0cQJSPTNniJuxpfqo11FzdQUABKZF3MQcQ1mi +BKJne60fcrnR1rwZfCd/lYyQYKoXnDqv2fm6mwIDAQABAoIBAAfOUXD4xJDAeNkq +liVCEUzzXu+3vEUhyaqI+KQfPmNIS/zqWNUPHBqR0YitZWVaQ3hYXhDRNLoIeFdM +6vEPBrMwHuYehl5nOcCSEK24Lw58TEuIkC7QBjmvv0+bZfe17ENsf5AoQIMJwQtL +koZNyrIc+/CSUPQ6mc4j69kb46Okb1T1BvlIJrvzAlb6RTvJD1IjCJFj7Tn5yXjF +mbCynwrOvlc5E2XQHlGbabQGf8uHIlu7P6+L4vPwP26t2nIVSY+1G9/IF0sW9PH7 +EFsdovxYlsKBAX6IdjEi65347IglmCloAcuRz1AOGys8dgPitkDNL2CMgdXuOuj1 +ONkpZ4ECgYEA9FEVK/Gpb0eHlWVJMu5bGa3yADpxUwy5xO7eOd103m39tSUXOw6E +LLb5AHcYkwsOGsz0KrsVPOLC69kQpvaxr2duIzs085x9XikxlDDLJzIfnLgVv7Ib +CSvRR3R8FSAuXxIa68Yx5rG910dcQV3nl8iFzCHcmH9ECI6NQohELisCgYEAzyAl +F8ZQbIA4LPvh8PKlBpuWc3lzqALmv64dtQvhBL3N39J1YpiyxjCgKljIKu5Qk+t/ +WLJAPAHXJCTZgzdQsmcc1U39ziUhJFVEcXfVP777+O0gQwIzLZcKGc9rOoXFlblW +7UjEGbrCQITS9Ir1iBSgfw04PUcUd5dbvWek3VECgYEAjFhTkCXHTgxJ/3Dqhp5T +qMG6ZZUs6idCQ7Vv5M+pRejrN/axjJQ/KyyEh1biv/02wgCANle30Hz2ueK8ZR0L +XxZMN2LYfSSlA8UoHNeWq9JoRG+e1rqqOy93jdOFP+F3oddVraDxo3Lw+cydW1Nl +KVTgPy4oeVWKMFwrG1AJ0ncCgYEAwPxbc051CtNhBBDg0UbUGlcHlKo55/ZkF37c +8R6TV36d/wiyFN7f26fc4/f68X1BGMHY0sSq8v5n/aZUAF2e08sdY2WasOOJgLft +4Kddy1pgnewbHjRDiRvs7mWDrHCNy9Z3tvkQtkR7z++yOuXqphNKA3dGylmbKV6e +vNiAFTECgYBAQyPzQTVSyS950qd5U87EsWKBKRRr0STW+ualYBsbDe4Mt5NlPdJm +WUhUPNkOObFZgVOJEzVYCfiuWu/XOe/PZgdG3hxtaRrRr4WTCt+T7pUhOTaTQM2C +lxTVrBYqlYMLrD+ZJtSJSRfgII4i6JBynH5aVxVjq6Hv2uZyTXcAgg== -----END RSA PRIVATE KEY----- From 4b21d67db3ba117806d31293157a920d6b047eeb Mon Sep 17 00:00:00 2001 From: Tim Grein Date: Thu, 15 Feb 2024 17:41:50 +0100 Subject: [PATCH 07/45] [Connectors API] Check for null sync job id in UpdateConnectorSyncJobIngestionStatsActionRequestTests (#105551) --- ...orSyncJobIngestionStatsActionRequestTests.java | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/syncjob/action/UpdateConnectorSyncJobIngestionStatsActionRequestTests.java b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/syncjob/action/UpdateConnectorSyncJobIngestionStatsActionRequestTests.java index 48ab14558db7e..4f78ad3ffa7e7 100644 --- a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/syncjob/action/UpdateConnectorSyncJobIngestionStatsActionRequestTests.java +++ b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/syncjob/action/UpdateConnectorSyncJobIngestionStatsActionRequestTests.java @@ -47,6 +47,21 @@ public void testValidate_WhenConnectorSyncJobIdIsEmpty_ExpectValidationError() { assertThat(exception.getMessage(), containsString(EMPTY_CONNECTOR_SYNC_JOB_ID_ERROR_MESSAGE)); } + public void testValidate_WhenConnectorSyncJobIdIsNull_ExpectValidationError() { + UpdateConnectorSyncJobIngestionStatsAction.Request request = new UpdateConnectorSyncJobIngestionStatsAction.Request( + null, + 0L, + 0L, + 0L, + 0L, + Instant.now() + ); + ActionRequestValidationException exception = request.validate(); + + assertThat(exception, notNullValue()); + assertThat(exception.getMessage(), containsString(EMPTY_CONNECTOR_SYNC_JOB_ID_ERROR_MESSAGE)); + } + public void testValidate_WhenDeletedDocumentCountIsNegative_ExpectValidationError() { UpdateConnectorSyncJobIngestionStatsAction.Request request = new UpdateConnectorSyncJobIngestionStatsAction.Request( randomAlphaOfLength(10), From e86c2ace46a4074f274a7c5c478eb1d7fb5794c1 Mon Sep 17 00:00:00 2001 From: David Roberts Date: Thu, 15 Feb 2024 17:29:07 +0000 Subject: [PATCH 08/45] [ML] Alter comment that referred to RapidJSON (#105562) We no longer use RapidJSON in the ML C++, so make a comment that referred to it more generic. --- .../xpack/ml/process/StateToProcessWriterHelper.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/process/StateToProcessWriterHelper.java b/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/process/StateToProcessWriterHelper.java index 4c42609c44833..72395e5295d7a 100644 --- a/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/process/StateToProcessWriterHelper.java +++ b/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/process/StateToProcessWriterHelper.java @@ -27,8 +27,8 @@ public static void writeStateToStream(BytesReference source, OutputStream stream --length; } source.slice(0, length).writeTo(stream); - // This is dictated by RapidJSON on the C++ side; it treats a '\0' as end-of-file - // even when it's not really end-of-file, and this is what we need because we're + // This is dictated by the JSON parser on the C++ side; it treats a '\0' as the character + // that separates distinct JSON documents, and this is what we need because we're // sending multiple JSON documents via the same named pipe. stream.write(0); } From 431f4f0bfe7ec7c1e0bbdf09c7a43bac7a9ee9c0 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 15 Feb 2024 18:16:06 +0000 Subject: [PATCH 09/45] Move AwaitsFix for #105543 --- .../elasticsearch/xpack/esql/action/EsqlActionBreakerIT.java | 2 ++ .../java/org/elasticsearch/xpack/esql/action/EsqlActionIT.java | 1 - 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionBreakerIT.java b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionBreakerIT.java index ead0173576ff7..85eb0c02625ad 100644 --- a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionBreakerIT.java +++ b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionBreakerIT.java @@ -7,6 +7,7 @@ package org.elasticsearch.xpack.esql.action; +import org.apache.lucene.tests.util.LuceneTestCase; import org.elasticsearch.ElasticsearchException; import org.elasticsearch.ExceptionsHelper; import org.elasticsearch.action.DocWriteResponse; @@ -34,6 +35,7 @@ import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.instanceOf; +@LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/105543") @TestLogging(value = "org.elasticsearch.xpack.esql:TRACE", reason = "debug") public class EsqlActionBreakerIT extends EsqlActionIT { diff --git a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionIT.java b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionIT.java index ef013749fc896..b23c75df6fa4f 100644 --- a/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionIT.java +++ b/x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlActionIT.java @@ -1316,7 +1316,6 @@ public void testStatsNestFields() { } } - @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/105543") public void testStatsMissingFieldWithStats() { final String node1, node2; if (randomBoolean()) { From 009a847273a48633e735c8934c6c994f0e9877f3 Mon Sep 17 00:00:00 2001 From: Lee Hinman Date: Thu, 15 Feb 2024 11:23:19 -0700 Subject: [PATCH 10/45] Always show `composed_of` field for composable index templates (#105315) * Always show `composed_of` field for composable index templates Prior to e786cfa7061b427cf6185ad907069838dd679574 we inadvertently always added composable index templates with `composed_of: []` beacuse https://github.com/elastic/elasticsearch/commit/e786cfa7061b427cf6185ad907069838dd679574#diff-5081302eb39033199deb1977d544d1cd7867212a92b8d77e0aa0ded361272b11L618-L630 created a new `ComposableIndexTemplate` from an existing one, and the `.composedOf()` field returned an empty list of no component templates were provided: https://github.com/elastic/elasticsearch/blob/89e714ee5dc60db8b4979ab6372ff767e108e9da/server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java#L172-L177 This meant that before 8.12.0 we would always show `composed_of: []` for composable index templates. This commit recreates this behavior, and always displays the empty list even if no component templates are used by a composable index template. Resolves #104627 --- docs/changelog/105315.yaml | 6 ++++++ .../test/indices.get_index_template/10_basic.yml | 13 +++++++++++++ .../cluster/metadata/ComposableIndexTemplate.java | 2 +- 3 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 docs/changelog/105315.yaml diff --git a/docs/changelog/105315.yaml b/docs/changelog/105315.yaml new file mode 100644 index 0000000000000..207e72467a689 --- /dev/null +++ b/docs/changelog/105315.yaml @@ -0,0 +1,6 @@ +pr: 105315 +summary: Always show `composed_of` field for composable index templates +area: Indices APIs +type: bug +issues: + - 104627 diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.get_index_template/10_basic.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.get_index_template/10_basic.yml index 41e5506c412cd..fcf4a75af2227 100644 --- a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.get_index_template/10_basic.yml +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.get_index_template/10_basic.yml @@ -188,3 +188,16 @@ setup: type: keyword lifecycle: data_retention: "30d" + +--- +"Get index template always shows composed_of": + - skip: + version: " - 8.12.99" + reason: "A bug was fixed in 8.13.0 to make `composed_of` always returned" + + - do: + indices.get_index_template: + name: test + + - match: {index_templates.0.name: test} + - match: {index_templates.0.index_template.composed_of: []} diff --git a/server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java b/server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java index dfd4431eb073c..7702ec0ac0b5c 100644 --- a/server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java +++ b/server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java @@ -125,7 +125,7 @@ private ComposableIndexTemplate( ) { this.indexPatterns = indexPatterns; this.template = template; - this.componentTemplates = componentTemplates; + this.componentTemplates = componentTemplates == null ? List.of() : componentTemplates; this.priority = priority; this.version = version; this.metadata = metadata; From d8e66259bbbf2be8285adcc4f06a5cff9798e7b7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Thu, 15 Feb 2024 19:20:19 +0000 Subject: [PATCH 11/45] Fix use-after-free at event-loop shutdown (#105486) We could still be manipulating a network message when the event loop shuts down, causing us to close the message while it's still in use. This is at best going to be a little surprising to the caller, and at worst could be an outright use-after-free bug. This commit moves the double-check for a leaked promise to happen strictly after the event loop has fully terminated, so that we can be sure we've finished using it by this point. Relates #105306, #97301 --- docs/changelog/105486.yaml | 5 ++ .../http/netty4/Netty4HttpChannel.java | 15 ++--- .../http/netty4/Netty4HttpServerChannel.java | 5 +- .../transport/netty4/Netty4TcpChannel.java | 60 ++---------------- .../netty4/Netty4TcpServerChannel.java | 4 +- .../transport/netty4/Netty4Utils.java | 61 +++++++++++++++++++ .../org/elasticsearch/http/HttpChannel.java | 4 +- .../elasticsearch/rest/RestController.java | 1 + .../transport/OutboundHandler.java | 8 +++ .../transport/TransportMessage.java | 7 +++ 10 files changed, 100 insertions(+), 70 deletions(-) create mode 100644 docs/changelog/105486.yaml diff --git a/docs/changelog/105486.yaml b/docs/changelog/105486.yaml new file mode 100644 index 0000000000000..befdaec2301c6 --- /dev/null +++ b/docs/changelog/105486.yaml @@ -0,0 +1,5 @@ +pr: 105486 +summary: Fix use-after-free at event-loop shutdown +area: Network +type: bug +issues: [] diff --git a/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpChannel.java b/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpChannel.java index 705cb02dfb246..83728b8ef73c2 100644 --- a/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpChannel.java +++ b/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpChannel.java @@ -14,12 +14,13 @@ import org.elasticsearch.common.util.concurrent.ListenableFuture; import org.elasticsearch.http.HttpChannel; import org.elasticsearch.http.HttpResponse; -import org.elasticsearch.transport.TransportException; -import org.elasticsearch.transport.netty4.Netty4TcpChannel; import java.net.InetSocketAddress; import java.net.SocketAddress; +import static org.elasticsearch.transport.netty4.Netty4Utils.addListener; +import static org.elasticsearch.transport.netty4.Netty4Utils.safeWriteAndFlush; + public class Netty4HttpChannel implements HttpChannel { private final Channel channel; @@ -27,18 +28,12 @@ public class Netty4HttpChannel implements HttpChannel { Netty4HttpChannel(Channel channel) { this.channel = channel; - Netty4TcpChannel.addListener(this.channel.closeFuture(), closeContext); + addListener(this.channel.closeFuture(), closeContext); } @Override public void sendResponse(HttpResponse response, ActionListener listener) { - // We need to both guard against double resolving the listener and not resolving it in case of event loop shutdown so we need to - // use #notifyOnce here until https://github.com/netty/netty/issues/8007 is resolved. - var wrapped = ActionListener.notifyOnce(listener); - channel.writeAndFlush(response, Netty4TcpChannel.addPromise(wrapped, channel)); - if (channel.eventLoop().isShutdown()) { - wrapped.onFailure(new TransportException("Cannot send HTTP response, event loop is shutting down.")); - } + safeWriteAndFlush(channel, response, listener); } @Override diff --git a/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpServerChannel.java b/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpServerChannel.java index 7356b7fb65cbc..58dc675b76bb1 100644 --- a/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpServerChannel.java +++ b/modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Netty4HttpServerChannel.java @@ -13,10 +13,11 @@ import org.elasticsearch.action.ActionListener; import org.elasticsearch.common.util.concurrent.ListenableFuture; import org.elasticsearch.http.HttpServerChannel; -import org.elasticsearch.transport.netty4.Netty4TcpChannel; import java.net.InetSocketAddress; +import static org.elasticsearch.transport.netty4.Netty4Utils.addListener; + public class Netty4HttpServerChannel implements HttpServerChannel { private final Channel channel; @@ -24,7 +25,7 @@ public class Netty4HttpServerChannel implements HttpServerChannel { Netty4HttpServerChannel(Channel channel) { this.channel = channel; - Netty4TcpChannel.addListener(this.channel.closeFuture(), closeContext); + addListener(this.channel.closeFuture(), closeContext); } @Override diff --git a/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpChannel.java b/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpChannel.java index 33fdb00e7abb2..726f1293c5cf3 100644 --- a/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpChannel.java +++ b/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpChannel.java @@ -11,19 +11,19 @@ import io.netty.channel.Channel; import io.netty.channel.ChannelFuture; import io.netty.channel.ChannelOption; -import io.netty.channel.ChannelPromise; -import org.elasticsearch.ExceptionsHelper; import org.elasticsearch.action.ActionListener; import org.elasticsearch.common.bytes.BytesReference; import org.elasticsearch.common.util.concurrent.ListenableFuture; import org.elasticsearch.core.IOUtils; import org.elasticsearch.core.Releasables; import org.elasticsearch.transport.TcpChannel; -import org.elasticsearch.transport.TransportException; import java.net.InetSocketAddress; +import static org.elasticsearch.transport.netty4.Netty4Utils.addListener; +import static org.elasticsearch.transport.netty4.Netty4Utils.safeWriteAndFlush; + public class Netty4TcpChannel implements TcpChannel { private final Channel channel; @@ -44,52 +44,6 @@ public class Netty4TcpChannel implements TcpChannel { addListener(connectFuture, connectContext); } - /** - * Adds a listener that completes the given {@link ListenableFuture} to the given {@link ChannelFuture}. - * @param channelFuture Channel future - * @param listener Listener to complete - */ - public static void addListener(ChannelFuture channelFuture, ListenableFuture listener) { - channelFuture.addListener(f -> { - if (f.isSuccess()) { - listener.onResponse(null); - } else { - Throwable cause = f.cause(); - if (cause instanceof Error) { - ExceptionsHelper.maybeDieOnAnotherThread(cause); - listener.onFailure(new Exception(cause)); - } else { - listener.onFailure((Exception) cause); - } - } - }); - } - - /** - * Creates a {@link ChannelPromise} for the given {@link Channel} and adds a listener that invokes the given {@link ActionListener} - * on its completion. - * @param listener lister to invoke - * @param channel channel - * @return write promise - */ - public static ChannelPromise addPromise(ActionListener listener, Channel channel) { - ChannelPromise writePromise = channel.newPromise(); - writePromise.addListener(f -> { - if (f.isSuccess()) { - listener.onResponse(null); - } else { - final Throwable cause = f.cause(); - ExceptionsHelper.maybeDieOnAnotherThread(cause); - if (cause instanceof Error) { - listener.onFailure(new Exception(cause)); - } else { - listener.onFailure((Exception) cause); - } - } - }); - return writePromise; - } - @Override public void close() { if (rstOnClose) { @@ -162,13 +116,7 @@ public InetSocketAddress getRemoteAddress() { @Override public void sendMessage(BytesReference reference, ActionListener listener) { - // We need to both guard against double resolving the listener and not resolving it in case of event loop shutdown so we need to - // use #notifyOnce here until https://github.com/netty/netty/issues/8007 is resolved. - var wrapped = ActionListener.notifyOnce(listener); - channel.writeAndFlush(reference, addPromise(wrapped, channel)); - if (channel.eventLoop().isShutdown()) { - wrapped.onFailure(new TransportException("Cannot send message, event loop is shutting down.")); - } + safeWriteAndFlush(channel, reference, listener); } public Channel getNettyChannel() { diff --git a/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpServerChannel.java b/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpServerChannel.java index 1279c50133643..13f691e6e0e5e 100644 --- a/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpServerChannel.java +++ b/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4TcpServerChannel.java @@ -16,6 +16,8 @@ import java.net.InetSocketAddress; +import static org.elasticsearch.transport.netty4.Netty4Utils.addListener; + public class Netty4TcpServerChannel implements TcpServerChannel { private final Channel channel; @@ -23,7 +25,7 @@ public class Netty4TcpServerChannel implements TcpServerChannel { Netty4TcpServerChannel(Channel channel) { this.channel = channel; - Netty4TcpChannel.addListener(this.channel.closeFuture(), closeContext); + addListener(this.channel.closeFuture(), closeContext); } @Override diff --git a/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4Utils.java b/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4Utils.java index b9986dbf00d87..1025ad11ba05e 100644 --- a/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4Utils.java +++ b/modules/transport-netty4/src/main/java/org/elasticsearch/transport/netty4/Netty4Utils.java @@ -11,22 +11,30 @@ import io.netty.buffer.ByteBuf; import io.netty.buffer.CompositeByteBuf; import io.netty.buffer.Unpooled; +import io.netty.channel.Channel; +import io.netty.channel.DefaultChannelPromise; import io.netty.util.NettyRuntime; +import io.netty.util.concurrent.Future; +import io.netty.util.concurrent.ImmediateEventExecutor; import org.apache.lucene.util.BytesRef; import org.apache.lucene.util.BytesRefIterator; +import org.elasticsearch.ExceptionsHelper; +import org.elasticsearch.action.ActionListener; import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.common.bytes.BytesReference; import org.elasticsearch.common.recycler.Recycler; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.util.concurrent.EsExecutors; import org.elasticsearch.core.Booleans; +import org.elasticsearch.transport.TransportException; import java.io.IOException; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.List; import java.util.Locale; +import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.atomic.AtomicBoolean; public class Netty4Utils { @@ -121,4 +129,57 @@ public static Recycler createRecycler(Settings settings) { setAvailableProcessors(EsExecutors.allocatedProcessors(settings)); return NettyAllocator.getRecycler(); } + + /** + * Calls {@link Channel#writeAndFlush} to write the given message to the given channel, but ensures that the listener is completed even + * if the event loop is concurrently shutting down since Netty does not offer this guarantee. + */ + public static void safeWriteAndFlush(Channel channel, Object message, ActionListener listener) { + // Use ImmediateEventExecutor.INSTANCE since we want to be able to complete this promise, and any waiting listeners, even if the + // channel's event loop has shut down. Normally this completion will happen on the channel's event loop anyway because the write op + // can only be completed by some network event from this point on. However... + final var promise = new DefaultChannelPromise(channel, ImmediateEventExecutor.INSTANCE); + addListener(promise, listener); + assert assertCorrectPromiseListenerThreading(channel, promise); + channel.writeAndFlush(message, promise); + if (channel.eventLoop().isShuttingDown()) { + // ... if we get here then the event loop may already have terminated, and https://github.com/netty/netty/issues/8007 means that + // we cannot know if the preceding writeAndFlush made it onto its queue before shutdown or whether it will just vanish without a + // trace, so to avoid a leak we must double-check that the final listener is completed. + channel.eventLoop().terminationFuture().addListener(ignored -> + // NB the promise executor is ImmediateEventExecutor.INSTANCE which means this call to tryFailure() will ensure its completion, + // and the completion of any waiting listeners, without forking away from the current thread. The current thread might be the + // thread that was running the event loop since that's where the terminationFuture is completed, or it might be a thread which + // called (and is still calling) safeWriteAndFlush. + promise.tryFailure(new TransportException("Cannot send network message, event loop is shutting down."))); + } + } + + private static boolean assertCorrectPromiseListenerThreading(Channel channel, Future promise) { + final var eventLoop = channel.eventLoop(); + promise.addListener(future -> { + assert eventLoop.inEventLoop() || future.cause() instanceof RejectedExecutionException || channel.eventLoop().isTerminated() + : future.cause(); + }); + return true; + } + + /** + * Subscribes the given {@link ActionListener} to the given {@link Future}. + */ + public static void addListener(Future future, ActionListener listener) { + future.addListener(f -> { + if (f.isSuccess()) { + listener.onResponse(null); + } else { + final Throwable cause = f.cause(); + ExceptionsHelper.maybeDieOnAnotherThread(cause); + if (cause instanceof Exception exception) { + listener.onFailure(exception); + } else { + listener.onFailure(new Exception(cause)); + } + } + }); + } } diff --git a/server/src/main/java/org/elasticsearch/http/HttpChannel.java b/server/src/main/java/org/elasticsearch/http/HttpChannel.java index 635e6c095c4d0..cf3dbcf9fd029 100644 --- a/server/src/main/java/org/elasticsearch/http/HttpChannel.java +++ b/server/src/main/java/org/elasticsearch/http/HttpChannel.java @@ -20,7 +20,9 @@ public interface HttpChannel extends CloseableChannel { * completed. * * @param response to send to channel - * @param listener to execute upon send completion + * @param listener to execute upon send completion. Note that this listener is usually completed on a network thread in a context in + * which there's a risk of stack overflows if on close it calls back into the network layer in a manner that might end + * up nesting too deeply. When in doubt, dispatch any further work onto a separate thread. */ void sendResponse(HttpResponse response, ActionListener listener); diff --git a/server/src/main/java/org/elasticsearch/rest/RestController.java b/server/src/main/java/org/elasticsearch/rest/RestController.java index e3cc95b65a165..d197fe50d60d5 100644 --- a/server/src/main/java/org/elasticsearch/rest/RestController.java +++ b/server/src/main/java/org/elasticsearch/rest/RestController.java @@ -885,6 +885,7 @@ public void close() { void addChunkLength(long chunkLength) { assert chunkLength >= 0L : chunkLength; assert Transports.assertTransportThread(); // always called on the transport worker, no need for sync + assert get() != null : "already closed"; responseLength += chunkLength; } } diff --git a/server/src/main/java/org/elasticsearch/transport/OutboundHandler.java b/server/src/main/java/org/elasticsearch/transport/OutboundHandler.java index 3261a6c9678b5..0bb8ddeb80c5f 100644 --- a/server/src/main/java/org/elasticsearch/transport/OutboundHandler.java +++ b/server/src/main/java/org/elasticsearch/transport/OutboundHandler.java @@ -71,6 +71,14 @@ void setSlowLogThreshold(TimeValue slowLogThreshold) { this.slowLogThresholdMs = slowLogThreshold.getMillis(); } + /** + * Send a raw message over the given channel. + * + * @param listener completed when the message has been sent, on the network thread (unless the network thread has shut down). Take care + * if calling back into the network layer from this listener without dispatching to a new thread since if we do that + * too many times in a row it can cause a stack overflow. When in doubt, dispatch any follow-up work onto a separate + * thread. + */ void sendBytes(TcpChannel channel, BytesReference bytes, ActionListener listener) { internalSend(channel, bytes, null, listener); } diff --git a/server/src/main/java/org/elasticsearch/transport/TransportMessage.java b/server/src/main/java/org/elasticsearch/transport/TransportMessage.java index d5257bf2840f3..f07cce5a731f8 100644 --- a/server/src/main/java/org/elasticsearch/transport/TransportMessage.java +++ b/server/src/main/java/org/elasticsearch/transport/TransportMessage.java @@ -51,6 +51,13 @@ public boolean tryIncRef() { return true; } + /** + * {@inheritDoc} + * + * Note that the lifetime of an outbound {@link TransportMessage} lasts at least until it has been fully sent over the network, and it + * may be closed on a network thread in a context in which there's a risk of stack overflows if on close it calls back into the network + * layer in a manner that might end up nesting too deeply. When in doubt, dispatch any further work onto a separate thread. + */ @Override public boolean decRef() { // noop, override to manage the life-cycle of resources held by a transport message From 24d7043828317f8a23b644d04f1e1945c9409e0f Mon Sep 17 00:00:00 2001 From: Kathleen DeRusso Date: Thu, 15 Feb 2024 15:30:22 -0500 Subject: [PATCH 12/45] Update skip version on rule_query tests (#105573) * Update skip version on rule_query tests * Update 260_rule_query_search.yml --- .../rest-api-spec/test/entsearch/260_rule_query_search.yml | 2 +- .../xpack/ml/queries/TextExpansionQueryBuilder.java | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml b/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml index 40cdb7839c9ed..f67c955126235 100644 --- a/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml +++ b/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/260_rule_query_search.yml @@ -239,7 +239,7 @@ setup: --- "Perform a rule query with an organic query that must be rewritten to another query type": - skip: - version: " - 8.13.99" + version: " - 8.12.1" reason: Bugfix that was broken in previous versions - do: diff --git a/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/queries/TextExpansionQueryBuilder.java b/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/queries/TextExpansionQueryBuilder.java index a392996fbb448..7d5197b9e9ba0 100644 --- a/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/queries/TextExpansionQueryBuilder.java +++ b/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/queries/TextExpansionQueryBuilder.java @@ -140,6 +140,7 @@ protected void doXContent(XContentBuilder builder, Params params) throws IOExcep @Override protected QueryBuilder doRewrite(QueryRewriteContext queryRewriteContext) throws IOException { + if (weightedTokensSupplier != null) { if (weightedTokensSupplier.get() == null) { return this; From 7870c3a53fcdc17b9d26c5e55c1efce86445e20d Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Fri, 16 Feb 2024 03:54:17 +0100 Subject: [PATCH 13/45] Save needless TreeMap.contains in TimeSeriesIdFieldMapper (#105583) We were burning quite a few cycles here on traversing the tree map twice. --- .../elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java index 6fe39232013d3..fb26debab2acc 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java @@ -345,11 +345,9 @@ public DocumentDimensions validate(final IndexSettings settings) { } private void add(String fieldName, BytesReference encoded) throws IOException { - final Dimension dimension = new Dimension(new BytesRef(fieldName), encoded); - if (dimensions.contains(dimension)) { + if (dimensions.add(new Dimension(new BytesRef(fieldName), encoded)) == false) { throw new IllegalArgumentException("Dimension field [" + fieldName + "] cannot be a multi-valued field."); } - dimensions.add(dimension); } } From 50c57e01fbfe288ee2b96de2a1e7dac79c635229 Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Fri, 16 Feb 2024 05:48:23 +0100 Subject: [PATCH 14/45] Two misc. memory savings in TsidExtractingIdFieldMapper (#105584) No need for a capturing lambda here. Also no need to needlessly setup and ArrayList for the field when we only care about the first value for the name. --- .../mapper/TsidExtractingIdFieldMapper.java | 20 ++++++++----------- 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java index 27e281ed7fb52..08735911c2955 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java @@ -26,7 +26,6 @@ import java.util.Arrays; import java.util.Collection; import java.util.Collections; -import java.util.List; import java.util.Locale; /** @@ -107,13 +106,13 @@ private TsidExtractingIdFieldMapper() { private static final long SEED = 0; public static void createField(DocumentParserContext context, IndexRouting.ExtractFromSource.Builder routingBuilder, BytesRef tsid) { - List timestampFields = context.rootDoc().getFields(DataStreamTimestampFieldMapper.DEFAULT_PATH); - if (timestampFields.isEmpty()) { + final IndexableField timestampField = context.rootDoc().getField(DataStreamTimestampFieldMapper.DEFAULT_PATH); + if (timestampField == null) { throw new IllegalArgumentException( "data stream timestamp field [" + DataStreamTimestampFieldMapper.DEFAULT_PATH + "] is missing" ); } - long timestamp = timestampFields.get(0).numericValue().longValue(); + long timestamp = timestampField.numericValue().longValue(); byte[] suffix = new byte[16]; String id = createId(context.hasDynamicMappers() == false, routingBuilder, tsid, timestamp, suffix); /* @@ -160,14 +159,11 @@ public static String createId( ByteUtils.writeLongLE(hash.h1, suffix, 0); ByteUtils.writeLongBE(timestamp, suffix, 8); // Big Ending shrinks the inverted index by ~37% - String id = routingBuilder.createId(suffix, () -> { - if (dynamicMappersExists == false) { - throw new IllegalStateException( - "Didn't find any fields to include in the routing which would be fine if there are" - + " dynamic mapping waiting but we couldn't find any of those either!" - ); - } - return 0; + String id = routingBuilder.createId(suffix, dynamicMappersExists ? () -> 0 : () -> { + throw new IllegalStateException( + "Didn't find any fields to include in the routing which would be fine if there are" + + " dynamic mapping waiting but we couldn't find any of those either!" + ); }); assert Uid.isURLBase64WithoutPadding(id); // Make sure we get to use Uid's nice optimizations return id; From 8b9c14f9003259f8ebb7b46995debfafb09ee404 Mon Sep 17 00:00:00 2001 From: Andrei Stefan Date: Fri, 16 Feb 2024 11:29:25 +0200 Subject: [PATCH 15/45] ESQL: Add invalid mappings tests with fields from plugins (#105544) --- .../esql/qa/server/single-node/build.gradle | 4 + .../xpack/esql/qa/single_node/Clusters.java | 7 ++ .../xpack/esql/qa/single_node/RestEsqlIT.java | 106 +++++++++++++++++- .../esql/qa/single_node/EsqlClientYamlIT.java | 1 + .../test/esql/50_index_patterns.yml | 3 +- 5 files changed, 118 insertions(+), 3 deletions(-) diff --git a/x-pack/plugin/esql/qa/server/single-node/build.gradle b/x-pack/plugin/esql/qa/server/single-node/build.gradle index b7d090d66b994..2293bc8b6f6d5 100644 --- a/x-pack/plugin/esql/qa/server/single-node/build.gradle +++ b/x-pack/plugin/esql/qa/server/single-node/build.gradle @@ -7,6 +7,10 @@ dependencies { javaRestTestImplementation project(xpackModule('esql:qa:testFixtures')) javaRestTestImplementation project(xpackModule('esql:qa:server')) yamlRestTestImplementation project(xpackModule('esql:qa:server')) + dependencies { + clusterPlugins project(':plugins:mapper-size') + clusterPlugins project(':plugins:mapper-murmur3') + } } restResources { diff --git a/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/Clusters.java b/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/Clusters.java index 498bac9ad69a7..cca5138c8d403 100644 --- a/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/Clusters.java +++ b/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/Clusters.java @@ -8,15 +8,22 @@ package org.elasticsearch.xpack.esql.qa.single_node; import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.elasticsearch.test.cluster.local.LocalClusterConfigProvider; import org.elasticsearch.test.cluster.local.distribution.DistributionType; public class Clusters { + public static ElasticsearchCluster testCluster() { + return testCluster(config -> {}); + } + + public static ElasticsearchCluster testCluster(LocalClusterConfigProvider configProvider) { return ElasticsearchCluster.local() .distribution(DistributionType.DEFAULT) .setting("xpack.security.enabled", "false") .setting("xpack.license.self_generated.type", "trial") .shared(true) + .apply(() -> configProvider) .build(); } } diff --git a/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java b/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java index 67d3da5b4e694..c7727d40d25f2 100644 --- a/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java +++ b/x-pack/plugin/esql/qa/server/single-node/src/javaRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/RestEsqlIT.java @@ -18,6 +18,7 @@ import org.elasticsearch.test.TestClustersThreadFilter; import org.elasticsearch.test.cluster.ElasticsearchCluster; import org.elasticsearch.xpack.esql.qa.rest.RestEsqlTestCase; +import org.hamcrest.Matchers; import org.junit.Assert; import org.junit.ClassRule; @@ -29,12 +30,15 @@ import java.util.Map; import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.core.Is.is; @ThreadLeakFilters(filters = TestClustersThreadFilter.class) public class RestEsqlIT extends RestEsqlTestCase { @ClassRule - public static ElasticsearchCluster cluster = Clusters.testCluster(); + public static ElasticsearchCluster cluster = Clusters.testCluster( + specBuilder -> specBuilder.plugin("mapper-size").plugin("mapper-murmur3") + ); @Override protected String getTestRestCluster() { @@ -100,4 +104,104 @@ public void testPragmaNotAllowed() throws IOException { ResponseException re = expectThrows(ResponseException.class, () -> runEsqlSync(builder)); assertThat(EntityUtils.toString(re.getResponse().getEntity()), containsString("[pragma] only allowed in snapshot builds")); } + + public void testIncompatibleMappingsErrors() throws IOException { + // create first index + Request request = new Request("PUT", "/index1"); + request.setJsonEntity(""" + { + "mappings": { + "_size": { + "enabled": true + }, + "properties": { + "message": { + "type": "keyword", + "fields": { + "hash": { + "type": "murmur3" + } + } + } + } + } + } + """); + assertEquals(200, client().performRequest(request).getStatusLine().getStatusCode()); + + // create second index + request = new Request("PUT", "/index2"); + request.setJsonEntity(""" + { + "mappings": { + "properties": { + "message": { + "type": "long", + "fields": { + "hash": { + "type": "integer" + } + } + } + } + } + } + """); + assertEquals(200, client().performRequest(request).getStatusLine().getStatusCode()); + + // create alias + request = new Request("POST", "/_aliases"); + request.setJsonEntity(""" + { + "actions": [ + { + "add": { + "index": "index1", + "alias": "test_alias" + } + }, + { + "add": { + "index": "index2", + "alias": "test_alias" + } + } + ] + } + """); + assertEquals(200, client().performRequest(request).getStatusLine().getStatusCode()); + assertException( + "from index1,index2 | stats count(message)", + "VerificationException", + "Cannot use field [message] due to ambiguities", + "incompatible types: [keyword] in [index1], [long] in [index2]" + ); + assertException( + "from test_alias | where message is not null", + "VerificationException", + "Cannot use field [message] due to ambiguities", + "incompatible types: [keyword] in [index1], [long] in [index2]" + ); + assertException("from test_alias | where _size is not null | limit 1", "Unknown column [_size]"); + assertException( + "from test_alias | where message.hash is not null | limit 1", + "Cannot use field [message.hash] due to ambiguities", + "incompatible types: [integer] in [index2], [murmur3] in [index1]" + ); + assertException( + "from index1 | where message.hash is not null | limit 1", + "Cannot use field [message.hash] with unsupported type [murmur3]" + ); + // clean up + assertThat(deleteIndex("index1").isAcknowledged(), Matchers.is(true)); + assertThat(deleteIndex("index2").isAcknowledged(), Matchers.is(true)); + } + + private void assertException(String query, String... errorMessages) throws IOException { + ResponseException re = expectThrows(ResponseException.class, () -> runEsqlSync(new RequestObjectBuilder().query(query))); + assertThat(re.getResponse().getStatusLine().getStatusCode(), equalTo(400)); + for (var error : errorMessages) { + assertThat(re.getMessage(), containsString(error)); + } + } } diff --git a/x-pack/plugin/esql/qa/server/single-node/src/yamlRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/EsqlClientYamlIT.java b/x-pack/plugin/esql/qa/server/single-node/src/yamlRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/EsqlClientYamlIT.java index e67ca751298be..7556b0916e1f6 100644 --- a/x-pack/plugin/esql/qa/server/single-node/src/yamlRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/EsqlClientYamlIT.java +++ b/x-pack/plugin/esql/qa/server/single-node/src/yamlRestTest/java/org/elasticsearch/xpack/esql/qa/single_node/EsqlClientYamlIT.java @@ -15,6 +15,7 @@ * Run the ESQL yaml tests against the synchronous API. */ public class EsqlClientYamlIT extends AbstractEsqlClientYamlIT { + public EsqlClientYamlIT(final ClientYamlTestCandidate testCandidate) { super(testCandidate); } diff --git a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/50_index_patterns.yml b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/50_index_patterns.yml index 38023b7791709..87af58cf89898 100644 --- a/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/50_index_patterns.yml +++ b/x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/50_index_patterns.yml @@ -311,7 +311,6 @@ same_name_different_type: message: type: long - - do: bulk: index: test1 @@ -338,7 +337,7 @@ same_name_different_type: - "No limit defined, adding default limit of \\[.*\\]" esql.query: body: - query: 'from test1,test2 ' + query: 'from test1,test2' - match: { columns.0.name: message } - match: { columns.0.type: unsupported } - length: { values: 4 } From 6a8c1d0480d742765bf07fb027a0dbb2e3937bd3 Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Fri, 16 Feb 2024 20:50:57 +0100 Subject: [PATCH 16/45] Dry up IdFieldMapper implementations (#105586) The field type is almost the same code across both implementations. --- .../index/mapper/IdFieldMapper.java | 59 +++++++++++++++ .../index/mapper/ProvidedIdFieldMapper.java | 52 +------------ .../mapper/TsidExtractingIdFieldMapper.java | 73 ++----------------- .../index/mapper/IdFieldTypeTests.java | 5 +- 4 files changed, 68 insertions(+), 121 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/index/mapper/IdFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/IdFieldMapper.java index 02151b305c546..c5b1f575d941f 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/IdFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/IdFieldMapper.java @@ -10,9 +10,17 @@ import org.apache.lucene.document.Field; import org.apache.lucene.document.StringField; +import org.apache.lucene.search.MatchAllDocsQuery; +import org.apache.lucene.search.Query; +import org.apache.lucene.search.TermInSetQuery; +import org.apache.lucene.util.BytesRef; import org.elasticsearch.common.lucene.Lucene; import org.elasticsearch.index.analysis.NamedAnalyzer; +import org.elasticsearch.index.query.SearchExecutionContext; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; import java.util.Map; /** @@ -72,4 +80,55 @@ public final SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { public static Field standardIdField(String id) { return new StringField(NAME, Uid.encodeId(id), Field.Store.YES); } + + protected abstract static class AbstractIdFieldType extends TermBasedFieldType { + + public AbstractIdFieldType() { + super(NAME, true, true, false, TextSearchInfo.SIMPLE_MATCH_ONLY, Collections.emptyMap()); + } + + @Override + public String typeName() { + return CONTENT_TYPE; + } + + @Override + public boolean isSearchable() { + // The _id field is always searchable. + return true; + } + + @Override + public Query termsQuery(Collection values, SearchExecutionContext context) { + failIfNotIndexed(); + BytesRef[] bytesRefs = values.stream().map(v -> { + Object idObject = v; + if (idObject instanceof BytesRef) { + idObject = ((BytesRef) idObject).utf8ToString(); + } + return Uid.encodeId(idObject.toString()); + }).toArray(BytesRef[]::new); + return new TermInSetQuery(name(), bytesRefs); + } + + @Override + public Query termQuery(Object value, SearchExecutionContext context) { + return termsQuery(Arrays.asList(value), context); + } + + @Override + public Query existsQuery(SearchExecutionContext context) { + return new MatchAllDocsQuery(); + } + + @Override + public BlockLoader blockLoader(BlockLoaderContext blContext) { + return new BlockStoredFieldsReader.IdBlockLoader(); + } + + @Override + public ValueFetcher valueFetcher(SearchExecutionContext context, String format) { + return new StoredValueFetcher(context.lookup(), NAME); + } + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/ProvidedIdFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/ProvidedIdFieldMapper.java index d8a4177ee3211..45dbda9d2946b 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/ProvidedIdFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/ProvidedIdFieldMapper.java @@ -9,10 +9,7 @@ package org.elasticsearch.index.mapper; import org.apache.lucene.index.LeafReaderContext; -import org.apache.lucene.search.MatchAllDocsQuery; -import org.apache.lucene.search.Query; import org.apache.lucene.search.SortField; -import org.apache.lucene.search.TermInSetQuery; import org.apache.lucene.util.BytesRef; import org.elasticsearch.common.logging.DeprecationCategory; import org.elasticsearch.common.logging.DeprecationLogger; @@ -41,8 +38,6 @@ import java.io.IOException; import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; import java.util.function.BooleanSupplier; /** @@ -59,69 +54,24 @@ public class ProvidedIdFieldMapper extends IdFieldMapper { public static final ProvidedIdFieldMapper NO_FIELD_DATA = new ProvidedIdFieldMapper(() -> false); - static final class IdFieldType extends TermBasedFieldType { + static final class IdFieldType extends AbstractIdFieldType { private final BooleanSupplier fieldDataEnabled; IdFieldType(BooleanSupplier fieldDataEnabled) { - super(NAME, true, true, false, TextSearchInfo.SIMPLE_MATCH_ONLY, Collections.emptyMap()); this.fieldDataEnabled = fieldDataEnabled; } - @Override - public String typeName() { - return CONTENT_TYPE; - } - - @Override - public boolean isSearchable() { - // The _id field is always searchable. - return true; - } - @Override public boolean mayExistInIndex(SearchExecutionContext context) { return true; } - @Override - public ValueFetcher valueFetcher(SearchExecutionContext context, String format) { - return new StoredValueFetcher(context.lookup(), NAME); - } - @Override public boolean isAggregatable() { return fieldDataEnabled.getAsBoolean(); } - @Override - public Query termQuery(Object value, SearchExecutionContext context) { - return termsQuery(Arrays.asList(value), context); - } - - @Override - public Query existsQuery(SearchExecutionContext context) { - return new MatchAllDocsQuery(); - } - - @Override - public Query termsQuery(Collection values, SearchExecutionContext context) { - failIfNotIndexed(); - BytesRef[] bytesRefs = values.stream().map(v -> { - Object idObject = v; - if (idObject instanceof BytesRef) { - idObject = ((BytesRef) idObject).utf8ToString(); - } - return Uid.encodeId(idObject.toString()); - }).toArray(BytesRef[]::new); - return new TermInSetQuery(name(), bytesRefs); - } - - @Override - public BlockLoader blockLoader(BlockLoaderContext blContext) { - return new BlockStoredFieldsReader.IdBlockLoader(); - } - @Override public IndexFieldData.Builder fielddataBuilder(FieldDataContext fieldDataContext) { if (fieldDataEnabled.getAsBoolean() == false) { diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java index 08735911c2955..54c495a2c9a6c 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TsidExtractingIdFieldMapper.java @@ -11,9 +11,6 @@ import org.apache.lucene.document.Field; import org.apache.lucene.document.StringField; import org.apache.lucene.index.IndexableField; -import org.apache.lucene.search.MatchAllDocsQuery; -import org.apache.lucene.search.Query; -import org.apache.lucene.search.TermInSetQuery; import org.apache.lucene.util.BytesRef; import org.elasticsearch.cluster.routing.IndexRouting; import org.elasticsearch.common.hash.MurmurHash3; @@ -21,11 +18,7 @@ import org.elasticsearch.common.util.ByteUtils; import org.elasticsearch.index.fielddata.FieldDataContext; import org.elasticsearch.index.fielddata.IndexFieldData; -import org.elasticsearch.index.query.SearchExecutionContext; -import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; import java.util.Locale; /** @@ -40,67 +33,13 @@ public class TsidExtractingIdFieldMapper extends IdFieldMapper { public static final TsidExtractingIdFieldMapper INSTANCE = new TsidExtractingIdFieldMapper(); - public static final TypeParser PARSER = new FixedTypeParser(MappingParserContext::idFieldMapper); - // NOTE: we use a prefix when hashing the tsid field so to be able later on (for instance for debugging purposes) - // to query documents whose tsid has been hashed. Using a prefix allows us to query using the prefix. - - static final class IdFieldType extends TermBasedFieldType { - IdFieldType() { - super(NAME, true, true, false, TextSearchInfo.SIMPLE_MATCH_ONLY, Collections.emptyMap()); - } - - @Override - public String typeName() { - return CONTENT_TYPE; - } - - @Override - public boolean isSearchable() { - // The _id field is always searchable. - return true; - } - - @Override - public ValueFetcher valueFetcher(SearchExecutionContext context, String format) { - return new StoredValueFetcher(context.lookup(), NAME); - } - - @Override - public Query termQuery(Object value, SearchExecutionContext context) { - return termsQuery(Arrays.asList(value), context); - } - - @Override - public Query existsQuery(SearchExecutionContext context) { - return new MatchAllDocsQuery(); - } - - @Override - public Query termsQuery(Collection values, SearchExecutionContext context) { - failIfNotIndexed(); - BytesRef[] bytesRefs = values.stream().map(v -> { - Object idObject = v; - if (idObject instanceof BytesRef) { - idObject = ((BytesRef) idObject).utf8ToString(); - } - return Uid.encodeId(idObject.toString()); - }).toArray(BytesRef[]::new); - return new TermInSetQuery(name(), bytesRefs); - } - - @Override - public BlockLoader blockLoader(BlockLoaderContext blContext) { - return new BlockStoredFieldsReader.IdBlockLoader(); - } - - @Override - public IndexFieldData.Builder fielddataBuilder(FieldDataContext fieldDataContext) { - throw new IllegalArgumentException("Fielddata is not supported on [_id] field in [time_series] indices"); - } - } - private TsidExtractingIdFieldMapper() { - super(new IdFieldType()); + super(new AbstractIdFieldType() { + @Override + public IndexFieldData.Builder fielddataBuilder(FieldDataContext fieldDataContext) { + throw new IllegalArgumentException("Fielddata is not supported on [_id] field in [time_series] indices"); + } + }); } private static final long SEED = 0; diff --git a/server/src/test/java/org/elasticsearch/index/mapper/IdFieldTypeTests.java b/server/src/test/java/org/elasticsearch/index/mapper/IdFieldTypeTests.java index 7aafb2b674448..6e6691de5d0b3 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/IdFieldTypeTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/IdFieldTypeTests.java @@ -23,7 +23,7 @@ public class IdFieldTypeTests extends ESTestCase { public void testRangeQuery() { MappedFieldType ft = randomBoolean() ? new ProvidedIdFieldMapper.IdFieldType(() -> false) - : new TsidExtractingIdFieldMapper.IdFieldType(); + : TsidExtractingIdFieldMapper.INSTANCE.fieldType(); IllegalArgumentException e = expectThrows( IllegalArgumentException.class, () -> ft.rangeQuery(null, null, randomBoolean(), randomBoolean(), null, null, null, null) @@ -58,7 +58,6 @@ public void testIsAggregatable() { ft = new ProvidedIdFieldMapper.IdFieldType(() -> true); assertTrue(ft.isAggregatable()); - ft = new TsidExtractingIdFieldMapper.IdFieldType(); - assertFalse(ft.isAggregatable()); + assertFalse(TsidExtractingIdFieldMapper.INSTANCE.fieldType().isAggregatable()); } } From b2e626e7df6e825c89ffaac17c649ec04ef69ddd Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Sun, 18 Feb 2024 17:37:28 +0200 Subject: [PATCH 17/45] Support Profile Activate with JWTs with client authn (#105439) Adds support for JWTs with client authentication to the activate user profile API. Closes #105342 --- docs/changelog/105439.yaml | 6 ++ .../security/activate-user-profile.asciidoc | 22 +++-- .../security/client-authentication.asciidoc | 16 ++++ .../rest-api/security/grant-api-keys.asciidoc | 24 +---- .../authc/jwt/JwtRealmSingleNodeTests.java | 95 +++++++++++++++++++ .../rest/action/SecurityBaseRestHandler.java | 26 +++++ .../action/apikey/RestGrantApiKeyAction.java | 31 +----- .../profile/RestActivateProfileAction.java | 27 +++--- .../RestActivateProfileActionTests.java | 54 +++++++++++ 9 files changed, 233 insertions(+), 68 deletions(-) create mode 100644 docs/changelog/105439.yaml create mode 100644 docs/reference/rest-api/security/client-authentication.asciidoc create mode 100644 x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileActionTests.java diff --git a/docs/changelog/105439.yaml b/docs/changelog/105439.yaml new file mode 100644 index 0000000000000..45bbede469542 --- /dev/null +++ b/docs/changelog/105439.yaml @@ -0,0 +1,6 @@ +pr: 105439 +summary: Support Profile Activate with JWTs with client authn +area: Authentication +type: enhancement +issues: + - 105342 diff --git a/docs/reference/rest-api/security/activate-user-profile.asciidoc b/docs/reference/rest-api/security/activate-user-profile.asciidoc index 476477a5b5b72..f6ce32e1bb19e 100644 --- a/docs/reference/rest-api/security/activate-user-profile.asciidoc +++ b/docs/reference/rest-api/security/activate-user-profile.asciidoc @@ -28,6 +28,9 @@ Creates or updates a user profile on behalf of another user. The activate user profile API creates or updates a profile document for end users with information that is extracted from the user's authentication object, including `username`, `full_name`, `roles`, and the authentication realm. +For example, in the JWT `access_token` case, the profile user's `username` is +extracted from the JWT token claim pointed to by the `claims.principal` +setting of the JWT realm that authenticated the token. When updating a profile document, the API enables the document if it was disabled. Any updates do not change existing content for either the `labels` or @@ -46,8 +49,11 @@ is intended for. `access_token`:: (Required*, string) -The user's access token. If you specify the `access_token` grant type, this -parameter is required. It is not valid with other grant types. +The user's <>, or JWT. Both <> and +<> JWT token types are supported, and they depend on the underlying JWT realm configuration. +If you specify the `access_token` grant type, this parameter is required. It is not valid with other grant types. + +include::client-authentication.asciidoc[] `grant_type`:: (Required, string) @@ -57,24 +63,22 @@ The type of grant. [%collapsible%open] ==== `access_token`:: -(Required*, string) -In this type of grant, you must supply an access token that was created by the -{es} token service. For more information, see -<> and <>. +In this type of grant, you must supply either an access token, that was created by the +{es} token service (see <> and <>), +or a <> (either a JWT `access_token` or a JWT `id_token`). `password`:: -(Required*, string) In this type of grant, you must supply the `username` and `password` for the user that you want to create the API key for. ==== `password`:: -(Optional*, string) +(Required*, string) The user's password. If you specify the `password` grant type, this parameter is required. It is not valid with other grant types. `username`:: -(Optional*, string) +(Required*, string) The username that identifies the user. If you specify the `password` grant type, this parameter is required. It is not valid with other grant types. diff --git a/docs/reference/rest-api/security/client-authentication.asciidoc b/docs/reference/rest-api/security/client-authentication.asciidoc new file mode 100644 index 0000000000000..3bbcd2e146557 --- /dev/null +++ b/docs/reference/rest-api/security/client-authentication.asciidoc @@ -0,0 +1,16 @@ +`client_authentication`:: +(Optional, object) When using the `access_token` grant type, and when supplying a +JWT, this specifies the client authentication for <> that +need it (i.e. what's normally specified by the `ES-Client-Authentication` request header). + +`scheme`::: +(Required, string) The scheme (case-sensitive) as it's supplied in the +`ES-Client-Authentication` request header. Currently, the only supported +value is <>. + +`value`::: +(Required, string) The value that follows the scheme for the client credentials +as it's supplied in the `ES-Client-Authentication` request header. For example, +if the request header would be `ES-Client-Authentication: SharedSecret myShar3dS3cret` +if the client were to authenticate directly with a JWT, then `value` here should +be `myShar3dS3cret`. \ No newline at end of file diff --git a/docs/reference/rest-api/security/grant-api-keys.asciidoc b/docs/reference/rest-api/security/grant-api-keys.asciidoc index 8feb6c3cd5f52..10c109b00bbf9 100644 --- a/docs/reference/rest-api/security/grant-api-keys.asciidoc +++ b/docs/reference/rest-api/security/grant-api-keys.asciidoc @@ -89,29 +89,13 @@ It supports nested data structure. Within the `metadata` object, keys beginning with `_` are reserved for system usage. -`client_authentication`:: -(Optional, object) When using the `access_token` grant type, and when supplying a -JWT, this specifies the client authentication for <> that -need it (i.e. what's normally specified by the `ES-Client-Authentication` request header). - -`scheme`::: -(Required, string) The scheme (case-sensitive) as it's supplied in the -`ES-Client-Authentication` request header. Currently, the only supported -value is <>. - -`value`::: -(Required, string) The value that follows the scheme for the client credentials -as it's supplied in the `ES-Client-Authentication` request header. For example, -if the request header would be `ES-Client-Authentication: SharedSecret myShar3dS3cret` -if the client were to authenticate directly with a JWT, then `value` here should -be `myShar3dS3cret`. +include::client-authentication.asciidoc[] `grant_type`:: (Required, string) The type of grant. Supported grant types are: `access_token`,`password`. `access_token`::: -(Required*, string) In this type of grant, you must supply either an access token, that was created by the {es} token service (see <> and <>), or a <> (either a JWT `access_token` or a JWT `id_token`). @@ -121,12 +105,12 @@ In this type of grant, you must supply the user ID and password for which you want to create the API key. `password`:: -(Optional*, string) +(Required*, string) The user's password. If you specify the `password` grant type, this parameter is required. It is not valid with other grant types. `username`:: -(Optional*, string) +(Required*, string) The user name that identifies the user. If you specify the `password` grant type, this parameter is required. It is not valid with other grant types. @@ -134,6 +118,8 @@ this parameter is required. It is not valid with other grant types. (Optional, string) The name of the user to be <>. +*Indicates that the setting is required in some, but not all situations. + [[security-api-grant-api-key-example]] ==== {api-examples-title} diff --git a/x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/jwt/JwtRealmSingleNodeTests.java b/x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/jwt/JwtRealmSingleNodeTests.java index ac033ba75798a..fba4df3c38031 100644 --- a/x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/jwt/JwtRealmSingleNodeTests.java +++ b/x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/jwt/JwtRealmSingleNodeTests.java @@ -41,6 +41,12 @@ import org.elasticsearch.xpack.core.security.action.apikey.CreateApiKeyResponse; import org.elasticsearch.xpack.core.security.action.apikey.GrantApiKeyAction; import org.elasticsearch.xpack.core.security.action.apikey.GrantApiKeyRequest; +import org.elasticsearch.xpack.core.security.action.profile.ActivateProfileAction; +import org.elasticsearch.xpack.core.security.action.profile.ActivateProfileRequest; +import org.elasticsearch.xpack.core.security.action.profile.ActivateProfileResponse; +import org.elasticsearch.xpack.core.security.action.profile.GetProfilesAction; +import org.elasticsearch.xpack.core.security.action.profile.GetProfilesRequest; +import org.elasticsearch.xpack.core.security.action.profile.GetProfilesResponse; import org.elasticsearch.xpack.core.security.action.user.AuthenticateAction; import org.elasticsearch.xpack.core.security.action.user.AuthenticateRequest; import org.elasticsearch.xpack.core.security.action.user.AuthenticateResponse; @@ -237,6 +243,84 @@ public void testGrantApiKeyForJWT() throws Exception { } } + public void testActivateProfileForJWT() throws Exception { + final JWTClaimsSet.Builder jwtClaims = new JWTClaimsSet.Builder(); + final String principal; + final String sharedSecret; + final String realmName; + // id_token or access_token + if (randomBoolean()) { + principal = "me"; + // JWT "id_token" valid for jwt0 + jwtClaims.audience("es-01") + .issuer("my-issuer-01") + .subject(principal) + .claim("groups", "admin") + .issueTime(Date.from(Instant.now())) + .expirationTime(Date.from(Instant.now().plusSeconds(600))) + .build(); + sharedSecret = jwt0SharedSecret; + realmName = "jwt0"; + } else { + principal = "me@example.com"; + // JWT "access_token" valid for jwt2 + jwtClaims.audience("es-03") + .issuer("my-issuer-03") + .subject("user-03") + .claim("groups", "admin") + .claim("email", principal) + .issueTime(Date.from(Instant.now())) + .expirationTime(Date.from(Instant.now().plusSeconds(300))); + sharedSecret = jwt2SharedSecret; + realmName = "jwt2"; + } + { + // JWT is valid but the client authentication is NOT + ActivateProfileRequest activateProfileRequest = getActivateProfileForJWT( + getSignedJWT(jwtClaims.build()), + randomFrom("WRONG", null) + ); + ElasticsearchSecurityException e = expectThrows( + ElasticsearchSecurityException.class, + () -> client().execute(ActivateProfileAction.INSTANCE, activateProfileRequest).actionGet() + ); + assertThat(e.getMessage(), containsString("unable to authenticate user")); + } + { + // both JWT and client authentication are valid + ActivateProfileRequest activateProfileRequest = getActivateProfileForJWT(getSignedJWT(jwtClaims.build()), sharedSecret); + ActivateProfileResponse activateProfileResponse = client().execute(ActivateProfileAction.INSTANCE, activateProfileRequest) + .actionGet(); + assertThat(activateProfileResponse.getProfile(), notNullValue()); + assertThat(activateProfileResponse.getProfile().uid(), notNullValue()); + assertThat(activateProfileResponse.getProfile().user().username(), is(principal)); + assertThat(activateProfileResponse.getProfile().user().realmName(), is(realmName)); + // test to get the profile by uid + GetProfilesRequest getProfilesRequest = new GetProfilesRequest(List.of(activateProfileResponse.getProfile().uid()), Set.of()); + GetProfilesResponse getProfilesResponse = client().execute(GetProfilesAction.INSTANCE, getProfilesRequest).actionGet(); + assertThat(getProfilesResponse.getProfiles().size(), is(1)); + assertThat(getProfilesResponse.getProfiles().get(0).uid(), is(activateProfileResponse.getProfile().uid())); + assertThat(getProfilesResponse.getProfiles().get(0).enabled(), is(true)); + assertThat(getProfilesResponse.getProfiles().get(0).user().username(), is(principal)); + assertThat(getProfilesResponse.getProfiles().get(0).user().realmName(), is(realmName)); + } + { + // client authentication is valid but the JWT is not + final SignedJWT wrongJWT; + if (randomBoolean()) { + wrongJWT = getSignedJWT(jwtClaims.build(), ("wrong key that's longer than 256 bits").getBytes(StandardCharsets.UTF_8)); + } else { + wrongJWT = getSignedJWT(jwtClaims.audience("wrong audience claim value").build()); + } + ActivateProfileRequest activateProfileRequest = getActivateProfileForJWT(wrongJWT, sharedSecret); + ElasticsearchSecurityException e = expectThrows( + ElasticsearchSecurityException.class, + () -> client().execute(ActivateProfileAction.INSTANCE, activateProfileRequest).actionGet() + ); + assertThat(e.getMessage(), containsString("unable to authenticate user")); + } + } + @SuppressWarnings("unchecked") public void testInvalidJWTDoesNotFallbackToAnonymousAccess() throws Exception { // anonymous access works when no valid Bearer @@ -696,4 +780,15 @@ private static GrantApiKeyRequest getGrantApiKeyForJWT(SignedJWT signedJWT, Stri grantApiKeyRequest.getApiKeyRequest().setName(randomAlphaOfLength(8)); return grantApiKeyRequest; } + + private static ActivateProfileRequest getActivateProfileForJWT(SignedJWT signedJWT, String sharedSecret) { + ActivateProfileRequest activateProfileRequest = new ActivateProfileRequest(); + activateProfileRequest.getGrant().setType("access_token"); + activateProfileRequest.getGrant().setAccessToken(new SecureString(signedJWT.serialize().toCharArray())); + if (sharedSecret != null) { + activateProfileRequest.getGrant() + .setClientAuthentication(new Grant.ClientAuthentication("SharedSecret", new SecureString(sharedSecret.toCharArray()))); + } + return activateProfileRequest; + } } diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/SecurityBaseRestHandler.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/SecurityBaseRestHandler.java index f1647c997f1db..f0405e42f1f22 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/SecurityBaseRestHandler.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/SecurityBaseRestHandler.java @@ -7,14 +7,21 @@ package org.elasticsearch.xpack.security.rest.action; import org.elasticsearch.client.internal.node.NodeClient; +import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.license.XPackLicenseState; import org.elasticsearch.rest.BaseRestHandler; import org.elasticsearch.rest.RestRequest; import org.elasticsearch.rest.RestResponse; +import org.elasticsearch.xcontent.ConstructingObjectParser; +import org.elasticsearch.xcontent.ObjectParser; +import org.elasticsearch.xcontent.ParseField; +import org.elasticsearch.xcontent.XContentParser; import org.elasticsearch.xpack.core.XPackSettings; +import org.elasticsearch.xpack.core.security.action.Grant; import java.io.IOException; +import java.util.Arrays; /** * Base class for security rest handlers. This handler takes care of ensuring that the license @@ -22,6 +29,25 @@ */ public abstract class SecurityBaseRestHandler extends BaseRestHandler { + protected static final ConstructingObjectParser CLIENT_AUTHENTICATION_PARSER = + new ConstructingObjectParser<>("client_authentication", a -> new Grant.ClientAuthentication((String) a[0], (SecureString) a[1])); + + static { + CLIENT_AUTHENTICATION_PARSER.declareString(ConstructingObjectParser.constructorArg(), new ParseField("scheme")); + CLIENT_AUTHENTICATION_PARSER.declareField( + ConstructingObjectParser.constructorArg(), + SecurityBaseRestHandler::getSecureString, + new ParseField("value"), + ObjectParser.ValueType.STRING + ); + } + + protected static SecureString getSecureString(XContentParser parser) throws IOException { + return new SecureString( + Arrays.copyOfRange(parser.textCharacters(), parser.textOffset(), parser.textOffset() + parser.textLength()) + ); + } + protected final Settings settings; protected final XPackLicenseState licenseState; diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/apikey/RestGrantApiKeyAction.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/apikey/RestGrantApiKeyAction.java index d07c5529e3ca1..a3b1ad86a7166 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/apikey/RestGrantApiKeyAction.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/apikey/RestGrantApiKeyAction.java @@ -11,7 +11,6 @@ import org.elasticsearch.ExceptionsHelper; import org.elasticsearch.action.support.WriteRequest; import org.elasticsearch.client.internal.node.NodeClient; -import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.license.XPackLicenseState; import org.elasticsearch.rest.RestRequest; @@ -20,19 +19,17 @@ import org.elasticsearch.rest.Scope; import org.elasticsearch.rest.ServerlessScope; import org.elasticsearch.rest.action.RestToXContentListener; -import org.elasticsearch.xcontent.ConstructingObjectParser; import org.elasticsearch.xcontent.ObjectParser; import org.elasticsearch.xcontent.ParseField; import org.elasticsearch.xcontent.XContentParser; -import org.elasticsearch.xpack.core.security.action.Grant; import org.elasticsearch.xpack.core.security.action.apikey.CreateApiKeyRequestBuilder; import org.elasticsearch.xpack.core.security.action.apikey.CreateApiKeyResponse; import org.elasticsearch.xpack.core.security.action.apikey.GrantApiKeyAction; import org.elasticsearch.xpack.core.security.action.apikey.GrantApiKeyRequest; import org.elasticsearch.xpack.security.authc.ApiKeyService; +import org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler; import java.io.IOException; -import java.util.Arrays; import java.util.List; import java.util.Set; @@ -46,31 +43,19 @@ @ServerlessScope(Scope.INTERNAL) public final class RestGrantApiKeyAction extends ApiKeyBaseRestHandler implements RestRequestFilter { - private static final ConstructingObjectParser CLIENT_AUTHENTICATION_PARSER = - new ConstructingObjectParser<>("client_authentication", a -> new Grant.ClientAuthentication((String) a[0], (SecureString) a[1])); - static { - CLIENT_AUTHENTICATION_PARSER.declareString(ConstructingObjectParser.constructorArg(), new ParseField("scheme")); - CLIENT_AUTHENTICATION_PARSER.declareField( - ConstructingObjectParser.constructorArg(), - RestGrantApiKeyAction::getSecureString, - new ParseField("value"), - ObjectParser.ValueType.STRING - ); - } - static final ObjectParser PARSER = new ObjectParser<>("grant_api_key_request", GrantApiKeyRequest::new); static { PARSER.declareString((req, str) -> req.getGrant().setType(str), new ParseField("grant_type")); PARSER.declareString((req, str) -> req.getGrant().setUsername(str), new ParseField("username")); PARSER.declareField( (req, secStr) -> req.getGrant().setPassword(secStr), - RestGrantApiKeyAction::getSecureString, + SecurityBaseRestHandler::getSecureString, new ParseField("password"), ObjectParser.ValueType.STRING ); PARSER.declareField( (req, secStr) -> req.getGrant().setAccessToken(secStr), - RestGrantApiKeyAction::getSecureString, + SecurityBaseRestHandler::getSecureString, new ParseField("access_token"), ObjectParser.ValueType.STRING ); @@ -87,12 +72,6 @@ public final class RestGrantApiKeyAction extends ApiKeyBaseRestHandler implement ); } - private static SecureString getSecureString(XContentParser parser) throws IOException { - return new SecureString( - Arrays.copyOfRange(parser.textCharacters(), parser.textOffset(), parser.textOffset() + parser.textLength()) - ); - } - public RestGrantApiKeyAction(Settings settings, XPackLicenseState licenseState) { super(settings, licenseState); } @@ -138,10 +117,8 @@ protected RestChannelConsumer innerPrepareRequest(final RestRequest request, fin } } - private static final Set FILTERED_FIELDS = Set.of("password", "access_token", "client_authentication.value"); - @Override public Set getFilteredFields() { - return FILTERED_FIELDS; + return Set.of("password", "access_token", "client_authentication.value"); } } diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileAction.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileAction.java index 4b214801cba2b..c069405b82132 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileAction.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileAction.java @@ -8,7 +8,6 @@ package org.elasticsearch.xpack.security.rest.action.profile; import org.elasticsearch.client.internal.node.NodeClient; -import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.license.XPackLicenseState; import org.elasticsearch.rest.RestRequest; @@ -24,7 +23,6 @@ import org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler; import java.io.IOException; -import java.util.Arrays; import java.util.List; import java.util.Set; @@ -42,16 +40,21 @@ public class RestActivateProfileAction extends SecurityBaseRestHandler implement PARSER.declareString((req, str) -> req.getGrant().setUsername(str), new ParseField("username")); PARSER.declareField( (req, secStr) -> req.getGrant().setPassword(secStr), - RestActivateProfileAction::getSecureString, + SecurityBaseRestHandler::getSecureString, new ParseField("password"), ObjectParser.ValueType.STRING ); PARSER.declareField( (req, secStr) -> req.getGrant().setAccessToken(secStr), - RestActivateProfileAction::getSecureString, + SecurityBaseRestHandler::getSecureString, new ParseField("access_token"), ObjectParser.ValueType.STRING ); + PARSER.declareObject( + (req, clientAuthentication) -> req.getGrant().setClientAuthentication(clientAuthentication), + CLIENT_AUTHENTICATION_PARSER, + new ParseField("client_authentication") + ); } public RestActivateProfileAction(Settings settings, XPackLicenseState licenseState) { @@ -68,23 +71,21 @@ public String getName() { return "xpack_security_activate_profile"; } + // package-private for tests + static ActivateProfileRequest fromXContent(XContentParser parser) throws IOException { + return PARSER.parse(parser, null); + } + @Override protected RestChannelConsumer innerPrepareRequest(RestRequest request, NodeClient client) throws IOException { try (XContentParser parser = request.contentParser()) { - final ActivateProfileRequest activateProfileRequest = PARSER.parse(parser, null); + final ActivateProfileRequest activateProfileRequest = fromXContent(parser); return channel -> client.execute(ActivateProfileAction.INSTANCE, activateProfileRequest, new RestToXContentListener<>(channel)); } } - // TODO: extract to standalone helper method - private static SecureString getSecureString(XContentParser parser) throws IOException { - return new SecureString( - Arrays.copyOfRange(parser.textCharacters(), parser.textOffset(), parser.textOffset() + parser.textLength()) - ); - } - @Override public Set getFilteredFields() { - return Set.of("password", "access_token"); + return Set.of("password", "access_token", "client_authentication.value"); } } diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileActionTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileActionTests.java new file mode 100644 index 0000000000000..5f2afd8082302 --- /dev/null +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/profile/RestActivateProfileActionTests.java @@ -0,0 +1,54 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.security.rest.action.profile; + +import org.elasticsearch.common.settings.SecureString; +import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.xcontent.XContentFactory; +import org.elasticsearch.xcontent.XContentParser; +import org.elasticsearch.xpack.core.security.action.profile.ActivateProfileRequest; + +import static org.hamcrest.Matchers.is; + +public class RestActivateProfileActionTests extends ESTestCase { + + public void testParseXContentForGrantApiKeyRequest() throws Exception { + final String grantType = randomAlphaOfLength(8); + final String username = randomAlphaOfLength(8); + final String password = randomAlphaOfLength(8); + final String accessToken = randomAlphaOfLength(8); + final String clientAuthenticationScheme = randomAlphaOfLength(8); + final String clientAuthenticationValue = randomAlphaOfLength(8); + try ( + XContentParser content = createParser( + XContentFactory.jsonBuilder() + .startObject() + .field("grant_type", grantType) + .field("username", username) + .field("password", password) + .field("access_token", accessToken) + .startObject("client_authentication") + .field("scheme", clientAuthenticationScheme) + .field("value", clientAuthenticationValue) + .endObject() + .endObject() + ) + ) { + ActivateProfileRequest activateProfileRequest = RestActivateProfileAction.fromXContent(content); + assertThat(activateProfileRequest.getGrant().getType(), is(grantType)); + assertThat(activateProfileRequest.getGrant().getUsername(), is(username)); + assertThat(activateProfileRequest.getGrant().getPassword(), is(new SecureString(password.toCharArray()))); + assertThat(activateProfileRequest.getGrant().getAccessToken(), is(new SecureString(accessToken.toCharArray()))); + assertThat(activateProfileRequest.getGrant().getClientAuthentication().scheme(), is(clientAuthenticationScheme)); + assertThat( + activateProfileRequest.getGrant().getClientAuthentication().value(), + is(new SecureString(clientAuthenticationValue.toCharArray())) + ); + } + } +} From 6fec837e32d3a1d57dc6a49e40ecd15c8e0d8d13 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Sun, 18 Feb 2024 17:38:56 +0200 Subject: [PATCH 18/45] [Doc] API Key deletion settings (#105392) This documents API Key delete settings. --- .../security/invalidate-api-keys.asciidoc | 7 +++- .../settings/security-settings.asciidoc | 38 ++++++++++++++++--- 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/docs/reference/rest-api/security/invalidate-api-keys.asciidoc b/docs/reference/rest-api/security/invalidate-api-keys.asciidoc index fab695a8f21ef..e4cc91000c9c9 100644 --- a/docs/reference/rest-api/security/invalidate-api-keys.asciidoc +++ b/docs/reference/rest-api/security/invalidate-api-keys.asciidoc @@ -27,8 +27,11 @@ in one of the three formats: [[security-api-invalidate-api-key-desc]] ==== {api-description-title} -The API keys created by <> can be -invalidated using this API. +This API invalidates API keys created by the <> or <> +APIs. +Invalidated API keys fail authentication, but they can still be viewed using the +<> and <> APIs, +for at least the <>, until they are automatically deleted. [[security-api-invalidate-api-key-request-body]] diff --git a/docs/reference/settings/security-settings.asciidoc b/docs/reference/settings/security-settings.asciidoc index fee38383896fa..ae8bf5d4e9006 100644 --- a/docs/reference/settings/security-settings.asciidoc +++ b/docs/reference/settings/security-settings.asciidoc @@ -202,11 +202,6 @@ You can set the following API key service settings in (<>) Set to `false` to disable the built-in API key service. Defaults to `true`. -`xpack.security.authc.api_key.hashing.algorithm`:: -(<>) -Specifies the hashing algorithm that is used for securing API key credentials. -See <>. Defaults to `pbkdf2`. - `xpack.security.authc.api_key.cache.ttl`:: (<>) The time-to-live for cached API key entries. A API key id and a hash of its @@ -224,6 +219,39 @@ The hashing algorithm that is used for the in-memory cached API key credentials. For possible values, see <>. Defaults to `ssha256`. +[[api-key-service-settings-delete-retention-period]] +`xpack.security.authc.api_key.delete.retention_period`:: +(<>) +Invalidated or expired API keys older than the retention period are eligible for deletion. +Defaults to `7d`. + +-- +NOTE: Large real-time clock inconsistency across cluster nodes can cause problems +with evaluating the API key retention period. That is, if the clock on the node +invalidating the API key is significantly different than the one performing the deletion, +the key may be retained for longer or shorter than the configured retention period. + +-- + +`xpack.security.authc.api_key.delete.interval`:: +(<>, Expert) +Cluster nodes schedule the automatic deletion of invalidated or expired API keys +that are older than the retention period. +This setting controls the minimum time interval between two such deletion jobs. +Defaults to `24h`. ++ +NOTE: This is a low-level setting that currently controls the interval between +deletion jobs triggered per-node, not across the cluster. + +`xpack.security.authc.api_key.delete.timeout`:: +(<>, Expert) +Sets the timeout of the internal search and delete call. + +`xpack.security.authc.api_key.hashing.algorithm`:: +(<>) +Specifies the hashing algorithm that is used for securing API key credentials. +See <>. Defaults to `pbkdf2`. + [discrete] [[security-domain-settings]] ==== Security domain settings From 3f8bc36788af228fe0943faee75a098dc0c4df8c Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Sun, 18 Feb 2024 17:56:22 +0100 Subject: [PATCH 19/45] Fix Gradle File leaks (#105597) Fixing a couple of file leaks (and cleaning up one missing try-with-resources). The directory descriptor leaks in particular were leaking massively on every precommit run, to the point where it slows down the whole system and/or we're running into descriptor limits. --- .../internal/conventions/VersionPropertiesLoader.java | 5 +---- .../gradle/internal/test/rest/CopyRestApiTask.java | 9 ++++++--- .../main/java/org/elasticsearch/gradle/LoggedExec.java | 4 +++- .../gradle/testclusters/ElasticsearchNode.java | 4 +++- 4 files changed, 13 insertions(+), 9 deletions(-) diff --git a/build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/VersionPropertiesLoader.java b/build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/VersionPropertiesLoader.java index 510a8df411285..17842461636c6 100644 --- a/build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/VersionPropertiesLoader.java +++ b/build-conventions/src/main/java/org/elasticsearch/gradle/internal/conventions/VersionPropertiesLoader.java @@ -21,11 +21,8 @@ public class VersionPropertiesLoader { static Properties loadBuildSrcVersion(File input, ProviderFactory providerFactory) throws IOException { Properties props = new Properties(); - InputStream is = new FileInputStream(input); - try { + try (InputStream is = new FileInputStream(input)) { props.load(is); - } finally { - is.close(); } loadBuildSrcVersion(props, providerFactory); return props; diff --git a/build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/CopyRestApiTask.java b/build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/CopyRestApiTask.java index 1fbc367804ed4..d4b680655a2e1 100644 --- a/build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/CopyRestApiTask.java +++ b/build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/rest/CopyRestApiTask.java @@ -146,13 +146,16 @@ private boolean projectHasYamlRestTests() { try { // check source folder for tests if (sourceResourceDir != null && new File(sourceResourceDir, REST_TEST_PREFIX).exists()) { - return Files.walk(sourceResourceDir.toPath().resolve(REST_TEST_PREFIX)) - .anyMatch(p -> p.getFileName().toString().endsWith("yml")); + try (var files = Files.walk(sourceResourceDir.toPath().resolve(REST_TEST_PREFIX))) { + return files.anyMatch(p -> p.getFileName().toString().endsWith("yml")); + } } // check output for cases where tests are copied programmatically File yamlTestOutputDir = new File(additionalYamlTestsDir.get().getAsFile(), REST_TEST_PREFIX); if (yamlTestOutputDir.exists()) { - return Files.walk(yamlTestOutputDir.toPath()).anyMatch(p -> p.getFileName().toString().endsWith("yml")); + try (var files = Files.walk(yamlTestOutputDir.toPath())) { + return files.anyMatch(p -> p.getFileName().toString().endsWith("yml")); + } } } catch (IOException e) { throw new IllegalStateException(String.format("Error determining if this project [%s] has rest tests.", getProject()), e); diff --git a/build-tools/src/main/java/org/elasticsearch/gradle/LoggedExec.java b/build-tools/src/main/java/org/elasticsearch/gradle/LoggedExec.java index acb526cf9a3bb..4fda91d332118 100644 --- a/build-tools/src/main/java/org/elasticsearch/gradle/LoggedExec.java +++ b/build-tools/src/main/java/org/elasticsearch/gradle/LoggedExec.java @@ -118,7 +118,9 @@ public void run() { try { // the file may not exist if the command never output anything if (Files.exists(spoolFile.toPath())) { - Files.lines(spoolFile.toPath()).forEach(logger::error); + try (var lines = Files.lines(spoolFile.toPath())) { + lines.forEach(logger::error); + } } } catch (IOException e) { throw new RuntimeException("could not log", e); diff --git a/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java b/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java index ce4fd7502f417..31e1cb882305a 100644 --- a/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java +++ b/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java @@ -458,7 +458,9 @@ public synchronized void start() { // make sure we always start fresh if (Files.exists(workingDir)) { if (preserveDataDir) { - Files.list(workingDir).filter(path -> path.equals(confPathData) == false).forEach(this::uncheckedDeleteWithRetry); + try (var files = Files.list(workingDir)) { + files.filter(path -> path.equals(confPathData) == false).forEach(this::uncheckedDeleteWithRetry); + } } else { deleteWithRetry(workingDir); } From ac574acca98d34838cd28ffee547bfdd90e00885 Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Sun, 18 Feb 2024 21:35:53 +0100 Subject: [PATCH 20/45] Speedup BytesArray.indexOf (#105595) This method is only relevant for searching the line breaks in bulk requests. We can assume that in most cases the number of bytes between two line breaks is well above 8. This makes the SIMD(ish) logic this PR introduces much faster than the simple loop. Benchmarking this using the TSDB track indexing step, shows about a 3x to 4x speedup when searching for the next byte during bulk request parsing. --- .../common/bytes/BytesArray.java | 83 ++++++++++++++++++- .../common/bytes/BytesArrayTests.java | 25 +++--- .../bytes/AbstractBytesReferenceTestCase.java | 6 +- 3 files changed, 98 insertions(+), 16 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/common/bytes/BytesArray.java b/server/src/main/java/org/elasticsearch/common/bytes/BytesArray.java index 1e171b954aa7d..40697a0c158a5 100644 --- a/server/src/main/java/org/elasticsearch/common/bytes/BytesArray.java +++ b/server/src/main/java/org/elasticsearch/common/bytes/BytesArray.java @@ -58,10 +58,87 @@ public byte get(int index) { @Override public int indexOf(byte marker, int from) { - for (int i = offset + from; i < offset + length; i++) { - if (bytes[i] == marker) { - return i - offset; + final int len = length - from; + int off = offset + from; + final int toIndex = offset + length; + // First, try to find the marker in the first few bytes, so we can enter the faster 8-byte aligned loop below. + // The idea for this logic is taken from Netty's io.netty.buffer.ByteBufUtil.firstIndexOf and optimized for little endian hardware. + // See e.g. https://richardstartin.github.io/posts/finding-bytes for the idea behind this optimization. + final int byteCount = len & 7; + if (byteCount > 0) { + final int index = unrolledFirstIndexOf(bytes, off, byteCount, marker); + if (index != -1) { + return index - offset; } + off += byteCount; + if (off == toIndex) { + return -1; + } + } + final int longCount = len >>> 3; + // faster SWAR (SIMD Within A Register) loop + final long pattern = compilePattern(marker); + for (int i = 0; i < longCount; i++) { + int index = findInLong(ByteUtils.readLongLE(bytes, off), pattern); + if (index < Long.BYTES) { + return off + index - offset; + } + off += Long.BYTES; + } + return -1; + } + + private static long compilePattern(byte byteToFind) { + return (byteToFind & 0xFFL) * 0x101010101010101L; + } + + private static int findInLong(long word, long pattern) { + long input = word ^ pattern; + long tmp = (input & 0x7F7F7F7F7F7F7F7FL) + 0x7F7F7F7F7F7F7F7FL; + tmp = ~(tmp | input | 0x7F7F7F7F7F7F7F7FL); + final int binaryPosition = Long.numberOfTrailingZeros(tmp); + return binaryPosition >>> 3; + } + + private static int unrolledFirstIndexOf(byte[] buffer, int fromIndex, int byteCount, byte value) { + if (buffer[fromIndex] == value) { + return fromIndex; + } + if (byteCount == 1) { + return -1; + } + if (buffer[fromIndex + 1] == value) { + return fromIndex + 1; + } + if (byteCount == 2) { + return -1; + } + if (buffer[fromIndex + 2] == value) { + return fromIndex + 2; + } + if (byteCount == 3) { + return -1; + } + if (buffer[fromIndex + 3] == value) { + return fromIndex + 3; + } + if (byteCount == 4) { + return -1; + } + if (buffer[fromIndex + 4] == value) { + return fromIndex + 4; + } + if (byteCount == 5) { + return -1; + } + if (buffer[fromIndex + 5] == value) { + return fromIndex + 5; + } + if (byteCount == 6) { + return -1; + } + if (buffer[fromIndex + 6] == value) { + return fromIndex + 6; } return -1; } diff --git a/server/src/test/java/org/elasticsearch/common/bytes/BytesArrayTests.java b/server/src/test/java/org/elasticsearch/common/bytes/BytesArrayTests.java index 9eadd4bb6d038..31eb56853ce9e 100644 --- a/server/src/test/java/org/elasticsearch/common/bytes/BytesArrayTests.java +++ b/server/src/test/java/org/elasticsearch/common/bytes/BytesArrayTests.java @@ -7,22 +7,22 @@ */ package org.elasticsearch.common.bytes; -import org.elasticsearch.common.io.stream.BytesStreamOutput; import org.hamcrest.Matchers; import java.io.IOException; import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.greaterThanOrEqualTo; public class BytesArrayTests extends AbstractBytesReferenceTestCase { @Override - protected BytesReference newBytesReference(int length) throws IOException { + protected BytesReference newBytesReference(int length) { return newBytesReference(length, randomInt(length)); } @Override - protected BytesReference newBytesReferenceWithOffsetOfZero(int length) throws IOException { + protected BytesReference newBytesReferenceWithOffsetOfZero(int length) { return newBytesReference(length, 0); } @@ -31,28 +31,29 @@ protected BytesReference newBytesReference(byte[] content) { return new BytesArray(content); } - private BytesReference newBytesReference(int length, int offset) throws IOException { - // we know bytes stream output always creates a paged bytes reference, we use it to create randomized content - final BytesStreamOutput out = new BytesStreamOutput(length + offset); - for (int i = 0; i < length + offset; i++) { - out.writeByte((byte) random().nextInt(1 << 8)); + private BytesReference newBytesReference(int length, int offset) { + // randomly add some bytes at the end of the bytes reference + int tail = randomBoolean() ? 0 : randomIntBetween(0, length + offset); + final byte[] out = new byte[length + offset + tail]; + for (int i = 0; i < length + offset + tail; i++) { + out[i] = (byte) random().nextInt(1 << 8); } - assertEquals(length + offset, out.size()); - BytesArray ref = new BytesArray(out.bytes().toBytesRef().bytes, offset, length); + BytesArray ref = new BytesArray(out, offset, length); assertEquals(length, ref.length()); assertTrue(ref instanceof BytesArray); assertThat(ref.length(), Matchers.equalTo(length)); return ref; } - public void testArray() throws IOException { + public void testArray() { int[] sizes = { 0, randomInt(PAGE_SIZE), PAGE_SIZE, randomIntBetween(2, PAGE_SIZE * randomIntBetween(2, 5)) }; for (int i = 0; i < sizes.length; i++) { BytesArray pbr = (BytesArray) newBytesReference(sizes[i]); byte[] array = pbr.array(); assertNotNull(array); - assertEquals(sizes[i], array.length - pbr.arrayOffset()); + assertEquals(sizes[i], pbr.length()); + assertThat(pbr.array().length, greaterThanOrEqualTo(pbr.arrayOffset() + pbr.length())); assertSame(array, pbr.array()); } } diff --git a/test/framework/src/main/java/org/elasticsearch/common/bytes/AbstractBytesReferenceTestCase.java b/test/framework/src/main/java/org/elasticsearch/common/bytes/AbstractBytesReferenceTestCase.java index d0004b7660167..d2d2837ee7a29 100644 --- a/test/framework/src/main/java/org/elasticsearch/common/bytes/AbstractBytesReferenceTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/common/bytes/AbstractBytesReferenceTestCase.java @@ -661,9 +661,13 @@ public void testIndexOf() throws IOException { map.forEach((value, positions) -> { for (int i = 0; i < positions.size(); i++) { final int pos = positions.get(i); - final int from = i == 0 ? randomIntBetween(0, pos) : positions.get(i - 1) + 1; + final int from = randomIntBetween(i == 0 ? 0 : positions.get(i - 1) + 1, pos); assertEquals(bytesReference.indexOf(value, from), pos); } + final int firstNotFoundPos = positions.get(positions.size() - 1) + 1; + if (firstNotFoundPos < bytesReference.length()) { + assertEquals(-1, bytesReference.indexOf(value, between(firstNotFoundPos, bytesReference.length() - 1))); + } }); final byte missing = randomValueOtherThanMany(map::containsKey, ESTestCase::randomByte); assertEquals(-1, bytesReference.indexOf(missing, randomIntBetween(0, Math.max(0, size - 1)))); From d9af4a90e8ffe80e9b6e99bed459fbb9e19accd3 Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Sun, 18 Feb 2024 20:32:59 -0500 Subject: [PATCH 21/45] Cross-cluster painless/execute actions should check permissions only on target remote cluster (#105360) Fixes issue #105084, where the cross-cluster painless execute API inappropriately throws a security exception (on a secured cluster) if permissions for the remote index target are not present in the local querying cluster. Since it is a remote request, the security check should be happening on the remote cluster, not the local one. We were not able to simply implement IndicesRequest.Replaceable to solve this case, as the painless execute API does not support wildcards and implementing Replaceable takes the security checks and index resolutions down paths that don't make sense for this use case. A new interface, IndicesRequest.SingleIndexNoWildcards was created to support the specific use case that allows remote indices, but no wildcards. Changes were made to RBACEngine and IndiciesAndAliasesResolver to handle this new interface appropriately. Two new IT tests were added - one covering clusters secured with RCS1 and the other clusters secured with RCS2. Both cover the bug fixed in this commit. --- docs/changelog/105360.yaml | 6 + .../action/PainlessExecuteAction.java | 48 ++- .../action/PainlessExecuteApiTests.java | 40 +++ .../elasticsearch/action/IndicesRequest.java | 13 + ...eClusterSecurityRCS1PainlessExecuteIT.java | 223 +++++++++++++ ...eClusterSecurityRCS2PainlessExecuteIT.java | 303 ++++++++++++++++++ ...RemoteClusterSecurityResolveClusterIT.java | 1 - .../security/authz/AuthorizationService.java | 6 +- .../authz/IndicesAndAliasesResolver.java | 29 +- .../xpack/security/authz/RBACEngine.java | 7 +- .../authz/IndicesAndAliasesResolverTests.java | 6 +- 11 files changed, 654 insertions(+), 28 deletions(-) create mode 100644 docs/changelog/105360.yaml create mode 100644 x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS1PainlessExecuteIT.java create mode 100644 x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS2PainlessExecuteIT.java diff --git a/docs/changelog/105360.yaml b/docs/changelog/105360.yaml new file mode 100644 index 0000000000000..41a7ea24e5500 --- /dev/null +++ b/docs/changelog/105360.yaml @@ -0,0 +1,6 @@ +pr: 105360 +summary: Cross-cluster painless/execute actions should check permissions only on target + remote cluster +area: Search +type: bug +issues: [] diff --git a/modules/lang-painless/src/main/java/org/elasticsearch/painless/action/PainlessExecuteAction.java b/modules/lang-painless/src/main/java/org/elasticsearch/painless/action/PainlessExecuteAction.java index 3b67b76f59a31..874b2316406ef 100644 --- a/modules/lang-painless/src/main/java/org/elasticsearch/painless/action/PainlessExecuteAction.java +++ b/modules/lang-painless/src/main/java/org/elasticsearch/painless/action/PainlessExecuteAction.java @@ -24,6 +24,7 @@ import org.elasticsearch.action.ActionRequestValidationException; import org.elasticsearch.action.ActionResponse; import org.elasticsearch.action.ActionType; +import org.elasticsearch.action.IndicesRequest; import org.elasticsearch.action.RemoteClusterActionType; import org.elasticsearch.action.support.ActionFilters; import org.elasticsearch.action.support.IndicesOptions; @@ -91,6 +92,7 @@ import org.elasticsearch.search.lookup.SearchLookup; import org.elasticsearch.tasks.Task; import org.elasticsearch.threadpool.ThreadPool; +import org.elasticsearch.transport.RemoteClusterAware; import org.elasticsearch.transport.TransportService; import org.elasticsearch.xcontent.ConstructingObjectParser; import org.elasticsearch.xcontent.NamedXContentRegistry; @@ -102,6 +104,7 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -121,7 +124,7 @@ public class PainlessExecuteAction { private PainlessExecuteAction() {/* no instances */} - public static class Request extends SingleShardRequest implements ToXContentObject { + public static class Request extends SingleShardRequest implements ToXContentObject, IndicesRequest.SingleIndexNoWildcards { private static final ParseField SCRIPT_FIELD = new ParseField("script"); private static final ParseField CONTEXT_FIELD = new ParseField("context"); @@ -217,7 +220,9 @@ static ContextSetup parse(XContentParser parser, Void context) throws IOExceptio } /** - * @param indexExpression should be of the form "index" or "cluster:index". Wildcards are OK. + * @param indexExpression should be of the form "index" or "cluster:index". Wildcards are not allowed + * (if wildcards are present, an exception will be thrown in later processing, so + * we don't check it here). * @return Tuple where first entry is clusterAlias, which will be null if not in the indexExpression * and second entry is the index name * Tuple(null, null) will be returned if indexExpression is null @@ -229,7 +234,8 @@ static Tuple parseClusterAliasAndIndex(String indexExpression) { return new Tuple<>(null, null); } String trimmed = indexExpression.trim(); - if (trimmed.startsWith(":") || trimmed.endsWith(":")) { + String sep = String.valueOf(RemoteClusterAware.REMOTE_CLUSTER_INDEX_SEPARATOR); + if (trimmed.startsWith(sep) || trimmed.endsWith(sep)) { throw new IllegalArgumentException( "Unable to parse one single valid index name from the provided index: [" + indexExpression + "]" ); @@ -241,10 +247,10 @@ static Tuple parseClusterAliasAndIndex(String indexExpression) { // Instead, it will fail with the inaccurate and confusing error message: // "Cross-cluster calls are not supported in this context but remote indices were requested: [blogs,remote1:blogs]" // which comes later out of the IndexNameExpressionResolver pathway this code uses. - String[] parts = indexExpression.split(":", 2); + String[] parts = indexExpression.split(sep, 2); if (parts.length == 1) { return new Tuple<>(null, parts[0]); - } else if (parts.length == 2 && parts[1].contains(":") == false) { + } else if (parts.length == 2 && parts[1].contains(sep) == false) { return new Tuple<>(parts[0], parts[1]); } else { throw new IllegalArgumentException( @@ -358,7 +364,11 @@ static Request parse(XContentParser parser) throws IOException { this.context = scriptContextName != null ? fromScriptContextName(scriptContextName) : PainlessTestScript.CONTEXT; if (setup != null) { this.contextSetup = setup; - index(contextSetup.index); + if (contextSetup.getClusterAlias() == null) { + index(contextSetup.getIndex()); + } else { + index(contextSetup.getClusterAlias() + RemoteClusterAware.REMOTE_CLUSTER_INDEX_SEPARATOR + contextSetup.getIndex()); + } } else { contextSetup = null; } @@ -527,14 +537,34 @@ protected void doExecute(Task task, Request request, ActionListener li if (request.getContextSetup() == null || request.getContextSetup().getClusterAlias() == null) { super.doExecute(task, request, listener); } else { - // forward to remote cluster - String clusterAlias = request.getContextSetup().getClusterAlias(); + // forward to remote cluster after stripping off the clusterAlias from the index expression + removeClusterAliasFromIndexExpression(request); transportService.getRemoteClusterService() - .getRemoteClusterClient(clusterAlias, EsExecutors.DIRECT_EXECUTOR_SERVICE) + .getRemoteClusterClient(request.getContextSetup().getClusterAlias(), EsExecutors.DIRECT_EXECUTOR_SERVICE) .execute(PainlessExecuteAction.REMOTE_TYPE, request, listener); } } + // Visible for testing + static void removeClusterAliasFromIndexExpression(Request request) { + if (request.index() != null) { + String[] split = request.index().split(String.valueOf(RemoteClusterAware.REMOTE_CLUSTER_INDEX_SEPARATOR)); + if (split.length > 1) { + /* + * if the cluster alias is null and the index field has a clusterAlias (clusterAlias:index notation) + * that means this is executing on a remote cluster (it was forwarded by the querying cluster). + * The clusterAlias is not Writeable, so it will be null in the ContextSetup on the remote cluster. + * We need to strip off the clusterAlias from the index before executing the script locally, + * so it will resolve to a local index + */ + assert split.length == 2 + : "If the index contains the REMOTE_CLUSTER_INDEX_SEPARATOR it should have only two parts but it has " + + Arrays.toString(split); + request.index(split[1]); + } + } + } + @Override protected Writeable.Reader getResponseReader() { return Response::new; diff --git a/modules/lang-painless/src/test/java/org/elasticsearch/painless/action/PainlessExecuteApiTests.java b/modules/lang-painless/src/test/java/org/elasticsearch/painless/action/PainlessExecuteApiTests.java index 3fcddc6c2c895..9da040e1d34bc 100644 --- a/modules/lang-painless/src/test/java/org/elasticsearch/painless/action/PainlessExecuteApiTests.java +++ b/modules/lang-painless/src/test/java/org/elasticsearch/painless/action/PainlessExecuteApiTests.java @@ -36,6 +36,7 @@ import static java.util.Collections.singletonMap; import static org.elasticsearch.painless.action.PainlessExecuteAction.TransportAction.innerShardOperation; import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.nullValue; public class PainlessExecuteApiTests extends ESSingleNodeTestCase { @@ -515,4 +516,43 @@ record ValidTestCase(String input, Tuple output) {} expectThrows(IllegalArgumentException.class, () -> Request.ContextSetup.parseClusterAliasAndIndex("remote1:foo,remote2:bar")); expectThrows(IllegalArgumentException.class, () -> Request.ContextSetup.parseClusterAliasAndIndex("a:b,c:d,e:f")); } + + public void testRemoveClusterAliasFromIndexExpression() { + { + // index expressions with no clusterAlias should come back unchanged + PainlessExecuteAction.Request request = createRequest("blogs"); + assertThat(request.index(), equalTo("blogs")); + PainlessExecuteAction.TransportAction.removeClusterAliasFromIndexExpression(request); + assertThat(request.index(), equalTo("blogs")); + } + { + // index expressions with no index specified should come back unchanged + PainlessExecuteAction.Request request = createRequest(null); + assertThat(request.index(), nullValue()); + PainlessExecuteAction.TransportAction.removeClusterAliasFromIndexExpression(request); + assertThat(request.index(), nullValue()); + } + { + // index expressions with clusterAlias should come back with it stripped off + PainlessExecuteAction.Request request = createRequest("remote1:blogs"); + assertThat(request.index(), equalTo("remote1:blogs")); + PainlessExecuteAction.TransportAction.removeClusterAliasFromIndexExpression(request); + assertThat(request.index(), equalTo("blogs")); + } + { + // index expressions with clusterAlias should come back with it stripped off + PainlessExecuteAction.Request request = createRequest("remote1:remote1"); + assertThat(request.index(), equalTo("remote1:remote1")); + PainlessExecuteAction.TransportAction.removeClusterAliasFromIndexExpression(request); + assertThat(request.index(), equalTo("remote1")); + } + } + + private PainlessExecuteAction.Request createRequest(String indexExpression) { + return new PainlessExecuteAction.Request( + new Script("100.0 / 1000.0"), + null, + new PainlessExecuteAction.Request.ContextSetup(indexExpression, null, null) + ); + } } diff --git a/server/src/main/java/org/elasticsearch/action/IndicesRequest.java b/server/src/main/java/org/elasticsearch/action/IndicesRequest.java index e9dfda3954c93..346b300349161 100644 --- a/server/src/main/java/org/elasticsearch/action/IndicesRequest.java +++ b/server/src/main/java/org/elasticsearch/action/IndicesRequest.java @@ -59,4 +59,17 @@ default boolean allowsRemoteIndices() { return false; } } + + /** + * For use cases where a Request instance cannot implement Replaceable due to not supporting wildcards + * and only supporting a single index at a time, this is an alternative interface that the + * security layer checks against to determine if remote indices are allowed for that Request type. + * + * This may change with https://github.com/elastic/elasticsearch/issues/105598 + */ + interface SingleIndexNoWildcards extends IndicesRequest { + default boolean allowsRemoteIndices() { + return true; + } + } } diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS1PainlessExecuteIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS1PainlessExecuteIT.java new file mode 100644 index 0000000000000..8fe54de5fe2f2 --- /dev/null +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS1PainlessExecuteIT.java @@ -0,0 +1,223 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.remotecluster; + +import org.apache.http.util.EntityUtils; +import org.elasticsearch.client.Request; +import org.elasticsearch.client.RequestOptions; +import org.elasticsearch.client.Response; +import org.elasticsearch.client.ResponseException; +import org.elasticsearch.core.Strings; +import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.junit.ClassRule; +import org.junit.rules.RuleChain; +import org.junit.rules.TestRule; + +import java.io.IOException; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.equalTo; + +/** + * Tests cross-cluster painless/execute API under RCS1.0 security model + */ +public class RemoteClusterSecurityRCS1PainlessExecuteIT extends AbstractRemoteClusterSecurityTestCase { + + static { + fulfillingCluster = ElasticsearchCluster.local().name("fulfilling-cluster").nodes(3).apply(commonClusterConfig).build(); + + queryCluster = ElasticsearchCluster.local().name("query-cluster").apply(commonClusterConfig).build(); + } + + @ClassRule + public static TestRule clusterRule = RuleChain.outerRule(fulfillingCluster).around(queryCluster); + + @SuppressWarnings({ "unchecked", "checkstyle:LineLength" }) + public void testPainlessExecute() throws Exception { + // Setup RCS 1.0 (basicSecurity=true) + configureRemoteCluster("my_remote_cluster", fulfillingCluster, true, randomBoolean(), randomBoolean()); + { + // Query cluster -> add role for test user - do not give any privileges for remote_indices + final var putRoleRequest = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); + putRoleRequest.setJsonEntity(""" + { + "indices": [ + { + "names": ["local_index", "my_local*"], + "privileges": ["read"] + } + ] + }"""); + assertOK(adminClient().performRequest(putRoleRequest)); + + // Query cluster -> create user and assign role + final var putUserRequest = new Request("PUT", "/_security/user/" + REMOTE_SEARCH_USER); + putUserRequest.setJsonEntity(""" + { + "password": "x-pack-test-password", + "roles" : ["remote_search"] + }"""); + assertOK(adminClient().performRequest(putUserRequest)); + + // Query cluster -> create test index + final var indexDocRequest = new Request("POST", "/local_index/_doc?refresh=true"); + indexDocRequest.setJsonEntity("{\"local_foo\": \"local_bar\"}"); + assertOK(client().performRequest(indexDocRequest)); + + // Fulfilling cluster -> create test indices + final Request bulkRequest = new Request("POST", "/_bulk?refresh=true"); + bulkRequest.setJsonEntity(Strings.format(""" + { "index": { "_index": "index1" } } + { "foo": "bar" } + { "index": { "_index": "secretindex" } } + { "bar": "foo" } + """)); + assertOK(performRequestAgainstFulfillingCluster(bulkRequest)); + } + + { + // TEST CASE 1: Query local cluster for local_index - should work since role has read perms for it + Request painlessExecuteLocal = createPainlessExecuteRequest("local_index"); + Response response = performRequestWithRemoteSearchUser(painlessExecuteLocal); + assertOK(response); + String responseBody = EntityUtils.toString(response.getEntity()); + assertThat(responseBody, equalTo("{\"result\":[\"test\"]}")); + } + { + // TEST CASE 2: Query remote cluster for index1 - should fail since no permissions granted for remote clusters yet + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index1"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [index1]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // add user role and user on remote cluster + var putRoleOnRemoteClusterRequest = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); + putRoleOnRemoteClusterRequest.setJsonEntity(""" + { + "indices": [ + { + "names": ["index*"], + "privileges": ["read", "read_cross_cluster"] + } + ] + }"""); + assertOK(performRequestAgainstFulfillingCluster(putRoleOnRemoteClusterRequest)); + + var putUserOnRemoteClusterRequest = new Request("PUT", "/_security/user/" + REMOTE_SEARCH_USER); + putUserOnRemoteClusterRequest.setJsonEntity(""" + { + "password": "x-pack-test-password", + "roles" : ["remote_search"] + }"""); + assertOK(performRequestAgainstFulfillingCluster(putUserOnRemoteClusterRequest)); + } + { + // TEST CASE 3: Query remote cluster for secretindex - should fail since no perms granted for it + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:secretindex"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [secretindex]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // TEST CASE 4: Query remote cluster for index1 - should succeed since read and cross-cluster-read perms granted + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index1"); + Response response = performRequestWithRemoteSearchUser(painlessExecuteRemote); + String responseBody = EntityUtils.toString(response.getEntity()); + assertOK(response); + assertThat(responseBody, equalTo("{\"result\":[\"test\"]}")); + } + { + // TEST CASE 5: Query local cluster for not_present index - should fail with 403 since role does not have perms for this index + Request painlessExecuteLocal = createPainlessExecuteRequest("index_not_present"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteLocal)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [index_not_present]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // TEST CASE 6: Query local cluster for my_local_123 index - role has perms for this pattern, but index does not exist, so 404 + Request painlessExecuteLocal = createPainlessExecuteRequest("my_local_123"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteLocal)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(404)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("\"type\":\"index_not_found_exception\"")); + } + { + // TEST CASE 7: Query local cluster for my_local* index - painless/execute does not allow wildcards, so fails with 400 + Request painlessExecuteLocal = createPainlessExecuteRequest("my_local*"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteLocal)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(400)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("indices:data/read/scripts/painless/execute does not support wildcards")); + assertThat(errorResponseBody, containsString("\"type\":\"illegal_argument_exception\"")); + } + { + // TEST CASE 8: Query remote cluster for cluster that does not exist, and user does not have perms for that pattern - 403 ??? + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:abc123"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [abc123]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // TEST CASE 9: Query remote cluster for cluster that does not exist, but has permissions for the index pattern - 404 + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index123"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(404)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("\"type\":\"index_not_found_exception\"")); + } + { + // TEST CASE 10: Query remote cluster with wildcard in index - painless/execute does not allow wildcards, so fails with 400 + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index*"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(400)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("indices:data/read/scripts/painless/execute does not support wildcards")); + assertThat(errorResponseBody, containsString("\"type\":\"illegal_argument_exception\"")); + } + } + + private static Request createPainlessExecuteRequest(String indexExpression) { + Request painlessExecuteLocal = new Request("POST", "_scripts/painless/_execute"); + String body = """ + { + "script": { + "source": "emit(\\"test\\")" + }, + "context": "keyword_field", + "context_setup": { + "index": "INDEX_EXPRESSION_HERE", + "document": { + "@timestamp": "2023-05-06T16:22:22.000Z" + } + } + }""".replace("INDEX_EXPRESSION_HERE", indexExpression); + painlessExecuteLocal.setJsonEntity(body); + return painlessExecuteLocal; + } + + private Response performRequestWithRemoteSearchUser(final Request request) throws IOException { + request.setOptions( + RequestOptions.DEFAULT.toBuilder().addHeader("Authorization", headerFromRandomAuthMethod(REMOTE_SEARCH_USER, PASS)) + ); + return client().performRequest(request); + } +} diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS2PainlessExecuteIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS2PainlessExecuteIT.java new file mode 100644 index 0000000000000..b24122c1302fc --- /dev/null +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityRCS2PainlessExecuteIT.java @@ -0,0 +1,303 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.remotecluster; + +import org.apache.http.util.EntityUtils; +import org.elasticsearch.client.Request; +import org.elasticsearch.client.RequestOptions; +import org.elasticsearch.client.Response; +import org.elasticsearch.client.ResponseException; +import org.elasticsearch.common.UUIDs; +import org.elasticsearch.core.Strings; +import org.elasticsearch.test.cluster.ElasticsearchCluster; +import org.elasticsearch.test.cluster.util.resource.Resource; +import org.elasticsearch.test.junit.RunnableTestRuleAdapter; +import org.junit.ClassRule; +import org.junit.rules.RuleChain; +import org.junit.rules.TestRule; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.util.Base64; +import java.util.Map; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.equalTo; + +/** + * Tests cross-cluster painless/execute API under RCS2.0 security model + */ +public class RemoteClusterSecurityRCS2PainlessExecuteIT extends AbstractRemoteClusterSecurityTestCase { + + private static final AtomicReference> API_KEY_MAP_REF = new AtomicReference<>(); + private static final AtomicReference> REST_API_KEY_MAP_REF = new AtomicReference<>(); + private static final AtomicBoolean SSL_ENABLED_REF = new AtomicBoolean(); + private static final AtomicBoolean NODE1_RCS_SERVER_ENABLED = new AtomicBoolean(); + private static final AtomicBoolean NODE2_RCS_SERVER_ENABLED = new AtomicBoolean(); + private static final AtomicInteger INVALID_SECRET_LENGTH = new AtomicInteger(); + + static { + fulfillingCluster = ElasticsearchCluster.local() + .name("fulfilling-cluster") + .nodes(3) + .apply(commonClusterConfig) + .setting("remote_cluster.port", "0") + .setting("xpack.security.remote_cluster_server.ssl.enabled", () -> String.valueOf(SSL_ENABLED_REF.get())) + .setting("xpack.security.remote_cluster_server.ssl.key", "remote-cluster.key") + .setting("xpack.security.remote_cluster_server.ssl.certificate", "remote-cluster.crt") + .setting("xpack.security.authc.token.enabled", "true") + .keystore("xpack.security.remote_cluster_server.ssl.secure_key_passphrase", "remote-cluster-password") + .node(0, spec -> spec.setting("remote_cluster_server.enabled", "true")) + .node(1, spec -> spec.setting("remote_cluster_server.enabled", () -> String.valueOf(NODE1_RCS_SERVER_ENABLED.get()))) + .node(2, spec -> spec.setting("remote_cluster_server.enabled", () -> String.valueOf(NODE2_RCS_SERVER_ENABLED.get()))) + .build(); + + queryCluster = ElasticsearchCluster.local() + .name("query-cluster") + .apply(commonClusterConfig) + .setting("xpack.security.remote_cluster_client.ssl.enabled", () -> String.valueOf(SSL_ENABLED_REF.get())) + .setting("xpack.security.remote_cluster_client.ssl.certificate_authorities", "remote-cluster-ca.crt") + .setting("xpack.security.authc.token.enabled", "true") + .keystore("cluster.remote.my_remote_cluster.credentials", () -> { + if (API_KEY_MAP_REF.get() == null) { + final Map apiKeyMap = createCrossClusterAccessApiKey(""" + { + "search": [ + { + "names": ["index*"] + } + ] + }"""); + API_KEY_MAP_REF.set(apiKeyMap); + } + return (String) API_KEY_MAP_REF.get().get("encoded"); + }) + // Define a bogus API key for another remote cluster + .keystore("cluster.remote.invalid_remote.credentials", randomEncodedApiKey()) + // Define remote with a REST API key to observe expected failure + .keystore("cluster.remote.wrong_api_key_type.credentials", () -> { + if (REST_API_KEY_MAP_REF.get() == null) { + initFulfillingClusterClient(); + final var createApiKeyRequest = new Request("POST", "/_security/api_key"); + createApiKeyRequest.setJsonEntity(""" + { + "name": "rest_api_key" + }"""); + try { + final Response createApiKeyResponse = performRequestWithAdminUser(fulfillingClusterClient, createApiKeyRequest); + assertOK(createApiKeyResponse); + REST_API_KEY_MAP_REF.set(responseAsMap(createApiKeyResponse)); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + return (String) REST_API_KEY_MAP_REF.get().get("encoded"); + }) + // Define a remote with invalid API key secret length + .keystore( + "cluster.remote.invalid_secret_length.credentials", + () -> Base64.getEncoder() + .encodeToString( + (UUIDs.base64UUID() + ":" + randomAlphaOfLength(INVALID_SECRET_LENGTH.get())).getBytes(StandardCharsets.UTF_8) + ) + ) + .rolesFile(Resource.fromClasspath("roles.yml")) + .user(REMOTE_METRIC_USER, PASS.toString(), "read_remote_shared_metrics", false) + .build(); + } + + @ClassRule + // Use a RuleChain to ensure that fulfilling cluster is started before query cluster + // `SSL_ENABLED_REF` is used to control the SSL-enabled setting on the test clusters + // We set it here, since randomization methods are not available in the static initialize context above + public static TestRule clusterRule = RuleChain.outerRule(new RunnableTestRuleAdapter(() -> { + SSL_ENABLED_REF.set(usually()); + NODE1_RCS_SERVER_ENABLED.set(randomBoolean()); + NODE2_RCS_SERVER_ENABLED.set(randomBoolean()); + INVALID_SECRET_LENGTH.set(randomValueOtherThan(22, () -> randomIntBetween(0, 99))); + })).around(fulfillingCluster).around(queryCluster); + + @SuppressWarnings({ "unchecked", "checkstyle:LineLength" }) + public void testPainlessExecute() throws Exception { + configureRemoteCluster(); + + { + // Query cluster -> add role for test user - do not give any privileges for remote_indices + final var putRoleRequest = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); + putRoleRequest.setJsonEntity(""" + { + "indices": [ + { + "names": ["local_index", "my_local*"], + "privileges": ["read"] + } + ] + }"""); + assertOK(adminClient().performRequest(putRoleRequest)); + + // Query cluster -> create user and assign role + final var putUserRequest = new Request("PUT", "/_security/user/" + REMOTE_SEARCH_USER); + putUserRequest.setJsonEntity(""" + { + "password": "x-pack-test-password", + "roles" : ["remote_search"] + }"""); + assertOK(adminClient().performRequest(putUserRequest)); + + // Query cluster -> create test index + final var indexDocRequest = new Request("POST", "/local_index/_doc?refresh=true"); + indexDocRequest.setJsonEntity("{\"local_foo\": \"local_bar\"}"); + assertOK(client().performRequest(indexDocRequest)); + + // Fulfilling cluster -> create test indices + final Request bulkRequest = new Request("POST", "/_bulk?refresh=true"); + bulkRequest.setJsonEntity(Strings.format(""" + { "index": { "_index": "index1" } } + { "foo": "bar" } + { "index": { "_index": "secretindex" } } + { "bar": "foo" } + """)); + assertOK(performRequestAgainstFulfillingCluster(bulkRequest)); + } + + { + // TEST CASE 1: Query local cluster for local_index - should work since role has read perms for it + Request painlessExecuteLocal = createPainlessExecuteRequest("local_index"); + + Response response = performRequestWithRemoteSearchUser(painlessExecuteLocal); + assertOK(response); + String responseBody = EntityUtils.toString(response.getEntity()); + assertThat(responseBody, equalTo("{\"result\":[\"test\"]}")); + } + { + // update role to have permissions to remote index* pattern + var updateRoleRequest = new Request("PUT", "/_security/role/" + REMOTE_SEARCH_ROLE); + updateRoleRequest.setJsonEntity(""" + { + "indices": [ + { + "names": ["local_index", "my_local*"], + "privileges": ["read"] + } + ], + "remote_indices": [ + { + "names": ["index*"], + "privileges": ["read", "read_cross_cluster"], + "clusters": ["my_remote_cluster"] + } + ] + }"""); + + assertOK(adminClient().performRequest(updateRoleRequest)); + } + { + // TEST CASE 2: Query remote cluster for secretindex - should fail since no perms granted for it + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:secretindex"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [secretindex]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // TEST CASE 3: Query remote cluster for index1 - should succeed since read and cross-cluster-read perms granted + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index1"); + Response response = performRequestWithRemoteSearchUser(painlessExecuteRemote); + String responseBody = EntityUtils.toString(response.getEntity()); + assertOK(response); + assertThat(responseBody, equalTo("{\"result\":[\"test\"]}")); + } + { + // TEST CASE 4: Query local cluster for not_present index - should fail with 403 since role does not have perms for this index + Request painlessExecuteLocal = createPainlessExecuteRequest("index_not_present"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteLocal)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [index_not_present]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // TEST CASE 5: Query local cluster for my_local_123 index - role has perms for this pattern, but index does not exist, so 404 + Request painlessExecuteLocal = createPainlessExecuteRequest("my_local_123"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteLocal)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(404)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("\"type\":\"index_not_found_exception\"")); + } + { + // TEST CASE 6: Query local cluster for my_local* index - painless/execute does not allow wildcards, so fails with 400 + Request painlessExecuteLocal = createPainlessExecuteRequest("my_local*"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteLocal)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(400)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("indices:data/read/scripts/painless/execute does not support wildcards")); + assertThat(errorResponseBody, containsString("\"type\":\"illegal_argument_exception\"")); + } + { + // TEST CASE 7: Query remote cluster for cluster that does not exist, and user does not have perms for that pattern - 403 ??? + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:abc123"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(403)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("unauthorized for user [remote_search_user]")); + assertThat(errorResponseBody, containsString("on indices [abc123]")); + assertThat(errorResponseBody, containsString("\"type\":\"security_exception\"")); + } + { + // TEST CASE 8: Query remote cluster for cluster that does not exist, but has permissions for the index pattern - 404 + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index123"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(404)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("\"type\":\"index_not_found_exception\"")); + } + { + // TEST CASE 9: Query remote cluster with wildcard in index - painless/execute does not allow wildcards, so fails with 400 + Request painlessExecuteRemote = createPainlessExecuteRequest("my_remote_cluster:index*"); + ResponseException exc = expectThrows(ResponseException.class, () -> performRequestWithRemoteSearchUser(painlessExecuteRemote)); + assertThat(exc.getResponse().getStatusLine().getStatusCode(), is(400)); + String errorResponseBody = EntityUtils.toString(exc.getResponse().getEntity()); + assertThat(errorResponseBody, containsString("indices:data/read/scripts/painless/execute does not support wildcards")); + assertThat(errorResponseBody, containsString("\"type\":\"illegal_argument_exception\"")); + } + } + + private static Request createPainlessExecuteRequest(String indexExpression) { + Request painlessExecuteLocal = new Request("POST", "_scripts/painless/_execute"); + String body = """ + { + "script": { + "source": "emit(\\"test\\")" + }, + "context": "keyword_field", + "context_setup": { + "index": "INDEX_EXPRESSION_HERE", + "document": { + "@timestamp": "2023-05-06T16:22:22.000Z" + } + } + }""".replace("INDEX_EXPRESSION_HERE", indexExpression); + painlessExecuteLocal.setJsonEntity(body); + return painlessExecuteLocal; + } + + private Response performRequestWithRemoteSearchUser(final Request request) throws IOException { + request.setOptions( + RequestOptions.DEFAULT.toBuilder().addHeader("Authorization", headerFromRandomAuthMethod(REMOTE_SEARCH_USER, PASS)) + ); + return client().performRequest(request); + } +} diff --git a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityResolveClusterIT.java b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityResolveClusterIT.java index e4b752613b753..da6d930371bc9 100644 --- a/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityResolveClusterIT.java +++ b/x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/RemoteClusterSecurityResolveClusterIT.java @@ -175,7 +175,6 @@ public void testResolveCluster() throws Exception { Response response = performRequestWithRemoteSearchUser(starResolveRequest); assertOK(response); Map responseMap = responseAsMap(response); - System.err.println(">> XXX CASE1 remoteClusterResponse: " + responseMap); assertLocalMatching(responseMap); Map remoteClusterResponse = (Map) responseMap.get("my_remote_cluster"); diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/AuthorizationService.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/AuthorizationService.java index c8920d0f498d0..c886f2fde55ab 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/AuthorizationService.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/AuthorizationService.java @@ -477,7 +477,7 @@ private void authorizeAction( resolvedIndicesListener.onResponse(resolvedIndices); return; } - final ResolvedIndices resolvedIndices = IndicesAndAliasesResolver.tryResolveWithoutWildcards(action, request); + final ResolvedIndices resolvedIndices = indicesAndAliasesResolver.tryResolveWithoutWildcards(action, request); if (resolvedIndices != null) { resolvedIndicesListener.onResponse(resolvedIndices); } else { @@ -870,8 +870,8 @@ private void authorizeBulkItems( }, listener::onFailure)); } - private static String resolveIndexNameDateMath(BulkItemRequest bulkItemRequest) { - final ResolvedIndices resolvedIndices = IndicesAndAliasesResolver.resolveIndicesAndAliasesWithoutWildcards( + private String resolveIndexNameDateMath(BulkItemRequest bulkItemRequest) { + final ResolvedIndices resolvedIndices = indicesAndAliasesResolver.resolveIndicesAndAliasesWithoutWildcards( getAction(bulkItemRequest), bulkItemRequest.request() ); diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java index a4163b6f10fc0..bf1bf7b7d3cee 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java @@ -46,7 +46,6 @@ import java.util.concurrent.CopyOnWriteArraySet; import java.util.function.Predicate; import java.util.function.Supplier; -import java.util.stream.Stream; import static org.elasticsearch.xpack.core.security.authz.IndicesAndAliasesResolverField.NO_INDEX_PLACEHOLDER; @@ -120,7 +119,7 @@ ResolvedIndices resolve( * @return The {@link ResolvedIndices} or null if wildcard expansion must be performed. */ @Nullable - static ResolvedIndices tryResolveWithoutWildcards(String action, TransportRequest transportRequest) { + ResolvedIndices tryResolveWithoutWildcards(String action, TransportRequest transportRequest) { // We only take care of IndicesRequest if (false == transportRequest instanceof IndicesRequest) { return null; @@ -145,7 +144,7 @@ private static boolean requiresWildcardExpansion(IndicesRequest indicesRequest) return false; } - static ResolvedIndices resolveIndicesAndAliasesWithoutWildcards(String action, IndicesRequest indicesRequest) { + ResolvedIndices resolveIndicesAndAliasesWithoutWildcards(String action, IndicesRequest indicesRequest) { assert false == requiresWildcardExpansion(indicesRequest) : "request must not require wildcard expansion"; final String[] indices = indicesRequest.indices(); if (indices == null || indices.length == 0) { @@ -162,20 +161,28 @@ static ResolvedIndices resolveIndicesAndAliasesWithoutWildcards(String action, I ); } + final ResolvedIndices split; + if (indicesRequest instanceof IndicesRequest.SingleIndexNoWildcards single && single.allowsRemoteIndices()) { + split = remoteClusterResolver.splitLocalAndRemoteIndexNames(indicesRequest.indices()); + } else { + split = new ResolvedIndices(Arrays.asList(indicesRequest.indices()), List.of()); + } + // NOTE: shard level requests do support wildcards (as they hold the original indices options) but don't support // replacing their indices. // That is fine though because they never contain wildcards, as they get replaced as part of the authorization of their // corresponding parent request on the coordinating node. Hence wildcards don't need to get replaced nor exploded for // shard level requests. - final List localIndices = new ArrayList<>(indices.length); - for (String name : indices) { + final List localIndices = new ArrayList<>(split.getLocal().size()); + for (String localName : split.getLocal()) { // TODO: Shard level requests have wildcard expanded already and do not need go through this check - if (Regex.isSimpleMatchPattern(name)) { - throwOnUnexpectedWildcards(action, indices); + if (Regex.isSimpleMatchPattern(localName)) { + throwOnUnexpectedWildcards(action, split.getLocal()); } - localIndices.add(IndexNameExpressionResolver.resolveDateMathExpression(name)); + localIndices.add(IndexNameExpressionResolver.resolveDateMathExpression(localName)); } - return new ResolvedIndices(localIndices, List.of()); + + return new ResolvedIndices(localIndices, split.getRemote()); } /** @@ -196,8 +203,8 @@ ResolvedIndices resolvePITIndices(SearchRequest request) { return split; } - private static void throwOnUnexpectedWildcards(String action, String[] indices) { - final List wildcards = Stream.of(indices).filter(Regex::isSimpleMatchPattern).toList(); + private static void throwOnUnexpectedWildcards(String action, List indices) { + final List wildcards = indices.stream().filter(Regex::isSimpleMatchPattern).toList(); assert wildcards.isEmpty() == false : "we already know that there's at least one wildcard in the indices"; throw new IllegalArgumentException( "the action " diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/RBACEngine.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/RBACEngine.java index 1b1d7c789b21d..05608187fcc68 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/RBACEngine.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/RBACEngine.java @@ -408,7 +408,12 @@ public void authorizeIndexAction( } private static boolean allowsRemoteIndices(TransportRequest transportRequest) { - return transportRequest instanceof IndicesRequest.Replaceable replaceable && replaceable.allowsRemoteIndices(); + // TODO this may need to change. See https://github.com/elastic/elasticsearch/issues/105598 + if (transportRequest instanceof IndicesRequest.SingleIndexNoWildcards single) { + return single.allowsRemoteIndices(); + } else { + return transportRequest instanceof IndicesRequest.Replaceable replaceable && replaceable.allowsRemoteIndices(); + } } private static boolean isChildActionAuthorizedByParentOnLocalNode(RequestInfo requestInfo, AuthorizationInfo authorizationInfo) { diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java index 7a70bb23e0db4..a7f6279fbe6f5 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java @@ -404,7 +404,7 @@ public void testDashIndicesAreAllowedInShardLevelRequests() { // aliases with names starting with '-' or '+' can be created up to version 5.x and can be around in 6.x ShardSearchRequest request = mock(ShardSearchRequest.class); when(request.indices()).thenReturn(new String[] { "-index10", "-index20", "+index30" }); - List indices = IndicesAndAliasesResolver.resolveIndicesAndAliasesWithoutWildcards( + List indices = defaultIndicesResolver.resolveIndicesAndAliasesWithoutWildcards( TransportSearchAction.TYPE.name() + "[s]", request ).getLocal(); @@ -418,7 +418,7 @@ public void testWildcardsAreNotAllowedInShardLevelRequests() { when(request.indices()).thenReturn(new String[] { "index*" }); IllegalArgumentException exception = expectThrows( IllegalArgumentException.class, - () -> IndicesAndAliasesResolver.resolveIndicesAndAliasesWithoutWildcards(TransportSearchAction.TYPE.name() + "[s]", request) + () -> defaultIndicesResolver.resolveIndicesAndAliasesWithoutWildcards(TransportSearchAction.TYPE.name() + "[s]", request) ); assertThat( exception, @@ -443,7 +443,7 @@ public void testAllIsNotAllowedInShardLevelRequests() { } IllegalArgumentException exception = expectThrows( IllegalArgumentException.class, - () -> IndicesAndAliasesResolver.resolveIndicesAndAliasesWithoutWildcards(TransportSearchAction.TYPE.name() + "[s]", request) + () -> defaultIndicesResolver.resolveIndicesAndAliasesWithoutWildcards(TransportSearchAction.TYPE.name() + "[s]", request) ); assertThat( From 8be33ae1892b267675c87a2b0806d8b8f79ab621 Mon Sep 17 00:00:00 2001 From: Ignacio Vera Date: Mon, 19 Feb 2024 08:30:22 +0100 Subject: [PATCH 22/45] GlobalOrdCardinalityAggregator should use HyperLogLogPlusPlus instead of HyperLogLogPlusPlusSparse (#105546) Use the generic HyperLogLogPlusPlus on GlobalOrdCardinalityAggregator so we promote the algorihtm to HLL when we reach the linear counting threshold. --- docs/changelog/105546.yaml | 6 ++++ .../GlobalOrdCardinalityAggregator.java | 5 ++- .../metrics/CardinalityAggregatorTests.java | 32 +++++++++++++++++++ 3 files changed, 40 insertions(+), 3 deletions(-) create mode 100644 docs/changelog/105546.yaml diff --git a/docs/changelog/105546.yaml b/docs/changelog/105546.yaml new file mode 100644 index 0000000000000..0b54e124f2495 --- /dev/null +++ b/docs/changelog/105546.yaml @@ -0,0 +1,6 @@ +pr: 105546 +summary: '`GlobalOrdCardinalityAggregator` should use `HyperLogLogPlusPlus` instead + of `HyperLogLogPlusPlusSparse`' +area: Aggregations +type: bug +issues: [] diff --git a/server/src/main/java/org/elasticsearch/search/aggregations/metrics/GlobalOrdCardinalityAggregator.java b/server/src/main/java/org/elasticsearch/search/aggregations/metrics/GlobalOrdCardinalityAggregator.java index 5e59da1591270..7661c3122db00 100644 --- a/server/src/main/java/org/elasticsearch/search/aggregations/metrics/GlobalOrdCardinalityAggregator.java +++ b/server/src/main/java/org/elasticsearch/search/aggregations/metrics/GlobalOrdCardinalityAggregator.java @@ -65,7 +65,7 @@ public class GlobalOrdCardinalityAggregator extends NumericMetricsAggregator.Sin // Build at post-collection phase @Nullable - private HyperLogLogPlusPlusSparse counts; + private HyperLogLogPlusPlus counts; private ObjectArray visitedOrds; private SortedSetDocValues values; @@ -285,7 +285,7 @@ public void collect(int doc, long bucketOrd) throws IOException { } protected void doPostCollection() throws IOException { - counts = new HyperLogLogPlusPlusSparse(precision, bigArrays, visitedOrds.size()); + counts = new HyperLogLogPlusPlus(precision, bigArrays, visitedOrds.size()); try (LongArray hashes = bigArrays.newLongArray(maxOrd, false)) { try (BitArray allVisitedOrds = new BitArray(maxOrd, bigArrays)) { for (long bucket = visitedOrds.size() - 1; bucket >= 0; --bucket) { @@ -308,7 +308,6 @@ protected void doPostCollection() throws IOException { try (BitArray bits = visitedOrds.get(bucket)) { if (bits != null) { visitedOrds.set(bucket, null); // remove bitset from array - counts.ensureCapacity(bucket, bits.cardinality()); for (long ord = bits.nextSetBit(0); ord < Long.MAX_VALUE; ord = ord + 1 < maxOrd ? bits.nextSetBit(ord + 1) : Long.MAX_VALUE) { diff --git a/server/src/test/java/org/elasticsearch/search/aggregations/metrics/CardinalityAggregatorTests.java b/server/src/test/java/org/elasticsearch/search/aggregations/metrics/CardinalityAggregatorTests.java index 3d02aa948d635..6efcb6c2b99e2 100644 --- a/server/src/test/java/org/elasticsearch/search/aggregations/metrics/CardinalityAggregatorTests.java +++ b/server/src/test/java/org/elasticsearch/search/aggregations/metrics/CardinalityAggregatorTests.java @@ -25,8 +25,10 @@ import org.apache.lucene.tests.index.RandomIndexWriter; import org.apache.lucene.util.BytesRef; import org.elasticsearch.common.geo.GeoPoint; +import org.elasticsearch.common.hash.MurmurHash3; import org.elasticsearch.common.network.InetAddresses; import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.common.util.BigArrays; import org.elasticsearch.core.CheckedConsumer; import org.elasticsearch.index.fielddata.ScriptDocValues; import org.elasticsearch.index.mapper.IpFieldMapper; @@ -290,6 +292,36 @@ public void testIndexedSingleValuedString() throws IOException { ); } + public void testGlobalOrdinals() throws IOException { + // aggregation with minimum precision so we don't need to generate too many distinct values + final CardinalityAggregationBuilder aggregationBuilder = new CardinalityAggregationBuilder("name").field("str_value") + .precisionThreshold(1); + final MappedFieldType mappedFieldTypes = new KeywordFieldMapper.KeywordFieldType("str_value"); + + // range big enough that will force force promotion to HHL the time to time + final BytesRef[] differentValues = new BytesRef[randomIntBetween(10, 30)]; + for (int i = 0; i < differentValues.length; i++) { + differentValues[i] = new BytesRef(randomAlphaOfLength(8)); + } + // large number of documents to force global ordinals + final int numDocs = randomIntBetween(1000, 10000); + final HyperLogLogPlusPlus hll = new HyperLogLogPlusPlus(HyperLogLogPlusPlus.MIN_PRECISION, BigArrays.NON_RECYCLING_INSTANCE, 1); + final MurmurHash3.Hash128 hash = new MurmurHash3.Hash128(); + final CheckedConsumer buildIndex = iw -> { + for (int i = 0; i < numDocs; i++) { + final BytesRef value = differentValues[i % differentValues.length]; + MurmurHash3.hash128(value.bytes, value.offset, value.length, 0, hash); + hll.collect(0, hash.h1); + iw.addDocument(List.of(new SortedDocValuesField("str_value", value))); + } + }; + + testAggregation(aggregationBuilder, new MatchAllDocsQuery(), buildIndex, card -> { + assertEquals(hll.cardinality(0), card.getValue(), 0); + assertTrue(AggregationInspectionHelper.hasValue(card)); + }, mappedFieldTypes); + } + public void testIndexedSingleValuedIP() throws IOException { // IP addresses are interesting to test because they use sorted doc values like keywords, but index data using points rather than an // inverted index, so this triggers a different code path to disable dynamic pruning From 5f7249b4a24394f88116b3516142c0d352198e33 Mon Sep 17 00:00:00 2001 From: Chris Cressman Date: Mon, 19 Feb 2024 02:53:18 -0500 Subject: [PATCH 23/45] [DOCS] Promote webinar on Elasticsearch docs landing (#105594) Collaboration with Elastic Marketing --- docs/reference/landing-page.asciidoc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/reference/landing-page.asciidoc b/docs/reference/landing-page.asciidoc index 6d6c257f0c594..e781dc0aff4e3 100644 --- a/docs/reference/landing-page.asciidoc +++ b/docs/reference/landing-page.asciidoc @@ -79,6 +79,11 @@

Get to know Elasticsearch

+

+ New webinar: + Architect search apps with Google Cloud +

+

From 0c573b821b4546e4ffa7a386ca76964a92eb9fd7 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 19 Feb 2024 08:51:01 +0000 Subject: [PATCH 24/45] Remove dead comment in ActionRequest ctor (#105604) The `listenerThreaded` concept only existed before 2.x, it was removed[^1] almost 9 years ago. This comment is definitely no longer needed. [^1]: see b87d360e79726813db90c7acecc965057a4a39ed. --- .../src/main/java/org/elasticsearch/action/ActionRequest.java | 3 --- 1 file changed, 3 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/action/ActionRequest.java b/server/src/main/java/org/elasticsearch/action/ActionRequest.java index d4593255a2e72..02b3001343e7f 100644 --- a/server/src/main/java/org/elasticsearch/action/ActionRequest.java +++ b/server/src/main/java/org/elasticsearch/action/ActionRequest.java @@ -18,9 +18,6 @@ public abstract class ActionRequest extends TransportRequest { public ActionRequest() { super(); - // this does not set the listenerThreaded API, if needed, its up to the caller to set it - // since most times, we actually want it to not be threaded... - // this.listenerThreaded = request.listenerThreaded(); } public ActionRequest(StreamInput in) throws IOException { From 0c3e765f45f768da5a6b332ae95c11dcd73b9186 Mon Sep 17 00:00:00 2001 From: Nikolaj Volgushev Date: Mon, 19 Feb 2024 10:14:52 +0100 Subject: [PATCH 25/45] Support Put Roles API customization (#105336) This PR adds support for injecting a `PutRoleRequest` builder to allow us to customize Put Roles API behavior. It also includes a few related changes: 1. Renames `responseRestricted` methods and vars to `pathRestricted` in the operator route check. This is more appropriate since the request or response may be restricted. To avoid breaking serverless CI, I'll rename the actual value in a follow up (where that rename will be the only change and thus easier to rule out other causes for test failures). 2. Exposes a few convenient methods 3. Adds an `exists()` check for role names on the `FileRolesStore` Relates: ES-7825. --- .../org/elasticsearch/rest/RestRequest.java | 20 ++++++++--- .../org/elasticsearch/rest/RestUtils.java | 6 ++-- .../elasticsearch/rest/RestRequestTests.java | 18 +++++----- .../elasticsearch/rest/RestUtilsTests.java | 6 ++-- .../role/PutRoleRequestBuilderFactory.java | 25 +++++++++++++ .../core/security/authz/RoleDescriptor.java | 4 +-- .../privilege/ClusterPrivilegeResolver.java | 8 ++++- .../authz/privilege/IndexPrivilege.java | 6 ++++ .../security/authz/privilege/Privilege.java | 2 +- .../xpack/security/Security.java | 35 ++++++++++++++----- .../action/role/TransportPutRoleAction.java | 1 + .../security/authz/store/FileRolesStore.java | 5 +++ .../operator/OperatorOnlyRegistry.java | 2 +- .../rest/action/role/RestPutRoleAction.java | 25 +++++++++---- .../authz/store/FileRolesStoreTests.java | 33 +++++++++++++++++ .../action/role/RestPutRoleActionTests.java | 2 +- 16 files changed, 156 insertions(+), 42 deletions(-) create mode 100644 x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestBuilderFactory.java diff --git a/server/src/main/java/org/elasticsearch/rest/RestRequest.java b/server/src/main/java/org/elasticsearch/rest/RestRequest.java index 3e418d5ca5d4e..e3e27ddf5cf5b 100644 --- a/server/src/main/java/org/elasticsearch/rest/RestRequest.java +++ b/server/src/main/java/org/elasticsearch/rest/RestRequest.java @@ -48,7 +48,11 @@ public class RestRequest implements ToXContent.Params, Traceable { + @Deprecated() + // TODO remove once Serverless is updated public static final String RESPONSE_RESTRICTED = "responseRestricted"; + // TODO rename to `pathRestricted` once Serverless is updated + public static final String PATH_RESTRICTED = "responseRestricted"; // tchar pattern as defined by RFC7230 section 3.2.6 private static final Pattern TCHAR_PATTERN = Pattern.compile("[a-zA-Z0-9!#$%&'*+\\-.\\^_`|~]+"); @@ -616,13 +620,19 @@ public boolean hasExplicitRestApiVersion() { return restApiVersion.isPresent(); } - public void markResponseRestricted(String restriction) { - if (params.containsKey(RESPONSE_RESTRICTED)) { - throw new IllegalArgumentException("The parameter [" + RESPONSE_RESTRICTED + "] is already defined."); + public void markPathRestricted(String restriction) { + if (params.containsKey(PATH_RESTRICTED)) { + throw new IllegalArgumentException("The parameter [" + PATH_RESTRICTED + "] is already defined."); } - params.put(RESPONSE_RESTRICTED, restriction); + params.put(PATH_RESTRICTED, restriction); // this parameter is intended be consumed via ToXContent.Params.param(..), not this.params(..) so don't require it is consumed here - consumedParams.add(RESPONSE_RESTRICTED); + consumedParams.add(PATH_RESTRICTED); + } + + @Deprecated() + // TODO remove once Serverless is updated + public void markResponseRestricted(String restriction) { + markPathRestricted(restriction); } @Override diff --git a/server/src/main/java/org/elasticsearch/rest/RestUtils.java b/server/src/main/java/org/elasticsearch/rest/RestUtils.java index c6d4f9ae38853..aa693f38a3e6d 100644 --- a/server/src/main/java/org/elasticsearch/rest/RestUtils.java +++ b/server/src/main/java/org/elasticsearch/rest/RestUtils.java @@ -19,7 +19,7 @@ import java.util.Optional; import java.util.regex.Pattern; -import static org.elasticsearch.rest.RestRequest.RESPONSE_RESTRICTED; +import static org.elasticsearch.rest.RestRequest.PATH_RESTRICTED; public class RestUtils { @@ -81,8 +81,8 @@ private static String decodeQueryStringParam(final String s) { } private static void addParam(Map params, String name, String value) { - if (RESPONSE_RESTRICTED.equalsIgnoreCase(name)) { - throw new IllegalArgumentException("parameter [" + RESPONSE_RESTRICTED + "] is reserved and may not set"); + if (PATH_RESTRICTED.equalsIgnoreCase(name)) { + throw new IllegalArgumentException("parameter [" + PATH_RESTRICTED + "] is reserved and may not set"); } params.put(name, value); } diff --git a/server/src/test/java/org/elasticsearch/rest/RestRequestTests.java b/server/src/test/java/org/elasticsearch/rest/RestRequestTests.java index 17e79b3f54b67..809bf528ba194 100644 --- a/server/src/test/java/org/elasticsearch/rest/RestRequestTests.java +++ b/server/src/test/java/org/elasticsearch/rest/RestRequestTests.java @@ -31,7 +31,7 @@ import static java.util.Collections.emptyMap; import static java.util.Collections.singletonMap; -import static org.elasticsearch.rest.RestRequest.RESPONSE_RESTRICTED; +import static org.elasticsearch.rest.RestRequest.PATH_RESTRICTED; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.instanceOf; import static org.hamcrest.Matchers.is; @@ -249,20 +249,20 @@ public void testRequiredContent() { assertEquals("unknown content type", e.getMessage()); } - public void testMarkResponseRestricted() { + public void testMarkPathRestricted() { RestRequest request1 = contentRestRequest("content", new HashMap<>()); - request1.markResponseRestricted("foo"); - assertEquals(request1.param(RESPONSE_RESTRICTED), "foo"); - IllegalArgumentException exception = expectThrows(IllegalArgumentException.class, () -> request1.markResponseRestricted("foo")); - assertThat(exception.getMessage(), is("The parameter [" + RESPONSE_RESTRICTED + "] is already defined.")); + request1.markPathRestricted("foo"); + assertEquals(request1.param(PATH_RESTRICTED), "foo"); + IllegalArgumentException exception = expectThrows(IllegalArgumentException.class, () -> request1.markPathRestricted("foo")); + assertThat(exception.getMessage(), is("The parameter [" + PATH_RESTRICTED + "] is already defined.")); RestRequest request2 = contentRestRequest("content", new HashMap<>() { { - put(RESPONSE_RESTRICTED, "foo"); + put(PATH_RESTRICTED, "foo"); } }); - exception = expectThrows(IllegalArgumentException.class, () -> request2.markResponseRestricted("bar")); - assertThat(exception.getMessage(), is("The parameter [" + RESPONSE_RESTRICTED + "] is already defined.")); + exception = expectThrows(IllegalArgumentException.class, () -> request2.markPathRestricted("bar")); + assertThat(exception.getMessage(), is("The parameter [" + PATH_RESTRICTED + "] is already defined.")); } public static RestRequest contentRestRequest(String content, Map params) { diff --git a/server/src/test/java/org/elasticsearch/rest/RestUtilsTests.java b/server/src/test/java/org/elasticsearch/rest/RestUtilsTests.java index beb944aeee928..ca516a00af239 100644 --- a/server/src/test/java/org/elasticsearch/rest/RestUtilsTests.java +++ b/server/src/test/java/org/elasticsearch/rest/RestUtilsTests.java @@ -16,7 +16,7 @@ import java.util.Map; import java.util.regex.Pattern; -import static org.elasticsearch.rest.RestRequest.RESPONSE_RESTRICTED; +import static org.elasticsearch.rest.RestRequest.PATH_RESTRICTED; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.nullValue; @@ -159,12 +159,12 @@ public void testCrazyURL() { public void testReservedParameters() { Map params = new HashMap<>(); - String uri = "something?" + RESPONSE_RESTRICTED + "=value"; + String uri = "something?" + PATH_RESTRICTED + "=value"; IllegalArgumentException exception = expectThrows( IllegalArgumentException.class, () -> RestUtils.decodeQueryString(uri, uri.indexOf('?') + 1, params) ); - assertEquals(exception.getMessage(), "parameter [" + RESPONSE_RESTRICTED + "] is reserved and may not set"); + assertEquals(exception.getMessage(), "parameter [" + PATH_RESTRICTED + "] is reserved and may not set"); } private void assertCorsSettingRegexIsNull(String settingsValue) { diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestBuilderFactory.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestBuilderFactory.java new file mode 100644 index 0000000000000..f965ec754a404 --- /dev/null +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/role/PutRoleRequestBuilderFactory.java @@ -0,0 +1,25 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +package org.elasticsearch.xpack.core.security.action.role; + +import org.elasticsearch.client.internal.Client; + +import java.util.function.Predicate; + +public interface PutRoleRequestBuilderFactory { + PutRoleRequestBuilder create(Client client, boolean restrictRequest, Predicate fileRolesStoreNameChecker); + + class Default implements PutRoleRequestBuilderFactory { + @Override + public PutRoleRequestBuilder create(Client client, boolean restrictRequest, Predicate fileRolesStoreNameChecker) { + // by default, we don't apply extra restrictions to Put Role requests and don't require checks against file-based roles + // these dependencies are only used by our stateless implementation + return new PutRoleRequestBuilder(client); + } + } +} diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java index 2857cbfd1bdd2..e9aa982a05d8b 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/RoleDescriptor.java @@ -621,7 +621,7 @@ private static XContentParser createParser(BytesReference source, XContentType x return XContentHelper.createParserNotCompressed(LoggingDeprecationHandler.XCONTENT_PARSER_CONFIG, source, xContentType); } - private static RoleDescriptor.IndicesPrivileges[] parseIndices(String roleName, XContentParser parser, boolean allow2xFormat) + public static RoleDescriptor.IndicesPrivileges[] parseIndices(String roleName, XContentParser parser, boolean allow2xFormat) throws IOException { if (parser.currentToken() != XContentParser.Token.START_ARRAY) { throw new ElasticsearchParseException( @@ -954,7 +954,7 @@ private static void checkIfExceptFieldsIsSubsetOfGrantedFields(String roleName, } } - private static ApplicationResourcePrivileges[] parseApplicationPrivileges(String roleName, XContentParser parser) throws IOException { + public static ApplicationResourcePrivileges[] parseApplicationPrivileges(String roleName, XContentParser parser) throws IOException { if (parser.currentToken() != XContentParser.Token.START_ARRAY) { throw new ElasticsearchParseException( "failed to parse application privileges for role [{}]. expected field [{}] value " diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java index 63179c7cf09e6..c514fb07cc32b 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java @@ -23,6 +23,7 @@ import org.elasticsearch.action.ingest.SimulatePipelineAction; import org.elasticsearch.common.Strings; import org.elasticsearch.common.util.Maps; +import org.elasticsearch.core.Nullable; import org.elasticsearch.transport.RemoteClusterService; import org.elasticsearch.transport.TransportRequest; import org.elasticsearch.xpack.core.action.XPackInfoAction; @@ -430,6 +431,11 @@ public static NamedClusterPrivilege resolve(String name) { } + @Nullable + public static NamedClusterPrivilege getNamedOrNull(String name) { + return VALUES.get(Objects.requireNonNull(name).toLowerCase(Locale.ROOT)); + } + public static Set names() { return Collections.unmodifiableSet(VALUES.keySet()); } @@ -461,7 +467,7 @@ public static Collection findPrivilegesThatGrant(String action, Transpor * Sorts the collection of privileges from least-privilege to most-privilege (to the extent possible), * returning them in a sorted map keyed by name. */ - static SortedMap sortByAccessLevel(Collection privileges) { + public static SortedMap sortByAccessLevel(Collection privileges) { // How many other privileges does this privilege imply. Those with a higher count are considered to be a higher privilege final Map impliesCount = Maps.newMapWithExpectedSize(privileges.size()); privileges.forEach( diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/IndexPrivilege.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/IndexPrivilege.java index 1d7fa83658e54..ba00864148c24 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/IndexPrivilege.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/IndexPrivilege.java @@ -32,6 +32,7 @@ import org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction; import org.elasticsearch.action.search.TransportSearchShardsAction; import org.elasticsearch.common.Strings; +import org.elasticsearch.core.Nullable; import org.elasticsearch.index.seqno.RetentionLeaseActions; import org.elasticsearch.xpack.core.ccr.action.ForgetFollowerAction; import org.elasticsearch.xpack.core.ccr.action.PutFollowAction; @@ -258,6 +259,11 @@ public static IndexPrivilege get(Set name) { }); } + @Nullable + public static IndexPrivilege getNamedOrNull(String name) { + return VALUES.get(name.toLowerCase(Locale.ROOT)); + } + private static IndexPrivilege resolve(Set name) { final int size = name.size(); if (size == 0) { diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/Privilege.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/Privilege.java index 47f95ee51dc8f..68e3f11751aac 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/Privilege.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/Privilege.java @@ -84,7 +84,7 @@ public Automaton getAutomaton() { /** * Sorts the map of privileges from least-privilege to most-privilege */ - static SortedMap sortByAccessLevel(Map privileges) { + public static SortedMap sortByAccessLevel(Map privileges) { // How many other privileges is this privilege a subset of. Those with a higher count are considered to be a lower privilege final Map subsetCount = Maps.newMapWithExpectedSize(privileges.size()); privileges.forEach( diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java index c59ccd8f73ed0..763eb2616175c 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java @@ -147,6 +147,7 @@ import org.elasticsearch.xpack.core.security.action.role.DeleteRoleAction; import org.elasticsearch.xpack.core.security.action.role.GetRolesAction; import org.elasticsearch.xpack.core.security.action.role.PutRoleAction; +import org.elasticsearch.xpack.core.security.action.role.PutRoleRequestBuilderFactory; import org.elasticsearch.xpack.core.security.action.rolemapping.DeleteRoleMappingAction; import org.elasticsearch.xpack.core.security.action.rolemapping.GetRoleMappingsAction; import org.elasticsearch.xpack.core.security.action.rolemapping.PutRoleMappingAction; @@ -558,6 +559,8 @@ public class Security extends Plugin private final SetOnce transportReference = new SetOnce<>(); private final SetOnce scriptServiceReference = new SetOnce<>(); private final SetOnce operatorOnlyRegistry = new SetOnce<>(); + private final SetOnce putRoleRequestBuilderFactory = new SetOnce<>(); + private final SetOnce fileRolesStore = new SetOnce<>(); private final SetOnce operatorPrivilegesService = new SetOnce<>(); private final SetOnce reservedRoleMappingAction = new SetOnce<>(); private final SetOnce workflowService = new SetOnce<>(); @@ -802,13 +805,8 @@ Collection createComponents( final ReservedRolesStore reservedRolesStore = new ReservedRolesStore(Set.copyOf(INCLUDED_RESERVED_ROLES_SETTING.get(settings))); dlsBitsetCache.set(new DocumentSubsetBitsetCache(settings, threadPool)); final FieldPermissionsCache fieldPermissionsCache = new FieldPermissionsCache(settings); - final FileRolesStore fileRolesStore = new FileRolesStore( - settings, - environment, - resourceWatcherService, - getLicenseState(), - xContentRegistry - ); + + this.fileRolesStore.set(new FileRolesStore(settings, environment, resourceWatcherService, getLicenseState(), xContentRegistry)); final NativeRolesStore nativeRolesStore = new NativeRolesStore( settings, client, @@ -817,6 +815,10 @@ Collection createComponents( clusterService ); RoleDescriptor.setFieldPermissionsCache(fieldPermissionsCache); + // Need to set to default if it wasn't set by an extension + if (putRoleRequestBuilderFactory.get() == null) { + putRoleRequestBuilderFactory.set(new PutRoleRequestBuilderFactory.Default()); + } final Map, ActionListener>>> customRoleProviders = new LinkedHashMap<>(); for (SecurityExtension extension : securityExtensions) { @@ -868,7 +870,7 @@ Collection createComponents( final RoleProviders roleProviders = new RoleProviders( reservedRolesStore, - fileRolesStore, + fileRolesStore.get(), nativeRolesStore, customRoleProviders, getLicenseState() @@ -1423,7 +1425,7 @@ public List getRestHandlers( new RestPutUserAction(settings, getLicenseState()), new RestDeleteUserAction(settings, getLicenseState()), new RestGetRolesAction(settings, getLicenseState()), - new RestPutRoleAction(settings, getLicenseState()), + new RestPutRoleAction(settings, getLicenseState(), putRoleRequestBuilderFactory.get(), fileRolesStore.get()), new RestDeleteRoleAction(settings, getLicenseState()), new RestChangePasswordAction(settings, securityContext.get(), getLicenseState()), new RestSetEnabledAction(settings, getLicenseState()), @@ -2041,6 +2043,21 @@ public void loadExtensions(ExtensionLoader loader) { operatorOnlyRegistry.getClass().getCanonicalName() ); } + + List builderFactories = loader.loadExtensions(PutRoleRequestBuilderFactory.class); + if (builderFactories.size() > 1) { + throw new IllegalStateException(PutRoleRequestBuilderFactory.class + " may not have multiple implementations"); + } else if (builderFactories.size() == 1) { + PutRoleRequestBuilderFactory builderFactory = builderFactories.get(0); + this.putRoleRequestBuilderFactory.set(builderFactory); + logger.debug( + "Loaded implementation [{}] for interface [{}]", + builderFactory.getClass().getCanonicalName(), + PutRoleRequestBuilderFactory.class + ); + } else { + logger.debug("Will fall back on default implementation for interface [{}]", PutRoleRequestBuilderFactory.class); + } } private synchronized SharedGroupFactory getNettySharedGroupFactory(Settings settings) { diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/role/TransportPutRoleAction.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/role/TransportPutRoleAction.java index 4e4a3f9ef8d42..b85ef3f3c819f 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/role/TransportPutRoleAction.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/role/TransportPutRoleAction.java @@ -57,6 +57,7 @@ protected void doExecute(Task task, final PutRoleRequest request, final ActionLi } private Exception validateRequest(final PutRoleRequest request) { + // TODO we can remove this -- `execute()` already calls `request.validate()` before `doExecute()` ActionRequestValidationException validationException = request.validate(); if (validationException != null) { return validationException; diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/FileRolesStore.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/FileRolesStore.java index 70d086cc5a831..d7c8f11c467f2 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/FileRolesStore.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authz/store/FileRolesStore.java @@ -121,6 +121,11 @@ Set roleDescriptors(Set roleNames) { return descriptors; } + public boolean exists(String name) { + final Map localPermissions = permissions; + return localPermissions.containsKey(name); + } + public Map usageStats() { final Map localPermissions = permissions; Map usageStats = Maps.newMapWithExpectedSize(3); diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/operator/OperatorOnlyRegistry.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/operator/OperatorOnlyRegistry.java index b6fa0b080c059..c72c72e144b97 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/operator/OperatorOnlyRegistry.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/operator/OperatorOnlyRegistry.java @@ -26,7 +26,7 @@ public interface OperatorOnlyRegistry { * fully or partially restricted. A fully restricted REST API mandates that the implementation call restChannel.sendResponse(...) and * return a {@link OperatorPrivilegesViolation}. A partially restricted REST API mandates that the {@link RestRequest} is marked as * restricted so that the downstream handler can behave appropriately. For example, to restrict the REST response the implementation - * should call {@link RestRequest#markResponseRestricted(String)} so that the downstream handler can properly restrict the response + * should call {@link RestRequest#markPathRestricted(String)} so that the downstream handler can properly restrict the response * before returning to the client. Note - a partial restriction should return null. * @param restHandler The {@link RestHandler} to check for any restrictions * @param restRequest The {@link RestRequest} to check for any restrictions and mark any partially restricted REST API's diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleAction.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleAction.java index 7c3f9249d3c4d..4be786177af65 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleAction.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleAction.java @@ -18,7 +18,9 @@ import org.elasticsearch.rest.action.RestBuilderListener; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xpack.core.security.action.role.PutRoleRequestBuilder; +import org.elasticsearch.xpack.core.security.action.role.PutRoleRequestBuilderFactory; import org.elasticsearch.xpack.core.security.action.role.PutRoleResponse; +import org.elasticsearch.xpack.security.authz.store.FileRolesStore; import java.io.IOException; import java.util.List; @@ -29,11 +31,21 @@ /** * Rest endpoint to add a Role to the security index */ -@ServerlessScope(Scope.INTERNAL) +@ServerlessScope(Scope.PUBLIC) public class RestPutRoleAction extends NativeRoleBaseRestHandler { - public RestPutRoleAction(Settings settings, XPackLicenseState licenseState) { + private final PutRoleRequestBuilderFactory builderFactory; + private final FileRolesStore fileRolesStore; + + public RestPutRoleAction( + Settings settings, + XPackLicenseState licenseState, + PutRoleRequestBuilderFactory builderFactory, + FileRolesStore fileRolesStore + ) { super(settings, licenseState); + this.builderFactory = builderFactory; + this.fileRolesStore = fileRolesStore; } @Override @@ -51,11 +63,10 @@ public String getName() { @Override public RestChannelConsumer innerPrepareRequest(RestRequest request, NodeClient client) throws IOException { - PutRoleRequestBuilder requestBuilder = new PutRoleRequestBuilder(client).source( - request.param("name"), - request.requiredContent(), - request.getXContentType() - ).setRefreshPolicy(request.param("refresh")); + final boolean restrictRequest = request.hasParam(RestRequest.PATH_RESTRICTED); + final PutRoleRequestBuilder requestBuilder = builderFactory.create(client, restrictRequest, fileRolesStore::exists) + .source(request.param("name"), request.requiredContent(), request.getXContentType()) + .setRefreshPolicy(request.param("refresh")); return channel -> requestBuilder.execute(new RestBuilderListener<>(channel) { @Override public RestResponse buildResponse(PutRoleResponse putRoleResponse, XContentBuilder builder) throws Exception { diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java index 0c23cf2e65048..0f9dd06983792 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/store/FileRolesStoreTests.java @@ -707,6 +707,39 @@ public void testUsageStats() throws Exception { assertThat(usageStats.get("dls"), is(flsDlsEnabled)); } + public void testExists() throws Exception { + Path path = getDataPath("roles.yml"); + Path home = createTempDir(); + Path tmp = home.resolve("config/roles.yml"); + Files.createDirectories(tmp.getParent()); + try (OutputStream stream = Files.newOutputStream(tmp)) { + Files.copy(path, stream); + } + + Settings settings = Settings.builder().put("resource.reload.interval.high", "500ms").put("path.home", home).build(); + Environment env = TestEnvironment.newEnvironment(settings); + FileRolesStore store = new FileRolesStore( + settings, + env, + mock(ResourceWatcherService.class), + TestUtils.newTestLicenseState(), + xContentRegistry() + ); + Map roles = FileRolesStore.parseFile( + path, + logger, + Settings.builder().put(XPackSettings.DLS_FLS_ENABLED.getKey(), true).build(), + TestUtils.newTestLicenseState(), + xContentRegistry() + ); + assertThat(roles, notNullValue()); + assertThat(roles.size(), is(10)); + for (var role : roles.keySet()) { + assertThat(store.exists(role), is(true)); + } + assertThat(store.exists(randomValueOtherThanMany(roles::containsKey, () -> randomAlphaOfLength(20))), is(false)); + } + // test that we can read a role where field permissions are stored in 2.x format (fields:...) public void testBWCFieldPermissions() throws IOException { Path path = getDataPath("roles2xformat.yml"); diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleActionTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleActionTests.java index 01219a97e5905..ad20576e7f8af 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleActionTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/rest/action/role/RestPutRoleActionTests.java @@ -34,7 +34,7 @@ public void testFailureWhenNativeRolesDisabled() throws Exception { final Settings securityDisabledSettings = Settings.builder().put(NativeRolesStore.NATIVE_ROLES_ENABLED, false).build(); final XPackLicenseState licenseState = mock(XPackLicenseState.class); when(licenseState.getOperationMode()).thenReturn(License.OperationMode.BASIC); - final RestPutRoleAction action = new RestPutRoleAction(securityDisabledSettings, licenseState); + final RestPutRoleAction action = new RestPutRoleAction(securityDisabledSettings, licenseState, mock(), mock()); final FakeRestRequest request = new FakeRestRequest.Builder(NamedXContentRegistry.EMPTY) // .withParams(Map.of("name", "dice")) .withContent(new BytesArray("{ }"), XContentType.JSON) From 69b4edfdb7bbb16ca35d422bbe3dd3ec62ff1abb Mon Sep 17 00:00:00 2001 From: Luigi Dell'Aquila Date: Mon, 19 Feb 2024 11:22:49 +0100 Subject: [PATCH 26/45] Disable insensitive equals operator (#105611) --- .../src/main/resources/ignore-case.csv-spec | 26 +++++++++---------- .../esql/src/main/antlr/EsqlBaseParser.g4 | 2 +- .../xpack/esql/parser/EsqlBaseParser.interp | 2 +- .../xpack/esql/parser/EsqlBaseParser.java | 5 ++-- .../xpack/esql/analysis/AnalyzerTests.java | 1 + .../optimizer/PhysicalPlanOptimizerTests.java | 2 ++ 6 files changed, 20 insertions(+), 18 deletions(-) diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/ignore-case.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/ignore-case.csv-spec index 7d90171d1d305..2aecce791fcef 100644 --- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/ignore-case.csv-spec +++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/ignore-case.csv-spec @@ -1,5 +1,5 @@ -simpleFilter#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +simpleFilter-Ignore from employees | where first_name =~ "mary" | keep emp_no, first_name, last_name; emp_no:integer | first_name:keyword | last_name:keyword @@ -7,7 +7,7 @@ emp_no:integer | first_name:keyword | last_name:keyword ; -simpleFilterUpper#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +simpleFilterUpper-Ignore from employees | where first_name =~ "MARY" | keep emp_no, first_name, last_name; emp_no:integer | first_name:keyword | last_name:keyword @@ -15,14 +15,14 @@ emp_no:integer | first_name:keyword | last_name:keyword ; -simpleFilterPartial#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +simpleFilterPartial-Ignore from employees | where first_name =~ "mar" | keep emp_no, first_name, last_name; emp_no:integer | first_name:keyword | last_name:keyword ; -mixedConditionsAnd#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +mixedConditionsAnd-Ignore from employees | where first_name =~ "mary" AND emp_no == 10011 | keep emp_no, first_name, last_name; emp_no:integer | first_name:keyword | last_name:keyword @@ -30,7 +30,7 @@ emp_no:integer | first_name:keyword | last_name:keyword ; -mixedConditionsOr#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +mixedConditionsOr-Ignore from employees | where first_name =~ "mary" OR emp_no == 10001 | keep emp_no, first_name, last_name |sort emp_no; emp_no:integer | first_name:keyword | last_name:keyword @@ -39,7 +39,7 @@ emp_no:integer | first_name:keyword | last_name:keyword ; -evalEquals#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +evalEquals-Ignore from employees | where emp_no == 10001 | eval a = first_name =~ "georgi", b = first_name == "georgi", c = first_name =~ "GEORGI", d = first_name =~ "Geor", e = first_name =~ "GeoRgI" | keep emp_no, first_name, a, b, c, d, e; @@ -59,21 +59,21 @@ foobar ; -noWildcardSimple#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +noWildcardSimple-Ignore row name = "foobar" | where name =~ "FoOb*"; name:keyword ; -noWildcard#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +noWildcard-Ignore from employees | where first_name =~ "Georg*" | sort emp_no | keep emp_no, first_name; emp_no:integer | first_name:keyword ; -noWildcardSingle#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +noWildcardSingle-Ignore from employees | where first_name =~ "Georg?" | sort emp_no | keep emp_no, first_name; emp_no:integer | first_name:keyword @@ -90,7 +90,7 @@ emp_no:integer | first_name:keyword ; -expressionsRight#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +expressionsRight-Ignore from employees | where first_name =~ concat("Tzv","ETAN") | keep emp_no, first_name; emp_no:integer | first_name:keyword @@ -98,7 +98,7 @@ emp_no:integer | first_name:keyword ; -expressionsLeft#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +expressionsLeft-Ignore from employees | where concat(first_name, "_foo") =~ "TzvETAN_fOo" | keep emp_no, first_name; emp_no:integer | first_name:keyword @@ -117,14 +117,14 @@ emp_no:integer | first_name:keyword | last_name:keyword ; -multiValuesExcluded#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +multiValuesExcluded-Ignore row a = ["Foo", "Bar"] | where a =~ "foo"; a:keyword ; -multiValuesPushedDownExcluded#[skip:-8.12.99, reason:case insensitive operators implemented in v 8.13] +multiValuesPushedDownExcluded-Ignore from employees | where job_positions =~ "reporting analyst" | sort emp_no | keep emp_no, job_positions; warning:Line 1:24: evaluation of [job_positions =~ \"reporting analyst\"] failed, treating result as null. Only first 20 failures recorded. warning:Line 1:24: java.lang.IllegalArgumentException: single-value function encountered multi-value diff --git a/x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4 b/x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4 index c3391d71daf0b..b34b9bb103b83 100644 --- a/x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4 +++ b/x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4 @@ -231,7 +231,7 @@ string ; comparisonOperator - : EQ | CIEQ | NEQ | LT | LTE | GT | GTE + : EQ | NEQ | LT | LTE | GT | GTE ; explainCommand diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp index f4348a84eff69..a75449a305d3f 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.interp @@ -266,4 +266,4 @@ enrichWithClause atn: -[4, 1, 104, 507, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 2, 4, 7, 4, 2, 5, 7, 5, 2, 6, 7, 6, 2, 7, 7, 7, 2, 8, 7, 8, 2, 9, 7, 9, 2, 10, 7, 10, 2, 11, 7, 11, 2, 12, 7, 12, 2, 13, 7, 13, 2, 14, 7, 14, 2, 15, 7, 15, 2, 16, 7, 16, 2, 17, 7, 17, 2, 18, 7, 18, 2, 19, 7, 19, 2, 20, 7, 20, 2, 21, 7, 21, 2, 22, 7, 22, 2, 23, 7, 23, 2, 24, 7, 24, 2, 25, 7, 25, 2, 26, 7, 26, 2, 27, 7, 27, 2, 28, 7, 28, 2, 29, 7, 29, 2, 30, 7, 30, 2, 31, 7, 31, 2, 32, 7, 32, 2, 33, 7, 33, 2, 34, 7, 34, 2, 35, 7, 35, 2, 36, 7, 36, 2, 37, 7, 37, 2, 38, 7, 38, 2, 39, 7, 39, 2, 40, 7, 40, 2, 41, 7, 41, 2, 42, 7, 42, 2, 43, 7, 43, 2, 44, 7, 44, 2, 45, 7, 45, 2, 46, 7, 46, 2, 47, 7, 47, 2, 48, 7, 48, 2, 49, 7, 49, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5, 1, 110, 8, 1, 10, 1, 12, 1, 113, 9, 1, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 119, 8, 2, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 3, 3, 134, 8, 3, 1, 4, 1, 4, 1, 4, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 3, 5, 146, 8, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 5, 5, 153, 8, 5, 10, 5, 12, 5, 156, 9, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 3, 5, 163, 8, 5, 1, 5, 1, 5, 3, 5, 167, 8, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 5, 5, 175, 8, 5, 10, 5, 12, 5, 178, 9, 5, 1, 6, 1, 6, 3, 6, 182, 8, 6, 1, 6, 1, 6, 1, 6, 1, 6, 1, 6, 3, 6, 189, 8, 6, 1, 6, 1, 6, 1, 6, 3, 6, 194, 8, 6, 1, 7, 1, 7, 1, 7, 1, 7, 1, 7, 3, 7, 201, 8, 7, 1, 8, 1, 8, 1, 8, 1, 8, 3, 8, 207, 8, 8, 1, 8, 1, 8, 1, 8, 1, 8, 1, 8, 1, 8, 5, 8, 215, 8, 8, 10, 8, 12, 8, 218, 9, 8, 1, 9, 1, 9, 1, 9, 1, 9, 1, 9, 1, 9, 1, 9, 3, 9, 227, 8, 9, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 5, 10, 235, 8, 10, 10, 10, 12, 10, 238, 9, 10, 3, 10, 240, 8, 10, 1, 10, 1, 10, 1, 11, 1, 11, 1, 11, 1, 12, 1, 12, 1, 12, 5, 12, 250, 8, 12, 10, 12, 12, 12, 253, 9, 12, 1, 13, 1, 13, 1, 13, 1, 13, 1, 13, 3, 13, 260, 8, 13, 1, 14, 1, 14, 1, 14, 1, 14, 5, 14, 266, 8, 14, 10, 14, 12, 14, 269, 9, 14, 1, 14, 3, 14, 272, 8, 14, 1, 15, 1, 15, 3, 15, 276, 8, 15, 1, 16, 1, 16, 1, 16, 1, 16, 5, 16, 282, 8, 16, 10, 16, 12, 16, 285, 9, 16, 1, 17, 1, 17, 1, 17, 1, 17, 1, 18, 1, 18, 1, 18, 1, 19, 1, 19, 3, 19, 296, 8, 19, 1, 19, 1, 19, 3, 19, 300, 8, 19, 1, 20, 1, 20, 1, 20, 1, 20, 3, 20, 306, 8, 20, 1, 21, 1, 21, 1, 22, 1, 22, 1, 22, 5, 22, 313, 8, 22, 10, 22, 12, 22, 316, 9, 22, 1, 23, 1, 23, 1, 23, 5, 23, 321, 8, 23, 10, 23, 12, 23, 324, 9, 23, 1, 24, 1, 24, 1, 25, 1, 25, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 5, 26, 343, 8, 26, 10, 26, 12, 26, 346, 9, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 5, 26, 354, 8, 26, 10, 26, 12, 26, 357, 9, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 5, 26, 365, 8, 26, 10, 26, 12, 26, 368, 9, 26, 1, 26, 1, 26, 3, 26, 372, 8, 26, 1, 27, 1, 27, 1, 27, 1, 28, 1, 28, 1, 28, 1, 28, 5, 28, 381, 8, 28, 10, 28, 12, 28, 384, 9, 28, 1, 29, 1, 29, 3, 29, 388, 8, 29, 1, 29, 1, 29, 3, 29, 392, 8, 29, 1, 30, 1, 30, 1, 30, 1, 30, 5, 30, 398, 8, 30, 10, 30, 12, 30, 401, 9, 30, 1, 31, 1, 31, 1, 31, 1, 31, 5, 31, 407, 8, 31, 10, 31, 12, 31, 410, 9, 31, 1, 32, 1, 32, 1, 32, 1, 32, 5, 32, 416, 8, 32, 10, 32, 12, 32, 419, 9, 32, 1, 33, 1, 33, 1, 33, 1, 33, 1, 34, 1, 34, 1, 34, 1, 34, 3, 34, 429, 8, 34, 1, 35, 1, 35, 1, 35, 1, 35, 1, 36, 1, 36, 1, 36, 1, 37, 1, 37, 1, 37, 5, 37, 441, 8, 37, 10, 37, 12, 37, 444, 9, 37, 1, 38, 1, 38, 1, 38, 1, 38, 1, 39, 1, 39, 1, 40, 1, 40, 3, 40, 454, 8, 40, 1, 41, 3, 41, 457, 8, 41, 1, 41, 1, 41, 1, 42, 3, 42, 462, 8, 42, 1, 42, 1, 42, 1, 43, 1, 43, 1, 44, 1, 44, 1, 45, 1, 45, 1, 45, 1, 46, 1, 46, 1, 46, 1, 46, 1, 47, 1, 47, 1, 47, 1, 47, 3, 47, 481, 8, 47, 1, 48, 1, 48, 1, 48, 1, 48, 3, 48, 487, 8, 48, 1, 48, 1, 48, 1, 48, 1, 48, 5, 48, 493, 8, 48, 10, 48, 12, 48, 496, 9, 48, 3, 48, 498, 8, 48, 1, 49, 1, 49, 1, 49, 3, 49, 503, 8, 49, 1, 49, 1, 49, 1, 49, 0, 3, 2, 10, 16, 50, 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 0, 8, 1, 0, 58, 59, 1, 0, 60, 62, 2, 0, 66, 66, 71, 71, 1, 0, 65, 66, 2, 0, 31, 31, 34, 34, 1, 0, 37, 38, 2, 0, 36, 36, 50, 50, 1, 0, 51, 57, 533, 0, 100, 1, 0, 0, 0, 2, 103, 1, 0, 0, 0, 4, 118, 1, 0, 0, 0, 6, 133, 1, 0, 0, 0, 8, 135, 1, 0, 0, 0, 10, 166, 1, 0, 0, 0, 12, 193, 1, 0, 0, 0, 14, 200, 1, 0, 0, 0, 16, 206, 1, 0, 0, 0, 18, 226, 1, 0, 0, 0, 20, 228, 1, 0, 0, 0, 22, 243, 1, 0, 0, 0, 24, 246, 1, 0, 0, 0, 26, 259, 1, 0, 0, 0, 28, 261, 1, 0, 0, 0, 30, 275, 1, 0, 0, 0, 32, 277, 1, 0, 0, 0, 34, 286, 1, 0, 0, 0, 36, 290, 1, 0, 0, 0, 38, 293, 1, 0, 0, 0, 40, 301, 1, 0, 0, 0, 42, 307, 1, 0, 0, 0, 44, 309, 1, 0, 0, 0, 46, 317, 1, 0, 0, 0, 48, 325, 1, 0, 0, 0, 50, 327, 1, 0, 0, 0, 52, 371, 1, 0, 0, 0, 54, 373, 1, 0, 0, 0, 56, 376, 1, 0, 0, 0, 58, 385, 1, 0, 0, 0, 60, 393, 1, 0, 0, 0, 62, 402, 1, 0, 0, 0, 64, 411, 1, 0, 0, 0, 66, 420, 1, 0, 0, 0, 68, 424, 1, 0, 0, 0, 70, 430, 1, 0, 0, 0, 72, 434, 1, 0, 0, 0, 74, 437, 1, 0, 0, 0, 76, 445, 1, 0, 0, 0, 78, 449, 1, 0, 0, 0, 80, 453, 1, 0, 0, 0, 82, 456, 1, 0, 0, 0, 84, 461, 1, 0, 0, 0, 86, 465, 1, 0, 0, 0, 88, 467, 1, 0, 0, 0, 90, 469, 1, 0, 0, 0, 92, 472, 1, 0, 0, 0, 94, 480, 1, 0, 0, 0, 96, 482, 1, 0, 0, 0, 98, 502, 1, 0, 0, 0, 100, 101, 3, 2, 1, 0, 101, 102, 5, 0, 0, 1, 102, 1, 1, 0, 0, 0, 103, 104, 6, 1, -1, 0, 104, 105, 3, 4, 2, 0, 105, 111, 1, 0, 0, 0, 106, 107, 10, 1, 0, 0, 107, 108, 5, 25, 0, 0, 108, 110, 3, 6, 3, 0, 109, 106, 1, 0, 0, 0, 110, 113, 1, 0, 0, 0, 111, 109, 1, 0, 0, 0, 111, 112, 1, 0, 0, 0, 112, 3, 1, 0, 0, 0, 113, 111, 1, 0, 0, 0, 114, 119, 3, 90, 45, 0, 115, 119, 3, 28, 14, 0, 116, 119, 3, 22, 11, 0, 117, 119, 3, 94, 47, 0, 118, 114, 1, 0, 0, 0, 118, 115, 1, 0, 0, 0, 118, 116, 1, 0, 0, 0, 118, 117, 1, 0, 0, 0, 119, 5, 1, 0, 0, 0, 120, 134, 3, 36, 18, 0, 121, 134, 3, 40, 20, 0, 122, 134, 3, 54, 27, 0, 123, 134, 3, 60, 30, 0, 124, 134, 3, 56, 28, 0, 125, 134, 3, 38, 19, 0, 126, 134, 3, 8, 4, 0, 127, 134, 3, 62, 31, 0, 128, 134, 3, 64, 32, 0, 129, 134, 3, 68, 34, 0, 130, 134, 3, 70, 35, 0, 131, 134, 3, 96, 48, 0, 132, 134, 3, 72, 36, 0, 133, 120, 1, 0, 0, 0, 133, 121, 1, 0, 0, 0, 133, 122, 1, 0, 0, 0, 133, 123, 1, 0, 0, 0, 133, 124, 1, 0, 0, 0, 133, 125, 1, 0, 0, 0, 133, 126, 1, 0, 0, 0, 133, 127, 1, 0, 0, 0, 133, 128, 1, 0, 0, 0, 133, 129, 1, 0, 0, 0, 133, 130, 1, 0, 0, 0, 133, 131, 1, 0, 0, 0, 133, 132, 1, 0, 0, 0, 134, 7, 1, 0, 0, 0, 135, 136, 5, 17, 0, 0, 136, 137, 3, 10, 5, 0, 137, 9, 1, 0, 0, 0, 138, 139, 6, 5, -1, 0, 139, 140, 5, 43, 0, 0, 140, 167, 3, 10, 5, 7, 141, 167, 3, 14, 7, 0, 142, 167, 3, 12, 6, 0, 143, 145, 3, 14, 7, 0, 144, 146, 5, 43, 0, 0, 145, 144, 1, 0, 0, 0, 145, 146, 1, 0, 0, 0, 146, 147, 1, 0, 0, 0, 147, 148, 5, 40, 0, 0, 148, 149, 5, 39, 0, 0, 149, 154, 3, 14, 7, 0, 150, 151, 5, 33, 0, 0, 151, 153, 3, 14, 7, 0, 152, 150, 1, 0, 0, 0, 153, 156, 1, 0, 0, 0, 154, 152, 1, 0, 0, 0, 154, 155, 1, 0, 0, 0, 155, 157, 1, 0, 0, 0, 156, 154, 1, 0, 0, 0, 157, 158, 5, 49, 0, 0, 158, 167, 1, 0, 0, 0, 159, 160, 3, 14, 7, 0, 160, 162, 5, 41, 0, 0, 161, 163, 5, 43, 0, 0, 162, 161, 1, 0, 0, 0, 162, 163, 1, 0, 0, 0, 163, 164, 1, 0, 0, 0, 164, 165, 5, 44, 0, 0, 165, 167, 1, 0, 0, 0, 166, 138, 1, 0, 0, 0, 166, 141, 1, 0, 0, 0, 166, 142, 1, 0, 0, 0, 166, 143, 1, 0, 0, 0, 166, 159, 1, 0, 0, 0, 167, 176, 1, 0, 0, 0, 168, 169, 10, 4, 0, 0, 169, 170, 5, 30, 0, 0, 170, 175, 3, 10, 5, 5, 171, 172, 10, 3, 0, 0, 172, 173, 5, 46, 0, 0, 173, 175, 3, 10, 5, 4, 174, 168, 1, 0, 0, 0, 174, 171, 1, 0, 0, 0, 175, 178, 1, 0, 0, 0, 176, 174, 1, 0, 0, 0, 176, 177, 1, 0, 0, 0, 177, 11, 1, 0, 0, 0, 178, 176, 1, 0, 0, 0, 179, 181, 3, 14, 7, 0, 180, 182, 5, 43, 0, 0, 181, 180, 1, 0, 0, 0, 181, 182, 1, 0, 0, 0, 182, 183, 1, 0, 0, 0, 183, 184, 5, 42, 0, 0, 184, 185, 3, 86, 43, 0, 185, 194, 1, 0, 0, 0, 186, 188, 3, 14, 7, 0, 187, 189, 5, 43, 0, 0, 188, 187, 1, 0, 0, 0, 188, 189, 1, 0, 0, 0, 189, 190, 1, 0, 0, 0, 190, 191, 5, 48, 0, 0, 191, 192, 3, 86, 43, 0, 192, 194, 1, 0, 0, 0, 193, 179, 1, 0, 0, 0, 193, 186, 1, 0, 0, 0, 194, 13, 1, 0, 0, 0, 195, 201, 3, 16, 8, 0, 196, 197, 3, 16, 8, 0, 197, 198, 3, 88, 44, 0, 198, 199, 3, 16, 8, 0, 199, 201, 1, 0, 0, 0, 200, 195, 1, 0, 0, 0, 200, 196, 1, 0, 0, 0, 201, 15, 1, 0, 0, 0, 202, 203, 6, 8, -1, 0, 203, 207, 3, 18, 9, 0, 204, 205, 7, 0, 0, 0, 205, 207, 3, 16, 8, 3, 206, 202, 1, 0, 0, 0, 206, 204, 1, 0, 0, 0, 207, 216, 1, 0, 0, 0, 208, 209, 10, 2, 0, 0, 209, 210, 7, 1, 0, 0, 210, 215, 3, 16, 8, 3, 211, 212, 10, 1, 0, 0, 212, 213, 7, 0, 0, 0, 213, 215, 3, 16, 8, 2, 214, 208, 1, 0, 0, 0, 214, 211, 1, 0, 0, 0, 215, 218, 1, 0, 0, 0, 216, 214, 1, 0, 0, 0, 216, 217, 1, 0, 0, 0, 217, 17, 1, 0, 0, 0, 218, 216, 1, 0, 0, 0, 219, 227, 3, 52, 26, 0, 220, 227, 3, 44, 22, 0, 221, 227, 3, 20, 10, 0, 222, 223, 5, 39, 0, 0, 223, 224, 3, 10, 5, 0, 224, 225, 5, 49, 0, 0, 225, 227, 1, 0, 0, 0, 226, 219, 1, 0, 0, 0, 226, 220, 1, 0, 0, 0, 226, 221, 1, 0, 0, 0, 226, 222, 1, 0, 0, 0, 227, 19, 1, 0, 0, 0, 228, 229, 3, 48, 24, 0, 229, 239, 5, 39, 0, 0, 230, 240, 5, 60, 0, 0, 231, 236, 3, 10, 5, 0, 232, 233, 5, 33, 0, 0, 233, 235, 3, 10, 5, 0, 234, 232, 1, 0, 0, 0, 235, 238, 1, 0, 0, 0, 236, 234, 1, 0, 0, 0, 236, 237, 1, 0, 0, 0, 237, 240, 1, 0, 0, 0, 238, 236, 1, 0, 0, 0, 239, 230, 1, 0, 0, 0, 239, 231, 1, 0, 0, 0, 239, 240, 1, 0, 0, 0, 240, 241, 1, 0, 0, 0, 241, 242, 5, 49, 0, 0, 242, 21, 1, 0, 0, 0, 243, 244, 5, 13, 0, 0, 244, 245, 3, 24, 12, 0, 245, 23, 1, 0, 0, 0, 246, 251, 3, 26, 13, 0, 247, 248, 5, 33, 0, 0, 248, 250, 3, 26, 13, 0, 249, 247, 1, 0, 0, 0, 250, 253, 1, 0, 0, 0, 251, 249, 1, 0, 0, 0, 251, 252, 1, 0, 0, 0, 252, 25, 1, 0, 0, 0, 253, 251, 1, 0, 0, 0, 254, 260, 3, 10, 5, 0, 255, 256, 3, 44, 22, 0, 256, 257, 5, 32, 0, 0, 257, 258, 3, 10, 5, 0, 258, 260, 1, 0, 0, 0, 259, 254, 1, 0, 0, 0, 259, 255, 1, 0, 0, 0, 260, 27, 1, 0, 0, 0, 261, 262, 5, 6, 0, 0, 262, 267, 3, 42, 21, 0, 263, 264, 5, 33, 0, 0, 264, 266, 3, 42, 21, 0, 265, 263, 1, 0, 0, 0, 266, 269, 1, 0, 0, 0, 267, 265, 1, 0, 0, 0, 267, 268, 1, 0, 0, 0, 268, 271, 1, 0, 0, 0, 269, 267, 1, 0, 0, 0, 270, 272, 3, 30, 15, 0, 271, 270, 1, 0, 0, 0, 271, 272, 1, 0, 0, 0, 272, 29, 1, 0, 0, 0, 273, 276, 3, 32, 16, 0, 274, 276, 3, 34, 17, 0, 275, 273, 1, 0, 0, 0, 275, 274, 1, 0, 0, 0, 276, 31, 1, 0, 0, 0, 277, 278, 5, 70, 0, 0, 278, 283, 3, 42, 21, 0, 279, 280, 5, 33, 0, 0, 280, 282, 3, 42, 21, 0, 281, 279, 1, 0, 0, 0, 282, 285, 1, 0, 0, 0, 283, 281, 1, 0, 0, 0, 283, 284, 1, 0, 0, 0, 284, 33, 1, 0, 0, 0, 285, 283, 1, 0, 0, 0, 286, 287, 5, 63, 0, 0, 287, 288, 3, 32, 16, 0, 288, 289, 5, 64, 0, 0, 289, 35, 1, 0, 0, 0, 290, 291, 5, 4, 0, 0, 291, 292, 3, 24, 12, 0, 292, 37, 1, 0, 0, 0, 293, 295, 5, 16, 0, 0, 294, 296, 3, 24, 12, 0, 295, 294, 1, 0, 0, 0, 295, 296, 1, 0, 0, 0, 296, 299, 1, 0, 0, 0, 297, 298, 5, 29, 0, 0, 298, 300, 3, 24, 12, 0, 299, 297, 1, 0, 0, 0, 299, 300, 1, 0, 0, 0, 300, 39, 1, 0, 0, 0, 301, 302, 5, 8, 0, 0, 302, 305, 3, 24, 12, 0, 303, 304, 5, 29, 0, 0, 304, 306, 3, 24, 12, 0, 305, 303, 1, 0, 0, 0, 305, 306, 1, 0, 0, 0, 306, 41, 1, 0, 0, 0, 307, 308, 7, 2, 0, 0, 308, 43, 1, 0, 0, 0, 309, 314, 3, 48, 24, 0, 310, 311, 5, 35, 0, 0, 311, 313, 3, 48, 24, 0, 312, 310, 1, 0, 0, 0, 313, 316, 1, 0, 0, 0, 314, 312, 1, 0, 0, 0, 314, 315, 1, 0, 0, 0, 315, 45, 1, 0, 0, 0, 316, 314, 1, 0, 0, 0, 317, 322, 3, 50, 25, 0, 318, 319, 5, 35, 0, 0, 319, 321, 3, 50, 25, 0, 320, 318, 1, 0, 0, 0, 321, 324, 1, 0, 0, 0, 322, 320, 1, 0, 0, 0, 322, 323, 1, 0, 0, 0, 323, 47, 1, 0, 0, 0, 324, 322, 1, 0, 0, 0, 325, 326, 7, 3, 0, 0, 326, 49, 1, 0, 0, 0, 327, 328, 5, 75, 0, 0, 328, 51, 1, 0, 0, 0, 329, 372, 5, 44, 0, 0, 330, 331, 3, 84, 42, 0, 331, 332, 5, 65, 0, 0, 332, 372, 1, 0, 0, 0, 333, 372, 3, 82, 41, 0, 334, 372, 3, 84, 42, 0, 335, 372, 3, 78, 39, 0, 336, 372, 5, 47, 0, 0, 337, 372, 3, 86, 43, 0, 338, 339, 5, 63, 0, 0, 339, 344, 3, 80, 40, 0, 340, 341, 5, 33, 0, 0, 341, 343, 3, 80, 40, 0, 342, 340, 1, 0, 0, 0, 343, 346, 1, 0, 0, 0, 344, 342, 1, 0, 0, 0, 344, 345, 1, 0, 0, 0, 345, 347, 1, 0, 0, 0, 346, 344, 1, 0, 0, 0, 347, 348, 5, 64, 0, 0, 348, 372, 1, 0, 0, 0, 349, 350, 5, 63, 0, 0, 350, 355, 3, 78, 39, 0, 351, 352, 5, 33, 0, 0, 352, 354, 3, 78, 39, 0, 353, 351, 1, 0, 0, 0, 354, 357, 1, 0, 0, 0, 355, 353, 1, 0, 0, 0, 355, 356, 1, 0, 0, 0, 356, 358, 1, 0, 0, 0, 357, 355, 1, 0, 0, 0, 358, 359, 5, 64, 0, 0, 359, 372, 1, 0, 0, 0, 360, 361, 5, 63, 0, 0, 361, 366, 3, 86, 43, 0, 362, 363, 5, 33, 0, 0, 363, 365, 3, 86, 43, 0, 364, 362, 1, 0, 0, 0, 365, 368, 1, 0, 0, 0, 366, 364, 1, 0, 0, 0, 366, 367, 1, 0, 0, 0, 367, 369, 1, 0, 0, 0, 368, 366, 1, 0, 0, 0, 369, 370, 5, 64, 0, 0, 370, 372, 1, 0, 0, 0, 371, 329, 1, 0, 0, 0, 371, 330, 1, 0, 0, 0, 371, 333, 1, 0, 0, 0, 371, 334, 1, 0, 0, 0, 371, 335, 1, 0, 0, 0, 371, 336, 1, 0, 0, 0, 371, 337, 1, 0, 0, 0, 371, 338, 1, 0, 0, 0, 371, 349, 1, 0, 0, 0, 371, 360, 1, 0, 0, 0, 372, 53, 1, 0, 0, 0, 373, 374, 5, 10, 0, 0, 374, 375, 5, 27, 0, 0, 375, 55, 1, 0, 0, 0, 376, 377, 5, 15, 0, 0, 377, 382, 3, 58, 29, 0, 378, 379, 5, 33, 0, 0, 379, 381, 3, 58, 29, 0, 380, 378, 1, 0, 0, 0, 381, 384, 1, 0, 0, 0, 382, 380, 1, 0, 0, 0, 382, 383, 1, 0, 0, 0, 383, 57, 1, 0, 0, 0, 384, 382, 1, 0, 0, 0, 385, 387, 3, 10, 5, 0, 386, 388, 7, 4, 0, 0, 387, 386, 1, 0, 0, 0, 387, 388, 1, 0, 0, 0, 388, 391, 1, 0, 0, 0, 389, 390, 5, 45, 0, 0, 390, 392, 7, 5, 0, 0, 391, 389, 1, 0, 0, 0, 391, 392, 1, 0, 0, 0, 392, 59, 1, 0, 0, 0, 393, 394, 5, 9, 0, 0, 394, 399, 3, 46, 23, 0, 395, 396, 5, 33, 0, 0, 396, 398, 3, 46, 23, 0, 397, 395, 1, 0, 0, 0, 398, 401, 1, 0, 0, 0, 399, 397, 1, 0, 0, 0, 399, 400, 1, 0, 0, 0, 400, 61, 1, 0, 0, 0, 401, 399, 1, 0, 0, 0, 402, 403, 5, 2, 0, 0, 403, 408, 3, 46, 23, 0, 404, 405, 5, 33, 0, 0, 405, 407, 3, 46, 23, 0, 406, 404, 1, 0, 0, 0, 407, 410, 1, 0, 0, 0, 408, 406, 1, 0, 0, 0, 408, 409, 1, 0, 0, 0, 409, 63, 1, 0, 0, 0, 410, 408, 1, 0, 0, 0, 411, 412, 5, 12, 0, 0, 412, 417, 3, 66, 33, 0, 413, 414, 5, 33, 0, 0, 414, 416, 3, 66, 33, 0, 415, 413, 1, 0, 0, 0, 416, 419, 1, 0, 0, 0, 417, 415, 1, 0, 0, 0, 417, 418, 1, 0, 0, 0, 418, 65, 1, 0, 0, 0, 419, 417, 1, 0, 0, 0, 420, 421, 3, 46, 23, 0, 421, 422, 5, 79, 0, 0, 422, 423, 3, 46, 23, 0, 423, 67, 1, 0, 0, 0, 424, 425, 5, 1, 0, 0, 425, 426, 3, 18, 9, 0, 426, 428, 3, 86, 43, 0, 427, 429, 3, 74, 37, 0, 428, 427, 1, 0, 0, 0, 428, 429, 1, 0, 0, 0, 429, 69, 1, 0, 0, 0, 430, 431, 5, 7, 0, 0, 431, 432, 3, 18, 9, 0, 432, 433, 3, 86, 43, 0, 433, 71, 1, 0, 0, 0, 434, 435, 5, 11, 0, 0, 435, 436, 3, 44, 22, 0, 436, 73, 1, 0, 0, 0, 437, 442, 3, 76, 38, 0, 438, 439, 5, 33, 0, 0, 439, 441, 3, 76, 38, 0, 440, 438, 1, 0, 0, 0, 441, 444, 1, 0, 0, 0, 442, 440, 1, 0, 0, 0, 442, 443, 1, 0, 0, 0, 443, 75, 1, 0, 0, 0, 444, 442, 1, 0, 0, 0, 445, 446, 3, 48, 24, 0, 446, 447, 5, 32, 0, 0, 447, 448, 3, 52, 26, 0, 448, 77, 1, 0, 0, 0, 449, 450, 7, 6, 0, 0, 450, 79, 1, 0, 0, 0, 451, 454, 3, 82, 41, 0, 452, 454, 3, 84, 42, 0, 453, 451, 1, 0, 0, 0, 453, 452, 1, 0, 0, 0, 454, 81, 1, 0, 0, 0, 455, 457, 7, 0, 0, 0, 456, 455, 1, 0, 0, 0, 456, 457, 1, 0, 0, 0, 457, 458, 1, 0, 0, 0, 458, 459, 5, 28, 0, 0, 459, 83, 1, 0, 0, 0, 460, 462, 7, 0, 0, 0, 461, 460, 1, 0, 0, 0, 461, 462, 1, 0, 0, 0, 462, 463, 1, 0, 0, 0, 463, 464, 5, 27, 0, 0, 464, 85, 1, 0, 0, 0, 465, 466, 5, 26, 0, 0, 466, 87, 1, 0, 0, 0, 467, 468, 7, 7, 0, 0, 468, 89, 1, 0, 0, 0, 469, 470, 5, 5, 0, 0, 470, 471, 3, 92, 46, 0, 471, 91, 1, 0, 0, 0, 472, 473, 5, 63, 0, 0, 473, 474, 3, 2, 1, 0, 474, 475, 5, 64, 0, 0, 475, 93, 1, 0, 0, 0, 476, 477, 5, 14, 0, 0, 477, 481, 5, 95, 0, 0, 478, 479, 5, 14, 0, 0, 479, 481, 5, 96, 0, 0, 480, 476, 1, 0, 0, 0, 480, 478, 1, 0, 0, 0, 481, 95, 1, 0, 0, 0, 482, 483, 5, 3, 0, 0, 483, 486, 5, 85, 0, 0, 484, 485, 5, 83, 0, 0, 485, 487, 3, 46, 23, 0, 486, 484, 1, 0, 0, 0, 486, 487, 1, 0, 0, 0, 487, 497, 1, 0, 0, 0, 488, 489, 5, 84, 0, 0, 489, 494, 3, 98, 49, 0, 490, 491, 5, 33, 0, 0, 491, 493, 3, 98, 49, 0, 492, 490, 1, 0, 0, 0, 493, 496, 1, 0, 0, 0, 494, 492, 1, 0, 0, 0, 494, 495, 1, 0, 0, 0, 495, 498, 1, 0, 0, 0, 496, 494, 1, 0, 0, 0, 497, 488, 1, 0, 0, 0, 497, 498, 1, 0, 0, 0, 498, 97, 1, 0, 0, 0, 499, 500, 3, 46, 23, 0, 500, 501, 5, 32, 0, 0, 501, 503, 1, 0, 0, 0, 502, 499, 1, 0, 0, 0, 502, 503, 1, 0, 0, 0, 503, 504, 1, 0, 0, 0, 504, 505, 3, 46, 23, 0, 505, 99, 1, 0, 0, 0, 50, 111, 118, 133, 145, 154, 162, 166, 174, 176, 181, 188, 193, 200, 206, 214, 216, 226, 236, 239, 251, 259, 267, 271, 275, 283, 295, 299, 305, 314, 322, 344, 355, 366, 371, 382, 387, 391, 399, 408, 417, 428, 442, 453, 456, 461, 480, 486, 494, 497, 502] \ No newline at end of file +[4, 1, 104, 507, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 2, 4, 7, 4, 2, 5, 7, 5, 2, 6, 7, 6, 2, 7, 7, 7, 2, 8, 7, 8, 2, 9, 7, 9, 2, 10, 7, 10, 2, 11, 7, 11, 2, 12, 7, 12, 2, 13, 7, 13, 2, 14, 7, 14, 2, 15, 7, 15, 2, 16, 7, 16, 2, 17, 7, 17, 2, 18, 7, 18, 2, 19, 7, 19, 2, 20, 7, 20, 2, 21, 7, 21, 2, 22, 7, 22, 2, 23, 7, 23, 2, 24, 7, 24, 2, 25, 7, 25, 2, 26, 7, 26, 2, 27, 7, 27, 2, 28, 7, 28, 2, 29, 7, 29, 2, 30, 7, 30, 2, 31, 7, 31, 2, 32, 7, 32, 2, 33, 7, 33, 2, 34, 7, 34, 2, 35, 7, 35, 2, 36, 7, 36, 2, 37, 7, 37, 2, 38, 7, 38, 2, 39, 7, 39, 2, 40, 7, 40, 2, 41, 7, 41, 2, 42, 7, 42, 2, 43, 7, 43, 2, 44, 7, 44, 2, 45, 7, 45, 2, 46, 7, 46, 2, 47, 7, 47, 2, 48, 7, 48, 2, 49, 7, 49, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5, 1, 110, 8, 1, 10, 1, 12, 1, 113, 9, 1, 1, 2, 1, 2, 1, 2, 1, 2, 3, 2, 119, 8, 2, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 1, 3, 3, 3, 134, 8, 3, 1, 4, 1, 4, 1, 4, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 3, 5, 146, 8, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 5, 5, 153, 8, 5, 10, 5, 12, 5, 156, 9, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 3, 5, 163, 8, 5, 1, 5, 1, 5, 3, 5, 167, 8, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 1, 5, 5, 5, 175, 8, 5, 10, 5, 12, 5, 178, 9, 5, 1, 6, 1, 6, 3, 6, 182, 8, 6, 1, 6, 1, 6, 1, 6, 1, 6, 1, 6, 3, 6, 189, 8, 6, 1, 6, 1, 6, 1, 6, 3, 6, 194, 8, 6, 1, 7, 1, 7, 1, 7, 1, 7, 1, 7, 3, 7, 201, 8, 7, 1, 8, 1, 8, 1, 8, 1, 8, 3, 8, 207, 8, 8, 1, 8, 1, 8, 1, 8, 1, 8, 1, 8, 1, 8, 5, 8, 215, 8, 8, 10, 8, 12, 8, 218, 9, 8, 1, 9, 1, 9, 1, 9, 1, 9, 1, 9, 1, 9, 1, 9, 3, 9, 227, 8, 9, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 1, 10, 5, 10, 235, 8, 10, 10, 10, 12, 10, 238, 9, 10, 3, 10, 240, 8, 10, 1, 10, 1, 10, 1, 11, 1, 11, 1, 11, 1, 12, 1, 12, 1, 12, 5, 12, 250, 8, 12, 10, 12, 12, 12, 253, 9, 12, 1, 13, 1, 13, 1, 13, 1, 13, 1, 13, 3, 13, 260, 8, 13, 1, 14, 1, 14, 1, 14, 1, 14, 5, 14, 266, 8, 14, 10, 14, 12, 14, 269, 9, 14, 1, 14, 3, 14, 272, 8, 14, 1, 15, 1, 15, 3, 15, 276, 8, 15, 1, 16, 1, 16, 1, 16, 1, 16, 5, 16, 282, 8, 16, 10, 16, 12, 16, 285, 9, 16, 1, 17, 1, 17, 1, 17, 1, 17, 1, 18, 1, 18, 1, 18, 1, 19, 1, 19, 3, 19, 296, 8, 19, 1, 19, 1, 19, 3, 19, 300, 8, 19, 1, 20, 1, 20, 1, 20, 1, 20, 3, 20, 306, 8, 20, 1, 21, 1, 21, 1, 22, 1, 22, 1, 22, 5, 22, 313, 8, 22, 10, 22, 12, 22, 316, 9, 22, 1, 23, 1, 23, 1, 23, 5, 23, 321, 8, 23, 10, 23, 12, 23, 324, 9, 23, 1, 24, 1, 24, 1, 25, 1, 25, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 5, 26, 343, 8, 26, 10, 26, 12, 26, 346, 9, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 5, 26, 354, 8, 26, 10, 26, 12, 26, 357, 9, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 1, 26, 5, 26, 365, 8, 26, 10, 26, 12, 26, 368, 9, 26, 1, 26, 1, 26, 3, 26, 372, 8, 26, 1, 27, 1, 27, 1, 27, 1, 28, 1, 28, 1, 28, 1, 28, 5, 28, 381, 8, 28, 10, 28, 12, 28, 384, 9, 28, 1, 29, 1, 29, 3, 29, 388, 8, 29, 1, 29, 1, 29, 3, 29, 392, 8, 29, 1, 30, 1, 30, 1, 30, 1, 30, 5, 30, 398, 8, 30, 10, 30, 12, 30, 401, 9, 30, 1, 31, 1, 31, 1, 31, 1, 31, 5, 31, 407, 8, 31, 10, 31, 12, 31, 410, 9, 31, 1, 32, 1, 32, 1, 32, 1, 32, 5, 32, 416, 8, 32, 10, 32, 12, 32, 419, 9, 32, 1, 33, 1, 33, 1, 33, 1, 33, 1, 34, 1, 34, 1, 34, 1, 34, 3, 34, 429, 8, 34, 1, 35, 1, 35, 1, 35, 1, 35, 1, 36, 1, 36, 1, 36, 1, 37, 1, 37, 1, 37, 5, 37, 441, 8, 37, 10, 37, 12, 37, 444, 9, 37, 1, 38, 1, 38, 1, 38, 1, 38, 1, 39, 1, 39, 1, 40, 1, 40, 3, 40, 454, 8, 40, 1, 41, 3, 41, 457, 8, 41, 1, 41, 1, 41, 1, 42, 3, 42, 462, 8, 42, 1, 42, 1, 42, 1, 43, 1, 43, 1, 44, 1, 44, 1, 45, 1, 45, 1, 45, 1, 46, 1, 46, 1, 46, 1, 46, 1, 47, 1, 47, 1, 47, 1, 47, 3, 47, 481, 8, 47, 1, 48, 1, 48, 1, 48, 1, 48, 3, 48, 487, 8, 48, 1, 48, 1, 48, 1, 48, 1, 48, 5, 48, 493, 8, 48, 10, 48, 12, 48, 496, 9, 48, 3, 48, 498, 8, 48, 1, 49, 1, 49, 1, 49, 3, 49, 503, 8, 49, 1, 49, 1, 49, 1, 49, 0, 3, 2, 10, 16, 50, 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 0, 8, 1, 0, 58, 59, 1, 0, 60, 62, 2, 0, 66, 66, 71, 71, 1, 0, 65, 66, 2, 0, 31, 31, 34, 34, 1, 0, 37, 38, 2, 0, 36, 36, 50, 50, 2, 0, 51, 51, 53, 57, 533, 0, 100, 1, 0, 0, 0, 2, 103, 1, 0, 0, 0, 4, 118, 1, 0, 0, 0, 6, 133, 1, 0, 0, 0, 8, 135, 1, 0, 0, 0, 10, 166, 1, 0, 0, 0, 12, 193, 1, 0, 0, 0, 14, 200, 1, 0, 0, 0, 16, 206, 1, 0, 0, 0, 18, 226, 1, 0, 0, 0, 20, 228, 1, 0, 0, 0, 22, 243, 1, 0, 0, 0, 24, 246, 1, 0, 0, 0, 26, 259, 1, 0, 0, 0, 28, 261, 1, 0, 0, 0, 30, 275, 1, 0, 0, 0, 32, 277, 1, 0, 0, 0, 34, 286, 1, 0, 0, 0, 36, 290, 1, 0, 0, 0, 38, 293, 1, 0, 0, 0, 40, 301, 1, 0, 0, 0, 42, 307, 1, 0, 0, 0, 44, 309, 1, 0, 0, 0, 46, 317, 1, 0, 0, 0, 48, 325, 1, 0, 0, 0, 50, 327, 1, 0, 0, 0, 52, 371, 1, 0, 0, 0, 54, 373, 1, 0, 0, 0, 56, 376, 1, 0, 0, 0, 58, 385, 1, 0, 0, 0, 60, 393, 1, 0, 0, 0, 62, 402, 1, 0, 0, 0, 64, 411, 1, 0, 0, 0, 66, 420, 1, 0, 0, 0, 68, 424, 1, 0, 0, 0, 70, 430, 1, 0, 0, 0, 72, 434, 1, 0, 0, 0, 74, 437, 1, 0, 0, 0, 76, 445, 1, 0, 0, 0, 78, 449, 1, 0, 0, 0, 80, 453, 1, 0, 0, 0, 82, 456, 1, 0, 0, 0, 84, 461, 1, 0, 0, 0, 86, 465, 1, 0, 0, 0, 88, 467, 1, 0, 0, 0, 90, 469, 1, 0, 0, 0, 92, 472, 1, 0, 0, 0, 94, 480, 1, 0, 0, 0, 96, 482, 1, 0, 0, 0, 98, 502, 1, 0, 0, 0, 100, 101, 3, 2, 1, 0, 101, 102, 5, 0, 0, 1, 102, 1, 1, 0, 0, 0, 103, 104, 6, 1, -1, 0, 104, 105, 3, 4, 2, 0, 105, 111, 1, 0, 0, 0, 106, 107, 10, 1, 0, 0, 107, 108, 5, 25, 0, 0, 108, 110, 3, 6, 3, 0, 109, 106, 1, 0, 0, 0, 110, 113, 1, 0, 0, 0, 111, 109, 1, 0, 0, 0, 111, 112, 1, 0, 0, 0, 112, 3, 1, 0, 0, 0, 113, 111, 1, 0, 0, 0, 114, 119, 3, 90, 45, 0, 115, 119, 3, 28, 14, 0, 116, 119, 3, 22, 11, 0, 117, 119, 3, 94, 47, 0, 118, 114, 1, 0, 0, 0, 118, 115, 1, 0, 0, 0, 118, 116, 1, 0, 0, 0, 118, 117, 1, 0, 0, 0, 119, 5, 1, 0, 0, 0, 120, 134, 3, 36, 18, 0, 121, 134, 3, 40, 20, 0, 122, 134, 3, 54, 27, 0, 123, 134, 3, 60, 30, 0, 124, 134, 3, 56, 28, 0, 125, 134, 3, 38, 19, 0, 126, 134, 3, 8, 4, 0, 127, 134, 3, 62, 31, 0, 128, 134, 3, 64, 32, 0, 129, 134, 3, 68, 34, 0, 130, 134, 3, 70, 35, 0, 131, 134, 3, 96, 48, 0, 132, 134, 3, 72, 36, 0, 133, 120, 1, 0, 0, 0, 133, 121, 1, 0, 0, 0, 133, 122, 1, 0, 0, 0, 133, 123, 1, 0, 0, 0, 133, 124, 1, 0, 0, 0, 133, 125, 1, 0, 0, 0, 133, 126, 1, 0, 0, 0, 133, 127, 1, 0, 0, 0, 133, 128, 1, 0, 0, 0, 133, 129, 1, 0, 0, 0, 133, 130, 1, 0, 0, 0, 133, 131, 1, 0, 0, 0, 133, 132, 1, 0, 0, 0, 134, 7, 1, 0, 0, 0, 135, 136, 5, 17, 0, 0, 136, 137, 3, 10, 5, 0, 137, 9, 1, 0, 0, 0, 138, 139, 6, 5, -1, 0, 139, 140, 5, 43, 0, 0, 140, 167, 3, 10, 5, 7, 141, 167, 3, 14, 7, 0, 142, 167, 3, 12, 6, 0, 143, 145, 3, 14, 7, 0, 144, 146, 5, 43, 0, 0, 145, 144, 1, 0, 0, 0, 145, 146, 1, 0, 0, 0, 146, 147, 1, 0, 0, 0, 147, 148, 5, 40, 0, 0, 148, 149, 5, 39, 0, 0, 149, 154, 3, 14, 7, 0, 150, 151, 5, 33, 0, 0, 151, 153, 3, 14, 7, 0, 152, 150, 1, 0, 0, 0, 153, 156, 1, 0, 0, 0, 154, 152, 1, 0, 0, 0, 154, 155, 1, 0, 0, 0, 155, 157, 1, 0, 0, 0, 156, 154, 1, 0, 0, 0, 157, 158, 5, 49, 0, 0, 158, 167, 1, 0, 0, 0, 159, 160, 3, 14, 7, 0, 160, 162, 5, 41, 0, 0, 161, 163, 5, 43, 0, 0, 162, 161, 1, 0, 0, 0, 162, 163, 1, 0, 0, 0, 163, 164, 1, 0, 0, 0, 164, 165, 5, 44, 0, 0, 165, 167, 1, 0, 0, 0, 166, 138, 1, 0, 0, 0, 166, 141, 1, 0, 0, 0, 166, 142, 1, 0, 0, 0, 166, 143, 1, 0, 0, 0, 166, 159, 1, 0, 0, 0, 167, 176, 1, 0, 0, 0, 168, 169, 10, 4, 0, 0, 169, 170, 5, 30, 0, 0, 170, 175, 3, 10, 5, 5, 171, 172, 10, 3, 0, 0, 172, 173, 5, 46, 0, 0, 173, 175, 3, 10, 5, 4, 174, 168, 1, 0, 0, 0, 174, 171, 1, 0, 0, 0, 175, 178, 1, 0, 0, 0, 176, 174, 1, 0, 0, 0, 176, 177, 1, 0, 0, 0, 177, 11, 1, 0, 0, 0, 178, 176, 1, 0, 0, 0, 179, 181, 3, 14, 7, 0, 180, 182, 5, 43, 0, 0, 181, 180, 1, 0, 0, 0, 181, 182, 1, 0, 0, 0, 182, 183, 1, 0, 0, 0, 183, 184, 5, 42, 0, 0, 184, 185, 3, 86, 43, 0, 185, 194, 1, 0, 0, 0, 186, 188, 3, 14, 7, 0, 187, 189, 5, 43, 0, 0, 188, 187, 1, 0, 0, 0, 188, 189, 1, 0, 0, 0, 189, 190, 1, 0, 0, 0, 190, 191, 5, 48, 0, 0, 191, 192, 3, 86, 43, 0, 192, 194, 1, 0, 0, 0, 193, 179, 1, 0, 0, 0, 193, 186, 1, 0, 0, 0, 194, 13, 1, 0, 0, 0, 195, 201, 3, 16, 8, 0, 196, 197, 3, 16, 8, 0, 197, 198, 3, 88, 44, 0, 198, 199, 3, 16, 8, 0, 199, 201, 1, 0, 0, 0, 200, 195, 1, 0, 0, 0, 200, 196, 1, 0, 0, 0, 201, 15, 1, 0, 0, 0, 202, 203, 6, 8, -1, 0, 203, 207, 3, 18, 9, 0, 204, 205, 7, 0, 0, 0, 205, 207, 3, 16, 8, 3, 206, 202, 1, 0, 0, 0, 206, 204, 1, 0, 0, 0, 207, 216, 1, 0, 0, 0, 208, 209, 10, 2, 0, 0, 209, 210, 7, 1, 0, 0, 210, 215, 3, 16, 8, 3, 211, 212, 10, 1, 0, 0, 212, 213, 7, 0, 0, 0, 213, 215, 3, 16, 8, 2, 214, 208, 1, 0, 0, 0, 214, 211, 1, 0, 0, 0, 215, 218, 1, 0, 0, 0, 216, 214, 1, 0, 0, 0, 216, 217, 1, 0, 0, 0, 217, 17, 1, 0, 0, 0, 218, 216, 1, 0, 0, 0, 219, 227, 3, 52, 26, 0, 220, 227, 3, 44, 22, 0, 221, 227, 3, 20, 10, 0, 222, 223, 5, 39, 0, 0, 223, 224, 3, 10, 5, 0, 224, 225, 5, 49, 0, 0, 225, 227, 1, 0, 0, 0, 226, 219, 1, 0, 0, 0, 226, 220, 1, 0, 0, 0, 226, 221, 1, 0, 0, 0, 226, 222, 1, 0, 0, 0, 227, 19, 1, 0, 0, 0, 228, 229, 3, 48, 24, 0, 229, 239, 5, 39, 0, 0, 230, 240, 5, 60, 0, 0, 231, 236, 3, 10, 5, 0, 232, 233, 5, 33, 0, 0, 233, 235, 3, 10, 5, 0, 234, 232, 1, 0, 0, 0, 235, 238, 1, 0, 0, 0, 236, 234, 1, 0, 0, 0, 236, 237, 1, 0, 0, 0, 237, 240, 1, 0, 0, 0, 238, 236, 1, 0, 0, 0, 239, 230, 1, 0, 0, 0, 239, 231, 1, 0, 0, 0, 239, 240, 1, 0, 0, 0, 240, 241, 1, 0, 0, 0, 241, 242, 5, 49, 0, 0, 242, 21, 1, 0, 0, 0, 243, 244, 5, 13, 0, 0, 244, 245, 3, 24, 12, 0, 245, 23, 1, 0, 0, 0, 246, 251, 3, 26, 13, 0, 247, 248, 5, 33, 0, 0, 248, 250, 3, 26, 13, 0, 249, 247, 1, 0, 0, 0, 250, 253, 1, 0, 0, 0, 251, 249, 1, 0, 0, 0, 251, 252, 1, 0, 0, 0, 252, 25, 1, 0, 0, 0, 253, 251, 1, 0, 0, 0, 254, 260, 3, 10, 5, 0, 255, 256, 3, 44, 22, 0, 256, 257, 5, 32, 0, 0, 257, 258, 3, 10, 5, 0, 258, 260, 1, 0, 0, 0, 259, 254, 1, 0, 0, 0, 259, 255, 1, 0, 0, 0, 260, 27, 1, 0, 0, 0, 261, 262, 5, 6, 0, 0, 262, 267, 3, 42, 21, 0, 263, 264, 5, 33, 0, 0, 264, 266, 3, 42, 21, 0, 265, 263, 1, 0, 0, 0, 266, 269, 1, 0, 0, 0, 267, 265, 1, 0, 0, 0, 267, 268, 1, 0, 0, 0, 268, 271, 1, 0, 0, 0, 269, 267, 1, 0, 0, 0, 270, 272, 3, 30, 15, 0, 271, 270, 1, 0, 0, 0, 271, 272, 1, 0, 0, 0, 272, 29, 1, 0, 0, 0, 273, 276, 3, 32, 16, 0, 274, 276, 3, 34, 17, 0, 275, 273, 1, 0, 0, 0, 275, 274, 1, 0, 0, 0, 276, 31, 1, 0, 0, 0, 277, 278, 5, 70, 0, 0, 278, 283, 3, 42, 21, 0, 279, 280, 5, 33, 0, 0, 280, 282, 3, 42, 21, 0, 281, 279, 1, 0, 0, 0, 282, 285, 1, 0, 0, 0, 283, 281, 1, 0, 0, 0, 283, 284, 1, 0, 0, 0, 284, 33, 1, 0, 0, 0, 285, 283, 1, 0, 0, 0, 286, 287, 5, 63, 0, 0, 287, 288, 3, 32, 16, 0, 288, 289, 5, 64, 0, 0, 289, 35, 1, 0, 0, 0, 290, 291, 5, 4, 0, 0, 291, 292, 3, 24, 12, 0, 292, 37, 1, 0, 0, 0, 293, 295, 5, 16, 0, 0, 294, 296, 3, 24, 12, 0, 295, 294, 1, 0, 0, 0, 295, 296, 1, 0, 0, 0, 296, 299, 1, 0, 0, 0, 297, 298, 5, 29, 0, 0, 298, 300, 3, 24, 12, 0, 299, 297, 1, 0, 0, 0, 299, 300, 1, 0, 0, 0, 300, 39, 1, 0, 0, 0, 301, 302, 5, 8, 0, 0, 302, 305, 3, 24, 12, 0, 303, 304, 5, 29, 0, 0, 304, 306, 3, 24, 12, 0, 305, 303, 1, 0, 0, 0, 305, 306, 1, 0, 0, 0, 306, 41, 1, 0, 0, 0, 307, 308, 7, 2, 0, 0, 308, 43, 1, 0, 0, 0, 309, 314, 3, 48, 24, 0, 310, 311, 5, 35, 0, 0, 311, 313, 3, 48, 24, 0, 312, 310, 1, 0, 0, 0, 313, 316, 1, 0, 0, 0, 314, 312, 1, 0, 0, 0, 314, 315, 1, 0, 0, 0, 315, 45, 1, 0, 0, 0, 316, 314, 1, 0, 0, 0, 317, 322, 3, 50, 25, 0, 318, 319, 5, 35, 0, 0, 319, 321, 3, 50, 25, 0, 320, 318, 1, 0, 0, 0, 321, 324, 1, 0, 0, 0, 322, 320, 1, 0, 0, 0, 322, 323, 1, 0, 0, 0, 323, 47, 1, 0, 0, 0, 324, 322, 1, 0, 0, 0, 325, 326, 7, 3, 0, 0, 326, 49, 1, 0, 0, 0, 327, 328, 5, 75, 0, 0, 328, 51, 1, 0, 0, 0, 329, 372, 5, 44, 0, 0, 330, 331, 3, 84, 42, 0, 331, 332, 5, 65, 0, 0, 332, 372, 1, 0, 0, 0, 333, 372, 3, 82, 41, 0, 334, 372, 3, 84, 42, 0, 335, 372, 3, 78, 39, 0, 336, 372, 5, 47, 0, 0, 337, 372, 3, 86, 43, 0, 338, 339, 5, 63, 0, 0, 339, 344, 3, 80, 40, 0, 340, 341, 5, 33, 0, 0, 341, 343, 3, 80, 40, 0, 342, 340, 1, 0, 0, 0, 343, 346, 1, 0, 0, 0, 344, 342, 1, 0, 0, 0, 344, 345, 1, 0, 0, 0, 345, 347, 1, 0, 0, 0, 346, 344, 1, 0, 0, 0, 347, 348, 5, 64, 0, 0, 348, 372, 1, 0, 0, 0, 349, 350, 5, 63, 0, 0, 350, 355, 3, 78, 39, 0, 351, 352, 5, 33, 0, 0, 352, 354, 3, 78, 39, 0, 353, 351, 1, 0, 0, 0, 354, 357, 1, 0, 0, 0, 355, 353, 1, 0, 0, 0, 355, 356, 1, 0, 0, 0, 356, 358, 1, 0, 0, 0, 357, 355, 1, 0, 0, 0, 358, 359, 5, 64, 0, 0, 359, 372, 1, 0, 0, 0, 360, 361, 5, 63, 0, 0, 361, 366, 3, 86, 43, 0, 362, 363, 5, 33, 0, 0, 363, 365, 3, 86, 43, 0, 364, 362, 1, 0, 0, 0, 365, 368, 1, 0, 0, 0, 366, 364, 1, 0, 0, 0, 366, 367, 1, 0, 0, 0, 367, 369, 1, 0, 0, 0, 368, 366, 1, 0, 0, 0, 369, 370, 5, 64, 0, 0, 370, 372, 1, 0, 0, 0, 371, 329, 1, 0, 0, 0, 371, 330, 1, 0, 0, 0, 371, 333, 1, 0, 0, 0, 371, 334, 1, 0, 0, 0, 371, 335, 1, 0, 0, 0, 371, 336, 1, 0, 0, 0, 371, 337, 1, 0, 0, 0, 371, 338, 1, 0, 0, 0, 371, 349, 1, 0, 0, 0, 371, 360, 1, 0, 0, 0, 372, 53, 1, 0, 0, 0, 373, 374, 5, 10, 0, 0, 374, 375, 5, 27, 0, 0, 375, 55, 1, 0, 0, 0, 376, 377, 5, 15, 0, 0, 377, 382, 3, 58, 29, 0, 378, 379, 5, 33, 0, 0, 379, 381, 3, 58, 29, 0, 380, 378, 1, 0, 0, 0, 381, 384, 1, 0, 0, 0, 382, 380, 1, 0, 0, 0, 382, 383, 1, 0, 0, 0, 383, 57, 1, 0, 0, 0, 384, 382, 1, 0, 0, 0, 385, 387, 3, 10, 5, 0, 386, 388, 7, 4, 0, 0, 387, 386, 1, 0, 0, 0, 387, 388, 1, 0, 0, 0, 388, 391, 1, 0, 0, 0, 389, 390, 5, 45, 0, 0, 390, 392, 7, 5, 0, 0, 391, 389, 1, 0, 0, 0, 391, 392, 1, 0, 0, 0, 392, 59, 1, 0, 0, 0, 393, 394, 5, 9, 0, 0, 394, 399, 3, 46, 23, 0, 395, 396, 5, 33, 0, 0, 396, 398, 3, 46, 23, 0, 397, 395, 1, 0, 0, 0, 398, 401, 1, 0, 0, 0, 399, 397, 1, 0, 0, 0, 399, 400, 1, 0, 0, 0, 400, 61, 1, 0, 0, 0, 401, 399, 1, 0, 0, 0, 402, 403, 5, 2, 0, 0, 403, 408, 3, 46, 23, 0, 404, 405, 5, 33, 0, 0, 405, 407, 3, 46, 23, 0, 406, 404, 1, 0, 0, 0, 407, 410, 1, 0, 0, 0, 408, 406, 1, 0, 0, 0, 408, 409, 1, 0, 0, 0, 409, 63, 1, 0, 0, 0, 410, 408, 1, 0, 0, 0, 411, 412, 5, 12, 0, 0, 412, 417, 3, 66, 33, 0, 413, 414, 5, 33, 0, 0, 414, 416, 3, 66, 33, 0, 415, 413, 1, 0, 0, 0, 416, 419, 1, 0, 0, 0, 417, 415, 1, 0, 0, 0, 417, 418, 1, 0, 0, 0, 418, 65, 1, 0, 0, 0, 419, 417, 1, 0, 0, 0, 420, 421, 3, 46, 23, 0, 421, 422, 5, 79, 0, 0, 422, 423, 3, 46, 23, 0, 423, 67, 1, 0, 0, 0, 424, 425, 5, 1, 0, 0, 425, 426, 3, 18, 9, 0, 426, 428, 3, 86, 43, 0, 427, 429, 3, 74, 37, 0, 428, 427, 1, 0, 0, 0, 428, 429, 1, 0, 0, 0, 429, 69, 1, 0, 0, 0, 430, 431, 5, 7, 0, 0, 431, 432, 3, 18, 9, 0, 432, 433, 3, 86, 43, 0, 433, 71, 1, 0, 0, 0, 434, 435, 5, 11, 0, 0, 435, 436, 3, 44, 22, 0, 436, 73, 1, 0, 0, 0, 437, 442, 3, 76, 38, 0, 438, 439, 5, 33, 0, 0, 439, 441, 3, 76, 38, 0, 440, 438, 1, 0, 0, 0, 441, 444, 1, 0, 0, 0, 442, 440, 1, 0, 0, 0, 442, 443, 1, 0, 0, 0, 443, 75, 1, 0, 0, 0, 444, 442, 1, 0, 0, 0, 445, 446, 3, 48, 24, 0, 446, 447, 5, 32, 0, 0, 447, 448, 3, 52, 26, 0, 448, 77, 1, 0, 0, 0, 449, 450, 7, 6, 0, 0, 450, 79, 1, 0, 0, 0, 451, 454, 3, 82, 41, 0, 452, 454, 3, 84, 42, 0, 453, 451, 1, 0, 0, 0, 453, 452, 1, 0, 0, 0, 454, 81, 1, 0, 0, 0, 455, 457, 7, 0, 0, 0, 456, 455, 1, 0, 0, 0, 456, 457, 1, 0, 0, 0, 457, 458, 1, 0, 0, 0, 458, 459, 5, 28, 0, 0, 459, 83, 1, 0, 0, 0, 460, 462, 7, 0, 0, 0, 461, 460, 1, 0, 0, 0, 461, 462, 1, 0, 0, 0, 462, 463, 1, 0, 0, 0, 463, 464, 5, 27, 0, 0, 464, 85, 1, 0, 0, 0, 465, 466, 5, 26, 0, 0, 466, 87, 1, 0, 0, 0, 467, 468, 7, 7, 0, 0, 468, 89, 1, 0, 0, 0, 469, 470, 5, 5, 0, 0, 470, 471, 3, 92, 46, 0, 471, 91, 1, 0, 0, 0, 472, 473, 5, 63, 0, 0, 473, 474, 3, 2, 1, 0, 474, 475, 5, 64, 0, 0, 475, 93, 1, 0, 0, 0, 476, 477, 5, 14, 0, 0, 477, 481, 5, 95, 0, 0, 478, 479, 5, 14, 0, 0, 479, 481, 5, 96, 0, 0, 480, 476, 1, 0, 0, 0, 480, 478, 1, 0, 0, 0, 481, 95, 1, 0, 0, 0, 482, 483, 5, 3, 0, 0, 483, 486, 5, 85, 0, 0, 484, 485, 5, 83, 0, 0, 485, 487, 3, 46, 23, 0, 486, 484, 1, 0, 0, 0, 486, 487, 1, 0, 0, 0, 487, 497, 1, 0, 0, 0, 488, 489, 5, 84, 0, 0, 489, 494, 3, 98, 49, 0, 490, 491, 5, 33, 0, 0, 491, 493, 3, 98, 49, 0, 492, 490, 1, 0, 0, 0, 493, 496, 1, 0, 0, 0, 494, 492, 1, 0, 0, 0, 494, 495, 1, 0, 0, 0, 495, 498, 1, 0, 0, 0, 496, 494, 1, 0, 0, 0, 497, 488, 1, 0, 0, 0, 497, 498, 1, 0, 0, 0, 498, 97, 1, 0, 0, 0, 499, 500, 3, 46, 23, 0, 500, 501, 5, 32, 0, 0, 501, 503, 1, 0, 0, 0, 502, 499, 1, 0, 0, 0, 502, 503, 1, 0, 0, 0, 503, 504, 1, 0, 0, 0, 504, 505, 3, 46, 23, 0, 505, 99, 1, 0, 0, 0, 50, 111, 118, 133, 145, 154, 162, 166, 174, 176, 181, 188, 193, 200, 206, 214, 216, 226, 236, 239, 251, 259, 267, 271, 275, 283, 295, 299, 305, 314, 322, 344, 355, 366, 371, 382, 387, 391, 399, 408, 417, 428, 442, 453, 456, 461, 480, 486, 494, 497, 502] \ No newline at end of file diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java index ab83872ce0ac3..e0e3a77c0ad6a 100644 --- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBaseParser.java @@ -4121,7 +4121,6 @@ public final StringContext string() throws RecognitionException { @SuppressWarnings("CheckReturnValue") public static class ComparisonOperatorContext extends ParserRuleContext { public TerminalNode EQ() { return getToken(EsqlBaseParser.EQ, 0); } - public TerminalNode CIEQ() { return getToken(EsqlBaseParser.CIEQ, 0); } public TerminalNode NEQ() { return getToken(EsqlBaseParser.NEQ, 0); } public TerminalNode LT() { return getToken(EsqlBaseParser.LT, 0); } public TerminalNode LTE() { return getToken(EsqlBaseParser.LTE, 0); } @@ -4156,7 +4155,7 @@ public final ComparisonOperatorContext comparisonOperator() throws RecognitionEx { setState(467); _la = _input.LA(1); - if ( !((((_la) & ~0x3f) == 0 && ((1L << _la) & 285978576338026496L) != 0)) ) { + if ( !((((_la) & ~0x3f) == 0 && ((1L << _la) & 281474976710656000L) != 0)) ) { _errHandler.recoverInline(this); } else { @@ -4657,7 +4656,7 @@ private boolean operatorExpression_sempred(OperatorExpressionContext _localctx, "\u0002\u0004\u0006\b\n\f\u000e\u0010\u0012\u0014\u0016\u0018\u001a\u001c"+ "\u001e \"$&(*,.02468:<>@BDFHJLNPRTVXZ\\^`b\u0000\b\u0001\u0000:;\u0001"+ "\u0000<>\u0002\u0000BBGG\u0001\u0000AB\u0002\u0000\u001f\u001f\"\"\u0001"+ - "\u0000%&\u0002\u0000$$22\u0001\u000039\u0215\u0000d\u0001\u0000\u0000"+ + "\u0000%&\u0002\u0000$$22\u0002\u00003359\u0215\u0000d\u0001\u0000\u0000"+ "\u0000\u0002g\u0001\u0000\u0000\u0000\u0004v\u0001\u0000\u0000\u0000\u0006"+ "\u0085\u0001\u0000\u0000\u0000\b\u0087\u0001\u0000\u0000\u0000\n\u00a6"+ "\u0001\u0000\u0000\u0000\f\u00c1\u0001\u0000\u0000\u0000\u000e\u00c8\u0001"+ diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java index 28173caaac39f..b9c0e9b34b552 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java @@ -1551,6 +1551,7 @@ public void testMissingAttributeException_InChainedEval() { assertThat(e.getMessage(), containsString("Unknown column [x5], did you mean any of [x1, x2, x3]?")); } + @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/103599") public void testInsensitiveEqualsWrongType() { var e = expectThrows(VerificationException.class, () -> analyze(""" from test diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java index 8995f954a8a0e..4030e7e0bcbef 100644 --- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java +++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/PhysicalPlanOptimizerTests.java @@ -1571,6 +1571,7 @@ public void testPushDownRLike() { * {"term":{"first_name":{"value":"foo","case_insensitive":true}}},"source":"first_name =~ \"foo\"@2:9"}}] * [_doc{f}#23], limit[500], sort[] estimatedRowSize[324] */ + @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/103599") public void testPushDownEqualsIgnoreCase() { var plan = physicalPlan(""" from test @@ -1601,6 +1602,7 @@ public void testPushDownEqualsIgnoreCase() { * \_FieldExtractExec[first_name{f}#7] * \_EsQueryExec[test], query[][_doc{f}#27], limit[], sort[] estimatedRowSize[374] */ + @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/103599") public void testNoPushDownEvalEqualsIgnoreCase() { var plan = physicalPlan(""" from test From 81332093550e6f11b1ecca1ddae345cce06d9c86 Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 19 Feb 2024 10:26:00 +0000 Subject: [PATCH 27/45] `URLRepository` should not block shutdown (#105588) Today a node with a registered `URLRepository` will not shut down cleanly because it never releases the last of the `activityRefs`. This commit fixes that. --- docs/changelog/105588.yaml | 5 +++++ .../repositories/url/URLSnapshotRestoreIT.java | 13 +++++++++++++ .../repositories/url/URLRepository.java | 1 + 3 files changed, 19 insertions(+) create mode 100644 docs/changelog/105588.yaml diff --git a/docs/changelog/105588.yaml b/docs/changelog/105588.yaml new file mode 100644 index 0000000000000..e43ff8cd75c60 --- /dev/null +++ b/docs/changelog/105588.yaml @@ -0,0 +1,5 @@ +pr: 105588 +summary: '`URLRepository` should not block shutdown' +area: Snapshot/Restore +type: bug +issues: [] diff --git a/modules/repository-url/src/internalClusterTest/java/org/elasticsearch/repositories/url/URLSnapshotRestoreIT.java b/modules/repository-url/src/internalClusterTest/java/org/elasticsearch/repositories/url/URLSnapshotRestoreIT.java index e0d8eb86613ba..80798d931e93f 100644 --- a/modules/repository-url/src/internalClusterTest/java/org/elasticsearch/repositories/url/URLSnapshotRestoreIT.java +++ b/modules/repository-url/src/internalClusterTest/java/org/elasticsearch/repositories/url/URLSnapshotRestoreIT.java @@ -125,4 +125,17 @@ public void testUrlRepository() throws Exception { getSnapshotsResponse = client.admin().cluster().prepareGetSnapshots("url-repo").get(); assertThat(getSnapshotsResponse.getSnapshots().size(), equalTo(0)); } + + public void testUrlRepositoryPermitsShutdown() throws Exception { + assertAcked( + client().admin() + .cluster() + .preparePutRepository("url-repo") + .setType(URLRepository.TYPE) + .setVerify(false) + .setSettings(Settings.builder().put(URLRepository.URL_SETTING.getKey(), "http://localhost/")) + ); + + internalCluster().fullRestart(); // just checking that URL repositories don't block node shutdown + } } diff --git a/modules/repository-url/src/main/java/org/elasticsearch/repositories/url/URLRepository.java b/modules/repository-url/src/main/java/org/elasticsearch/repositories/url/URLRepository.java index db68ecf93e4b7..3390a5ce8ab43 100644 --- a/modules/repository-url/src/main/java/org/elasticsearch/repositories/url/URLRepository.java +++ b/modules/repository-url/src/main/java/org/elasticsearch/repositories/url/URLRepository.java @@ -186,5 +186,6 @@ private static URL parseURL(String s) { @Override protected void doClose() { IOUtils.closeWhileHandlingException(httpClient); + super.doClose(); } } From cca26276b3965e24da9367c0ccfde411ec5124cd Mon Sep 17 00:00:00 2001 From: David Turner Date: Mon, 19 Feb 2024 10:47:46 +0000 Subject: [PATCH 28/45] Handle scheduler shutdown in `MasterService` (#105552) Today if `ThreadPool#scheduler` is shut down while submitting a task for execution by the `MasterService` then we directly throw the rejection exception to the caller. By the looks of it most callers don't expect this method to throw anything, so this commit adjusts the behaviour to fail the task instead. Closes #105549 --- .../cluster/service/MasterService.java | 31 ++++++++--- .../cluster/service/MasterServiceTests.java | 53 +++++++++++++++++++ .../elasticsearch/test/ESIntegTestCase.java | 13 ++++- 3 files changed, 90 insertions(+), 7 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/cluster/service/MasterService.java b/server/src/main/java/org/elasticsearch/cluster/service/MasterService.java index f037f4780b28d..48917ca84e89b 100644 --- a/server/src/main/java/org/elasticsearch/cluster/service/MasterService.java +++ b/server/src/main/java/org/elasticsearch/cluster/service/MasterService.java @@ -1470,8 +1470,13 @@ private void completeTask(Exception e) { @Override public String toString() { - return Strings.format("master service timeout handler for [%s][%s] after [%s]", source, taskHolder.get(), timeout); + return getTimeoutTaskDescription(source, taskHolder.get(), timeout); } + + } + + static String getTimeoutTaskDescription(String source, Object task, TimeValue timeout) { + return Strings.format("master service timeout handler for [%s][%s] after [%s]", source, task, timeout); } /** @@ -1522,11 +1527,25 @@ public void submitTask(String source, T task, @Nullable TimeValue timeout) { final var taskHolder = new AtomicReference<>(task); final Scheduler.Cancellable timeoutCancellable; if (timeout != null && timeout.millis() > 0) { - timeoutCancellable = threadPool.schedule( - new TaskTimeoutHandler<>(timeout, source, taskHolder), - timeout, - threadPool.generic() - ); + try { + timeoutCancellable = threadPool.schedule( + new TaskTimeoutHandler<>(timeout, source, taskHolder), + timeout, + threadPool.generic() + ); + } catch (Exception e) { + assert e instanceof EsRejectedExecutionException esre && esre.isExecutorShutdown() : e; + task.onFailure( + new FailedToCommitClusterStateException( + "could not schedule timeout handler for [%s][%s] on queue [%s]", + e, + source, + task, + name + ) + ); + return; + } } else { timeoutCancellable = null; } diff --git a/server/src/test/java/org/elasticsearch/cluster/service/MasterServiceTests.java b/server/src/test/java/org/elasticsearch/cluster/service/MasterServiceTests.java index 6e2d360a0d24f..77b2f0112ad43 100644 --- a/server/src/test/java/org/elasticsearch/cluster/service/MasterServiceTests.java +++ b/server/src/test/java/org/elasticsearch/cluster/service/MasterServiceTests.java @@ -2121,6 +2121,59 @@ public void onFailure(Exception e) { } } + public void testTimeoutRejectionBehaviourAtSubmission() { + + final var source = randomIdentifier(); + final var taskDescription = randomIdentifier(); + final var timeout = TimeValue.timeValueMillis(between(0, 100000)); + + final var actionCount = new AtomicInteger(); + final var deterministicTaskQueue = new DeterministicTaskQueue(); + final var threadPool = + // a threadpool which simulates the rejection of a master service timeout handler, but runs all other tasks as normal + deterministicTaskQueue.getThreadPool(r -> { + if (r.toString().equals(MasterService.getTimeoutTaskDescription(source, taskDescription, timeout))) { + // assertTrue because this should happen exactly once + assertTrue(actionCount.compareAndSet(0, 1)); + throw new EsRejectedExecutionException("simulated rejection", true); + } else { + return r; + } + }); + + try (var masterService = createMasterService(true, null, threadPool, new StoppableExecutorServiceWrapper(threadPool.generic()))) { + masterService.createTaskQueue( + "queue", + randomFrom(Priority.values()), + batchExecutionContext -> fail(null, "should not execute batch") + ).submitTask(source, new ClusterStateTaskListener() { + @Override + public void onFailure(Exception e) { + if (e instanceof FailedToCommitClusterStateException + && e.getMessage().startsWith("could not schedule timeout handler") + && e.getCause() instanceof EsRejectedExecutionException esre + && esre.isExecutorShutdown() + && esre.getMessage().equals("simulated rejection")) { + // assertTrue because we must receive the exception we synthesized, exactly once, after triggering the rejection + assertTrue(actionCount.compareAndSet(1, 2)); + } else { + // fail the test if we get anything else + throw new AssertionError("unexpected exception", e); + } + } + + @Override + public String toString() { + return taskDescription; + } + }, timeout); + + assertFalse(deterministicTaskQueue.hasRunnableTasks()); + assertFalse(deterministicTaskQueue.hasDeferredTasks()); + assertEquals(2, actionCount.get()); // ensures this test doesn't accidentally become trivial: both expected actions happened + } + } + @TestLogging(reason = "verifying DEBUG logs", value = "org.elasticsearch.cluster.service.MasterService:DEBUG") public void testRejectionBehaviourAtCompletion() { diff --git a/test/framework/src/main/java/org/elasticsearch/test/ESIntegTestCase.java b/test/framework/src/main/java/org/elasticsearch/test/ESIntegTestCase.java index 82e02ec1db70f..11d4754eaa596 100644 --- a/test/framework/src/main/java/org/elasticsearch/test/ESIntegTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/test/ESIntegTestCase.java @@ -177,6 +177,7 @@ import java.util.concurrent.Callable; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; import java.util.concurrent.Executor; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; @@ -971,7 +972,17 @@ private ClusterHealthStatus ensureColor( // been removed by the master so that the health check applies to the set of nodes we expect to be part of the cluster. .waitForNodes(Integer.toString(cluster().size())); - final ClusterHealthResponse clusterHealthResponse = clusterAdmin().health(healthRequest).actionGet(); + final ClusterHealthResponse clusterHealthResponse; + try { + clusterHealthResponse = clusterAdmin().health(healthRequest).get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + logger.error("interrupted while waiting for health response", e); + throw new AssertionError("interrupted while waiting for health response", e); + } catch (ExecutionException e) { + logger.error("failed to get health response", e); + throw new AssertionError("failed to get health response", e); + } if (clusterHealthResponse.isTimedOut()) { final var allocationExplainRef = new AtomicReference(); final var clusterStateRef = new AtomicReference(); From 44b0047db5ed516c92a9f5d952319b17f29047c8 Mon Sep 17 00:00:00 2001 From: Niels Bauman <33722607+nielsbauman@users.noreply.github.com> Date: Mon, 19 Feb 2024 11:59:39 +0100 Subject: [PATCH 29/45] Don't stop checking if the `HealthNode` persistent task is present (#105449) We assumed that once the `HealthNode` persistent task is registered, we won't need to register it again. However, when, for instance, we restore from a snapshot (including cluster state) that was created in version <= 8.4.3, that task doesn't exist yet, which will result in the task being removed after the restore. By keeping the listener active, we will re-add the task after such a restore (or any other potential situation where the task might get deleted). --- docs/changelog/105449.yaml | 6 ++++ .../selection/HealthNodeTaskExecutor.java | 3 -- .../health/node/LocalHealthMonitorTests.java | 3 +- ....java => HealthNodeTaskExecutorTests.java} | 31 +++++++++---------- 4 files changed, 22 insertions(+), 21 deletions(-) create mode 100644 docs/changelog/105449.yaml rename server/src/test/java/org/elasticsearch/health/node/selection/{HealthNodeExecutorTests.java => HealthNodeTaskExecutorTests.java} (89%) diff --git a/docs/changelog/105449.yaml b/docs/changelog/105449.yaml new file mode 100644 index 0000000000000..b565d6c782bd9 --- /dev/null +++ b/docs/changelog/105449.yaml @@ -0,0 +1,6 @@ +pr: 105449 +summary: Don't stop checking if the `HealthNode` persistent task is present +area: Health +type: bug +issues: + - 98926 diff --git a/server/src/main/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutor.java b/server/src/main/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutor.java index 7b95d76bb0e7d..f183d8c7f1a82 100644 --- a/server/src/main/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutor.java +++ b/server/src/main/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutor.java @@ -159,9 +159,6 @@ void startTask(ClusterChangedEvent event) { if (event.state().clusterRecovered() && featureService.clusterHasFeature(event.state(), HealthFeatures.SUPPORTS_HEALTH)) { boolean healthNodeTaskExists = HealthNode.findTask(event.state()) != null; boolean isElectedMaster = event.localNodeMaster(); - if (isElectedMaster || healthNodeTaskExists) { - clusterService.removeListener(taskStarter); - } if (isElectedMaster && healthNodeTaskExists == false) { persistentTasksService.sendStartRequest( TASK_NAME, diff --git a/server/src/test/java/org/elasticsearch/health/node/LocalHealthMonitorTests.java b/server/src/test/java/org/elasticsearch/health/node/LocalHealthMonitorTests.java index bb24d3118d77b..768b646d84beb 100644 --- a/server/src/test/java/org/elasticsearch/health/node/LocalHealthMonitorTests.java +++ b/server/src/test/java/org/elasticsearch/health/node/LocalHealthMonitorTests.java @@ -27,7 +27,6 @@ import org.elasticsearch.health.HealthFeatures; import org.elasticsearch.health.HealthStatus; import org.elasticsearch.health.metadata.HealthMetadata; -import org.elasticsearch.health.node.selection.HealthNodeExecutorTests; import org.elasticsearch.health.node.tracker.HealthTracker; import org.elasticsearch.test.ESTestCase; import org.elasticsearch.threadpool.TestThreadPool; @@ -63,7 +62,7 @@ public class LocalHealthMonitorTests extends ESTestCase { @BeforeClass public static void setUpThreadPool() { - threadPool = new TestThreadPool(HealthNodeExecutorTests.class.getSimpleName()); + threadPool = new TestThreadPool(LocalHealthMonitorTests.class.getSimpleName()); } @AfterClass diff --git a/server/src/test/java/org/elasticsearch/health/node/selection/HealthNodeExecutorTests.java b/server/src/test/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutorTests.java similarity index 89% rename from server/src/test/java/org/elasticsearch/health/node/selection/HealthNodeExecutorTests.java rename to server/src/test/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutorTests.java index 0f76758d038c1..5460a45569d71 100644 --- a/server/src/test/java/org/elasticsearch/health/node/selection/HealthNodeExecutorTests.java +++ b/server/src/test/java/org/elasticsearch/health/node/selection/HealthNodeTaskExecutorTests.java @@ -8,6 +8,7 @@ package org.elasticsearch.health.node.selection; +import org.elasticsearch.action.ActionListener; import org.elasticsearch.cluster.ClusterChangedEvent; import org.elasticsearch.cluster.ClusterName; import org.elasticsearch.cluster.ClusterState; @@ -46,7 +47,7 @@ import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; -public class HealthNodeExecutorTests extends ESTestCase { +public class HealthNodeTaskExecutorTests extends ESTestCase { /** Needed by {@link ClusterService} **/ private static ThreadPool threadPool; @@ -66,7 +67,7 @@ public class HealthNodeExecutorTests extends ESTestCase { @BeforeClass public static void setUpThreadPool() { - threadPool = new TestThreadPool(HealthNodeExecutorTests.class.getSimpleName()); + threadPool = new TestThreadPool(HealthNodeTaskExecutorTests.class.getSimpleName()); } @Before @@ -91,20 +92,18 @@ public void tearDown() throws Exception { clusterService.close(); } - public void testTaskCreation() { - HealthNodeTaskExecutor executor = HealthNodeTaskExecutor.create( - clusterService, - persistentTasksService, - featureService, - settings, - clusterSettings - ); - executor.startTask(new ClusterChangedEvent("", initialState(), ClusterState.EMPTY_STATE)); - verify(persistentTasksService, times(1)).sendStartRequest( - eq("health-node"), - eq("health-node"), - eq(new HealthNodeTaskParams()), - any() + public void testTaskCreation() throws Exception { + HealthNodeTaskExecutor.create(clusterService, persistentTasksService, featureService, settings, clusterSettings); + clusterService.getClusterApplierService().onNewClusterState("initialization", this::initialState, ActionListener.noop()); + // Ensure that if the task is gone, it will be recreated. + clusterService.getClusterApplierService().onNewClusterState("initialization", this::initialState, ActionListener.noop()); + assertBusy( + () -> verify(persistentTasksService, times(2)).sendStartRequest( + eq("health-node"), + eq("health-node"), + eq(new HealthNodeTaskParams()), + any() + ) ); } From a103e3c7a44fbf3ba1df37119f949a8ae098fa6d Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Mon, 19 Feb 2024 12:05:51 +0100 Subject: [PATCH 30/45] Infrastructure for metering the update requests (#105063) udpate request that are sending a document (or part of it) should allow for metering the size of that doc the update request that are using a script should not be metered - reported size 0. this commit is following up on #104859 The parsing is of the update's document is being done in UpdateHelper - the same pattern we use to meter parsing in IngestService. If the script is being used, the size observed will be 0. The value observed is then reported in the TransportShardBulkAction and thanks to the value being 0 or positive it will not be metering the modified document again. This commit also renames the getDocumentParsingSupplier to getDocumentParsingProvider (this was accidentally omitted in the #104859) --- docs/changelog/105063.yaml | 5 +++ .../DocumentSizeObserverWithPipelinesIT.java | 2 +- .../internal/DocumentSizeObserverIT.java | 8 ++-- .../action/bulk/TransportShardBulkAction.java | 10 +++-- .../action/index/IndexRequest.java | 37 +++++++++++++++---- .../action/update/UpdateHelper.java | 21 ++++++++--- .../elasticsearch/node/NodeConstruction.java | 27 +++++++++----- .../DocumentParsingProviderPlugin.java | 6 +-- .../action/update/UpdateRequestTests.java | 12 +++--- .../snapshots/SnapshotResiliencyTests.java | 2 +- .../authz/AuthorizationServiceTests.java | 3 +- 11 files changed, 92 insertions(+), 41 deletions(-) create mode 100644 docs/changelog/105063.yaml diff --git a/docs/changelog/105063.yaml b/docs/changelog/105063.yaml new file mode 100644 index 0000000000000..668f8ac104493 --- /dev/null +++ b/docs/changelog/105063.yaml @@ -0,0 +1,5 @@ +pr: 105063 +summary: Infrastructure for metering the update requests +area: Infra/Metrics +type: enhancement +issues: [] diff --git a/modules/ingest-common/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverWithPipelinesIT.java b/modules/ingest-common/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverWithPipelinesIT.java index 95c19547df428..d1cdc719b02f1 100644 --- a/modules/ingest-common/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverWithPipelinesIT.java +++ b/modules/ingest-common/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverWithPipelinesIT.java @@ -78,7 +78,7 @@ public static class TestDocumentParsingProviderPlugin extends Plugin implements public TestDocumentParsingProviderPlugin() {} @Override - public DocumentParsingProvider getDocumentParsingSupplier() { + public DocumentParsingProvider getDocumentParsingProvider() { // returns a static instance, because we want to assert that the wrapping is called only once return new DocumentParsingProvider() { @Override diff --git a/server/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverIT.java b/server/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverIT.java index 645d4f2aec30c..fd6151e8eadde 100644 --- a/server/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverIT.java +++ b/server/src/internalClusterTest/java/org/elasticsearch/plugins/internal/DocumentSizeObserverIT.java @@ -39,7 +39,7 @@ public void testDocumentIsReportedUponBulk() throws IOException { new IndexRequest(TEST_INDEX_NAME).id("1").source(jsonBuilder().startObject().field("test", "I am sam i am").endObject()) ).actionGet(); assertTrue(hasWrappedParser); - // there are more assertions in a TestDocumentParsingSupplierPlugin + // there are more assertions in a TestDocumentParsingProviderPlugin hasWrappedParser = false; // the format of the request does not matter @@ -47,7 +47,7 @@ public void testDocumentIsReportedUponBulk() throws IOException { new IndexRequest(TEST_INDEX_NAME).id("2").source(cborBuilder().startObject().field("test", "I am sam i am").endObject()) ).actionGet(); assertTrue(hasWrappedParser); - // there are more assertions in a TestDocumentParsingSupplierPlugin + // there are more assertions in a TestDocumentParsingProviderPlugin hasWrappedParser = false; // white spaces does not matter @@ -59,7 +59,7 @@ public void testDocumentIsReportedUponBulk() throws IOException { } """, XContentType.JSON)).actionGet(); assertTrue(hasWrappedParser); - // there are more assertions in a TestDocumentParsingSupplierPlugin + // there are more assertions in a TestDocumentParsingProviderPlugin } @Override @@ -72,7 +72,7 @@ public static class TestDocumentParsingProviderPlugin extends Plugin implements public TestDocumentParsingProviderPlugin() {} @Override - public DocumentParsingProvider getDocumentParsingSupplier() { + public DocumentParsingProvider getDocumentParsingProvider() { return new DocumentParsingProvider() { @Override diff --git a/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java b/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java index 5b64428441ed3..70168f6a2b516 100644 --- a/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java +++ b/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java @@ -455,16 +455,20 @@ public void onFailure(Exception e) { } /** - * Creates a new document size observerl + * Creates a new document size observer * @param documentParsingProvider a provider to create a new observer. * @param request an index request to provide information about bytes being already parsed. - * @return a Fixed version of DocumentSizeObserver if parsing already happened (in IngestService). + * @return a Fixed version of DocumentSizeObserver if parsing already happened (in IngestService, UpdateHelper) + * and there is a value to be reported >0 * It would be pre-populated with information about how many bytes were already parsed - * or return a new 'empty' DocumentSizeObserver. + * or a noop instance if parsed bytes in IngestService/UpdateHelper was 0 (like when empty doc or script in update) + * or return a new DocumentSizeObserver that will be used when parsing. */ private static DocumentSizeObserver getDocumentSizeObserver(DocumentParsingProvider documentParsingProvider, IndexRequest request) { if (request.getNormalisedBytesParsed() != -1) { return documentParsingProvider.newFixedSizeDocumentObserver(request.getNormalisedBytesParsed()); + } else if (request.getNormalisedBytesParsed() == 0) { + return DocumentSizeObserver.EMPTY_INSTANCE; } return documentParsingProvider.newDocumentSizeObserver(); } diff --git a/server/src/main/java/org/elasticsearch/action/index/IndexRequest.java b/server/src/main/java/org/elasticsearch/action/index/IndexRequest.java index 03a99969a7deb..5bdd197b80d2c 100644 --- a/server/src/main/java/org/elasticsearch/action/index/IndexRequest.java +++ b/server/src/main/java/org/elasticsearch/action/index/IndexRequest.java @@ -59,14 +59,14 @@ /** * Index request to index a typed JSON document into a specific index and make it searchable. - * + *

* The index requires the {@link #index()}, {@link #id(String)} and * {@link #source(byte[], XContentType)} to be set. - * + *

* The source (content to index) can be set in its bytes form using ({@link #source(byte[], XContentType)}), * its string form ({@link #source(String, XContentType)}) or using a {@link org.elasticsearch.xcontent.XContentBuilder} * ({@link #source(org.elasticsearch.xcontent.XContentBuilder)}). - * + *

* If the {@link #id(String)} is not set, it will be automatically generated. * * @see IndexResponse @@ -453,7 +453,7 @@ public IndexRequest source(Map source, XContentType contentType, bool /** * Sets the document source to index. - * + *

* Note, its preferable to either set it using {@link #source(org.elasticsearch.xcontent.XContentBuilder)} * or using the {@link #source(byte[], XContentType)}. */ @@ -632,7 +632,7 @@ public IndexRequest versionType(VersionType versionType) { /** * only perform this indexing request if the document was last modification was assigned the given * sequence number. Must be used in combination with {@link #setIfPrimaryTerm(long)} - * + *

* If the document last modification was assigned a different sequence number a * {@link org.elasticsearch.index.engine.VersionConflictEngineException} will be thrown. */ @@ -647,7 +647,7 @@ public IndexRequest setIfSeqNo(long seqNo) { /** * only performs this indexing request if the document was last modification was assigned the given * primary term. Must be used in combination with {@link #setIfSeqNo(long)} - * + *

* If the document last modification was assigned a different term a * {@link org.elasticsearch.index.engine.VersionConflictEngineException} will be thrown. */ @@ -670,7 +670,7 @@ public long ifSeqNo() { /** * If set, only perform this indexing request if the document was last modification was assigned this primary term. - * + *

* If the document last modification was assigned a different term a * {@link org.elasticsearch.index.engine.VersionConflictEngineException} will be thrown. */ @@ -933,16 +933,36 @@ public void setRawTimestamp(Object rawTimestamp) { this.rawTimestamp = rawTimestamp; } + /** + * Returns a number of bytes observed when parsing a document in earlier stages of ingestion (like update/ingest service) + * Defaults to -1 when a document size was not observed in earlier stages. + * @return a number of bytes observed + */ public long getNormalisedBytesParsed() { return normalisedBytesParsed; } - public void setNormalisedBytesParsed(long normalisedBytesParsed) { + /** + * Sets number of bytes observed by a DocumentSizeObserver + * @return an index request + */ + public IndexRequest setNormalisedBytesParsed(long normalisedBytesParsed) { this.normalisedBytesParsed = normalisedBytesParsed; + return this; + } + + /** + * when observing document size while parsing, this method indicates that this request should not be recorded. + * @return an index request + */ + public IndexRequest noParsedBytesToReport() { + this.normalisedBytesParsed = 0; + return this; } /** * Adds the pipeline to the list of executed pipelines, if listExecutedPipelines is true + * * @param pipeline */ public void addPipeline(String pipeline) { @@ -957,6 +977,7 @@ public void addPipeline(String pipeline) { /** * This returns the list of pipelines executed on the document for this request. If listExecutedPipelines is false, the response will be * null, even if pipelines were executed. If listExecutedPipelines is true but no pipelines were executed, the list will be empty. + * * @return */ @Nullable diff --git a/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java b/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java index de6cffa10f48d..0738e2cb111bb 100644 --- a/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java +++ b/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java @@ -26,6 +26,8 @@ import org.elasticsearch.index.mapper.RoutingFieldMapper; import org.elasticsearch.index.shard.IndexShard; import org.elasticsearch.index.shard.ShardId; +import org.elasticsearch.plugins.internal.DocumentParsingProvider; +import org.elasticsearch.plugins.internal.DocumentSizeObserver; import org.elasticsearch.script.Script; import org.elasticsearch.script.ScriptService; import org.elasticsearch.script.UpdateCtxMap; @@ -47,9 +49,11 @@ public class UpdateHelper { private static final Logger logger = LogManager.getLogger(UpdateHelper.class); private final ScriptService scriptService; + private final DocumentParsingProvider documentParsingProvider; - public UpdateHelper(ScriptService scriptService) { + public UpdateHelper(ScriptService scriptService, DocumentParsingProvider documentParsingProvider) { this.scriptService = scriptService; + this.documentParsingProvider = documentParsingProvider; } /** @@ -174,14 +178,19 @@ static String calculateRouting(GetResult getResult, @Nullable IndexRequest updat * Prepare the request for merging the existing document with a new one, can optionally detect a noop change. Returns a {@code Result} * containing a new {@code IndexRequest} to be executed on the primary and replicas. */ - static Result prepareUpdateIndexRequest(ShardId shardId, UpdateRequest request, GetResult getResult, boolean detectNoop) { + Result prepareUpdateIndexRequest(ShardId shardId, UpdateRequest request, GetResult getResult, boolean detectNoop) { final IndexRequest currentRequest = request.doc(); final String routing = calculateRouting(getResult, currentRequest); + final DocumentSizeObserver documentSizeObserver = documentParsingProvider.newDocumentSizeObserver(); final Tuple> sourceAndContent = XContentHelper.convertToMap(getResult.internalSourceRef(), true); final XContentType updateSourceContentType = sourceAndContent.v1(); final Map updatedSourceAsMap = sourceAndContent.v2(); - final boolean noop = XContentHelper.update(updatedSourceAsMap, currentRequest.sourceAsMap(), detectNoop) == false; + final boolean noop = XContentHelper.update( + updatedSourceAsMap, + currentRequest.sourceAsMap(documentSizeObserver), + detectNoop + ) == false; // We can only actually turn the update into a noop if detectNoop is true to preserve backwards compatibility and to handle cases // where users repopulating multi-fields or adding synonyms, etc. @@ -216,7 +225,8 @@ static Result prepareUpdateIndexRequest(ShardId shardId, UpdateRequest request, .setIfPrimaryTerm(getResult.getPrimaryTerm()) .waitForActiveShards(request.waitForActiveShards()) .timeout(request.timeout()) - .setRefreshPolicy(request.getRefreshPolicy()); + .setRefreshPolicy(request.getRefreshPolicy()) + .setNormalisedBytesParsed(documentSizeObserver.normalisedBytesParsed()); return new Result(finalIndexRequest, DocWriteResponse.Result.UPDATED, updatedSourceAsMap, updateSourceContentType); } } @@ -258,7 +268,8 @@ Result prepareUpdateScriptRequest(ShardId shardId, UpdateRequest request, GetRes .setIfPrimaryTerm(getResult.getPrimaryTerm()) .waitForActiveShards(request.waitForActiveShards()) .timeout(request.timeout()) - .setRefreshPolicy(request.getRefreshPolicy()); + .setRefreshPolicy(request.getRefreshPolicy()) + .noParsedBytesToReport(); return new Result(indexRequest, DocWriteResponse.Result.UPDATED, updatedSourceAsMap, updateSourceContentType); } case DELETE -> { diff --git a/server/src/main/java/org/elasticsearch/node/NodeConstruction.java b/server/src/main/java/org/elasticsearch/node/NodeConstruction.java index 72345b7d860f6..9323ec63c0d2d 100644 --- a/server/src/main/java/org/elasticsearch/node/NodeConstruction.java +++ b/server/src/main/java/org/elasticsearch/node/NodeConstruction.java @@ -256,9 +256,12 @@ static NodeConstruction prepareConstruction( SearchModule searchModule = constructor.createSearchModule(settingsModule.getSettings(), threadPool); constructor.createClientAndRegistries(settingsModule.getSettings(), threadPool, searchModule); + DocumentParsingProvider documentParsingProvider = constructor.getDocumentParsingProvider(); ScriptService scriptService = constructor.createScriptService(settingsModule, threadPool, serviceProvider); + constructor.createUpdateHelper(documentParsingProvider, scriptService); + constructor.construct( threadPool, settingsModule, @@ -267,7 +270,8 @@ static NodeConstruction prepareConstruction( constructor.createAnalysisRegistry(), serviceProvider, forbidPrivateIndexSettings, - telemetryProvider + telemetryProvider, + documentParsingProvider ); return constructor; @@ -570,14 +574,18 @@ private ScriptService createScriptService(SettingsModule settingsModule, ThreadP threadPool::absoluteTimeInMillis ); ScriptModule.registerClusterSettingsListeners(scriptService, settingsModule.getClusterSettings()); - modules.add(b -> { - b.bind(ScriptService.class).toInstance(scriptService); - b.bind(UpdateHelper.class).toInstance(new UpdateHelper(scriptService)); - }); + modules.add(b -> { b.bind(ScriptService.class).toInstance(scriptService); }); return scriptService; } + private UpdateHelper createUpdateHelper(DocumentParsingProvider documentParsingProvider, ScriptService scriptService) { + UpdateHelper updateHelper = new UpdateHelper(scriptService, documentParsingProvider); + + modules.add(b -> { b.bind(UpdateHelper.class).toInstance(new UpdateHelper(scriptService, documentParsingProvider)); }); + return updateHelper; + } + private AnalysisRegistry createAnalysisRegistry() throws IOException { AnalysisRegistry registry = new AnalysisModule( environment, @@ -596,7 +604,8 @@ private void construct( AnalysisRegistry analysisRegistry, NodeServiceProvider serviceProvider, boolean forbidPrivateIndexSettings, - TelemetryProvider telemetryProvider + TelemetryProvider telemetryProvider, + DocumentParsingProvider documentParsingProvider ) throws IOException { Settings settings = settingsModule.getSettings(); @@ -612,12 +621,10 @@ private void construct( ).collect(Collectors.toSet()), telemetryProvider.getTracer() ); - final Tracer tracer = telemetryProvider.getTracer(); ClusterService clusterService = createClusterService(settingsModule, threadPool, taskManager); clusterService.addStateApplier(scriptService); - DocumentParsingProvider documentParsingProvider = getDocumentParsingSupplier(); modules.bindToInstance(DocumentParsingProvider.class, documentParsingProvider); final IngestService ingestService = new IngestService( @@ -1298,8 +1305,8 @@ private void postInjection( logger.info("initialized"); } - private DocumentParsingProvider getDocumentParsingSupplier() { - return getSinglePlugin(DocumentParsingProviderPlugin.class).map(DocumentParsingProviderPlugin::getDocumentParsingSupplier) + private DocumentParsingProvider getDocumentParsingProvider() { + return getSinglePlugin(DocumentParsingProviderPlugin.class).map(DocumentParsingProviderPlugin::getDocumentParsingProvider) .orElse(DocumentParsingProvider.EMPTY_INSTANCE); } diff --git a/server/src/main/java/org/elasticsearch/plugins/internal/DocumentParsingProviderPlugin.java b/server/src/main/java/org/elasticsearch/plugins/internal/DocumentParsingProviderPlugin.java index 547ef5c57807b..6f0dc00e1ee23 100644 --- a/server/src/main/java/org/elasticsearch/plugins/internal/DocumentParsingProviderPlugin.java +++ b/server/src/main/java/org/elasticsearch/plugins/internal/DocumentParsingProviderPlugin.java @@ -9,12 +9,12 @@ package org.elasticsearch.plugins.internal; /** - * An internal plugin that will return a supplier of DocumentParsingSupplier. + * An internal plugin that will return a DocumentParsingProvider. */ public interface DocumentParsingProviderPlugin { /** - * @return a DocumentParsingSupplier to create instances of observer and reporter of parsing events + * @return a DocumentParsingProvider to create instances of observer and reporter of parsing events */ - DocumentParsingProvider getDocumentParsingSupplier(); + DocumentParsingProvider getDocumentParsingProvider(); } diff --git a/server/src/test/java/org/elasticsearch/action/update/UpdateRequestTests.java b/server/src/test/java/org/elasticsearch/action/update/UpdateRequestTests.java index 7ee4d2d6bba9b..fa136d22cbd0b 100644 --- a/server/src/test/java/org/elasticsearch/action/update/UpdateRequestTests.java +++ b/server/src/test/java/org/elasticsearch/action/update/UpdateRequestTests.java @@ -22,6 +22,7 @@ import org.elasticsearch.env.Environment; import org.elasticsearch.index.get.GetResult; import org.elasticsearch.index.shard.ShardId; +import org.elasticsearch.plugins.internal.DocumentParsingProvider; import org.elasticsearch.script.MockScriptEngine; import org.elasticsearch.script.Script; import org.elasticsearch.script.ScriptEngine; @@ -59,6 +60,7 @@ import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.instanceOf; import static org.hamcrest.Matchers.notNullValue; +import static org.mockito.Mockito.mock; public class UpdateRequestTests extends ESTestCase { @@ -114,7 +116,7 @@ public void setUp() throws Exception { final MockScriptEngine engine = new MockScriptEngine("mock", scripts, Collections.emptyMap()); Map engines = Collections.singletonMap(engine.getType(), engine); ScriptService scriptService = new ScriptService(baseSettings, engines, ScriptModule.CORE_CONTEXTS, () -> 1L); - updateHelper = new UpdateHelper(scriptService); + updateHelper = new UpdateHelper(scriptService, DocumentParsingProvider.EMPTY_INSTANCE); } @SuppressWarnings("unchecked") @@ -590,14 +592,14 @@ public void testNoopDetection() throws Exception { try (var parser = createParser(JsonXContent.jsonXContent, new BytesArray("{\"doc\": {\"body\": \"foo\"}}"))) { request = new UpdateRequest("test", "1").fromXContent(parser); } - - UpdateHelper.Result result = UpdateHelper.prepareUpdateIndexRequest(shardId, request, getResult, true); + UpdateHelper updateHelper = new UpdateHelper(mock(ScriptService.class), DocumentParsingProvider.EMPTY_INSTANCE); + UpdateHelper.Result result = updateHelper.prepareUpdateIndexRequest(shardId, request, getResult, true); assertThat(result.action(), instanceOf(UpdateResponse.class)); assertThat(result.getResponseResult(), equalTo(DocWriteResponse.Result.NOOP)); // Try again, with detectNoop turned off - result = UpdateHelper.prepareUpdateIndexRequest(shardId, request, getResult, false); + result = updateHelper.prepareUpdateIndexRequest(shardId, request, getResult, false); assertThat(result.action(), instanceOf(IndexRequest.class)); assertThat(result.getResponseResult(), equalTo(DocWriteResponse.Result.UPDATED)); assertThat(result.updatedSourceAsMap().get("body").toString(), equalTo("foo")); @@ -605,7 +607,7 @@ public void testNoopDetection() throws Exception { try (var parser = createParser(JsonXContent.jsonXContent, new BytesArray("{\"doc\": {\"body\": \"bar\"}}"))) { // Change the request to be a different doc request = new UpdateRequest("test", "1").fromXContent(parser); - result = UpdateHelper.prepareUpdateIndexRequest(shardId, request, getResult, true); + result = updateHelper.prepareUpdateIndexRequest(shardId, request, getResult, true); assertThat(result.action(), instanceOf(IndexRequest.class)); assertThat(result.getResponseResult(), equalTo(DocWriteResponse.Result.UPDATED)); diff --git a/server/src/test/java/org/elasticsearch/snapshots/SnapshotResiliencyTests.java b/server/src/test/java/org/elasticsearch/snapshots/SnapshotResiliencyTests.java index 96c5075dd3ca7..54c97ea8dc1a3 100644 --- a/server/src/test/java/org/elasticsearch/snapshots/SnapshotResiliencyTests.java +++ b/server/src/test/java/org/elasticsearch/snapshots/SnapshotResiliencyTests.java @@ -2200,7 +2200,7 @@ protected void assertSnapshotOrGenericThread() { threadPool, shardStateAction, mappingUpdatedAction, - new UpdateHelper(scriptService), + new UpdateHelper(scriptService, DocumentParsingProvider.EMPTY_INSTANCE), actionFilters, indexingMemoryLimits, EmptySystemIndices.INSTANCE, diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/AuthorizationServiceTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/AuthorizationServiceTests.java index 74ddd4a90b0ec..8a0041ef2bb76 100644 --- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/AuthorizationServiceTests.java +++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/AuthorizationServiceTests.java @@ -112,6 +112,7 @@ import org.elasticsearch.indices.TestIndexNameExpressionResolver; import org.elasticsearch.license.MockLicenseState; import org.elasticsearch.license.XPackLicenseState; +import org.elasticsearch.plugins.internal.DocumentParsingProvider; import org.elasticsearch.script.ScriptService; import org.elasticsearch.search.SearchPhaseResult; import org.elasticsearch.search.SearchShardTarget; @@ -1576,7 +1577,7 @@ public void testDenialErrorMessagesForBulkIngest() throws Exception { TransportShardBulkAction.performOnPrimary( request, indexShard, - new UpdateHelper(mock(ScriptService.class)), + new UpdateHelper(mock(ScriptService.class), DocumentParsingProvider.EMPTY_INSTANCE), System::currentTimeMillis, mappingUpdater, waitForMappingUpdate, From bdf32e34c990e6f440bcea807f2d71e2facfa7f2 Mon Sep 17 00:00:00 2001 From: Ignacio Vera Date: Mon, 19 Feb 2024 14:28:11 +0100 Subject: [PATCH 31/45] Reduce InternalSignificantTerms in a streaming fashion (#105481) --- .../terms/InternalSignificantTerms.java | 83 +++++++++---------- 1 file changed, 39 insertions(+), 44 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalSignificantTerms.java b/server/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalSignificantTerms.java index 43c8d492ed870..a84b0e369e223 100644 --- a/server/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalSignificantTerms.java +++ b/server/src/main/java/org/elasticsearch/search/aggregations/bucket/terms/InternalSignificantTerms.java @@ -9,6 +9,7 @@ import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; +import org.elasticsearch.core.Releasables; import org.elasticsearch.search.DocValueFormat; import org.elasticsearch.search.aggregations.AggregationReduceContext; import org.elasticsearch.search.aggregations.Aggregator; @@ -16,12 +17,12 @@ import org.elasticsearch.search.aggregations.InternalAggregation; import org.elasticsearch.search.aggregations.InternalAggregations; import org.elasticsearch.search.aggregations.InternalMultiBucketAggregation; +import org.elasticsearch.search.aggregations.bucket.MultiBucketAggregatorsReducer; import org.elasticsearch.search.aggregations.bucket.terms.heuristic.SignificanceHeuristic; import org.elasticsearch.search.aggregations.support.SamplingContext; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; -import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; import java.util.List; @@ -201,50 +202,44 @@ protected AggregatorReducer getLeaderReducer(AggregationReduceContext reduceCont return new AggregatorReducer() { long globalSubsetSize = 0; long globalSupersetSize = 0; + final Map> buckets = new HashMap<>(); - final List> aggregations = new ArrayList<>(size); - - // Compute the overall result set size and the corpus size using the - // top-level Aggregations from each shard @Override public void accept(InternalAggregation aggregation) { @SuppressWarnings("unchecked") - InternalSignificantTerms terms = (InternalSignificantTerms) aggregation; + final InternalSignificantTerms terms = (InternalSignificantTerms) aggregation; + // Compute the overall result set size and the corpus size using the + // top-level Aggregations from each shard globalSubsetSize += terms.getSubsetSize(); globalSupersetSize += terms.getSupersetSize(); - aggregations.add(terms); + for (B bucket : terms.getBuckets()) { + final ReducerAndProto reducerAndProto = buckets.computeIfAbsent( + bucket.getKeyAsString(), + k -> new ReducerAndProto<>(new MultiBucketAggregatorsReducer(reduceContext, size), bucket) + ); + reducerAndProto.reducer.accept(bucket); + reducerAndProto.subsetDf[0] += bucket.subsetDf; + reducerAndProto.supersetDf[0] += bucket.supersetDf; + } } @Override public InternalAggregation get() { - final Map> buckets = new HashMap<>(); - for (InternalSignificantTerms terms : aggregations) { - for (B bucket : terms.getBuckets()) { - List existingBuckets = buckets.computeIfAbsent(bucket.getKeyAsString(), k -> new ArrayList<>(size)); - // Adjust the buckets with the global stats representing the - // total size of the pots from which the stats are drawn - existingBuckets.add( - createBucket( - bucket.getSubsetDf(), - globalSubsetSize, - bucket.getSupersetDf(), - globalSupersetSize, - bucket.aggregations, - bucket - ) - ); - } - } - final SignificanceHeuristic heuristic = getSignificanceHeuristic().rewrite(reduceContext); final int size = reduceContext.isFinalReduce() == false ? buckets.size() : Math.min(requiredSize, buckets.size()); final BucketSignificancePriorityQueue ordered = new BucketSignificancePriorityQueue<>(size); - for (Map.Entry> entry : buckets.entrySet()) { - List sameTermBuckets = entry.getValue(); - final B b = reduceBucket(sameTermBuckets, reduceContext); + for (ReducerAndProto reducerAndProto : buckets.values()) { + final B b = createBucket( + reducerAndProto.subsetDf[0], + globalSubsetSize, + reducerAndProto.supersetDf[0], + globalSupersetSize, + reducerAndProto.reducer.get(), + reducerAndProto.proto + ); b.updateScore(heuristic); if (((b.score > 0) && (b.subsetDf >= minDocCount)) || reduceContext.isFinalReduce() == false) { - B removed = ordered.insertWithOverflow(b); + final B removed = ordered.insertWithOverflow(b); if (removed == null) { reduceContext.consumeBucketsAndMaybeBreak(1); } else { @@ -254,15 +249,28 @@ public InternalAggregation get() { reduceContext.consumeBucketsAndMaybeBreak(-countInnerBucket(b)); } } - B[] list = createBucketsArray(ordered.size()); + final B[] list = createBucketsArray(ordered.size()); for (int i = ordered.size() - 1; i >= 0; i--) { list[i] = ordered.pop(); } return create(globalSubsetSize, globalSupersetSize, Arrays.asList(list)); } + + @Override + public void close() { + for (ReducerAndProto reducerAndProto : buckets.values()) { + Releasables.close(reducerAndProto.reducer); + } + } }; } + private record ReducerAndProto(MultiBucketAggregatorsReducer reducer, B proto, long[] subsetDf, long[] supersetDf) { + private ReducerAndProto(MultiBucketAggregatorsReducer reducer, B proto) { + this(reducer, proto, new long[] { 0 }, new long[] { 0 }); + } + } + @Override public InternalAggregation finalizeSampling(SamplingContext samplingContext) { long supersetSize = samplingContext.scaleUp(getSupersetSize()); @@ -285,19 +293,6 @@ public InternalAggregation finalizeSampling(SamplingContext samplingContext) { ); } - private B reduceBucket(List buckets, AggregationReduceContext context) { - assert buckets.isEmpty() == false; - long subsetDf = 0; - long supersetDf = 0; - for (B bucket : buckets) { - subsetDf += bucket.subsetDf; - supersetDf += bucket.supersetDf; - } - final List aggregations = new BucketAggregationList<>(buckets); - final InternalAggregations aggs = InternalAggregations.reduce(aggregations, context); - return createBucket(subsetDf, buckets.get(0).subsetSize, supersetDf, buckets.get(0).supersetSize, aggs, buckets.get(0)); - } - abstract B createBucket( long subsetDf, long subsetSize, From 3470565d332e8adf91da0e692d3d5ce286a91ff0 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 19 Feb 2024 15:42:17 +0200 Subject: [PATCH 32/45] Make the Get API keys API local only (#105497) The Get API Key transport action is always executed locally, which means that the transport requests and responses are never serialized over the wire between cluster nodes. This PR removes the serialization dead code. Relates: #100111 #104653 --- .../core/security/action/apikey/ApiKey.java | 78 +-------- .../action/apikey/GetApiKeyRequest.java | 44 +---- .../action/apikey/GetApiKeyResponse.java | 15 +- .../apikey/ApiKeySerializationTests.java | 81 --------- .../action/apikey/GetApiKeyRequestTests.java | 161 ++---------------- .../action/apikey/GetApiKeyResponseTests.java | 55 ------ .../apikey/TransportGetApiKeyAction.java | 7 +- 7 files changed, 20 insertions(+), 421 deletions(-) delete mode 100644 x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKeySerializationTests.java diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKey.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKey.java index a2df837448754..3ab487560f2b8 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKey.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKey.java @@ -7,11 +7,6 @@ package org.elasticsearch.xpack.core.security.action.apikey; -import org.elasticsearch.TransportVersion; -import org.elasticsearch.TransportVersions; -import org.elasticsearch.common.io.stream.StreamInput; -import org.elasticsearch.common.io.stream.StreamOutput; -import org.elasticsearch.common.io.stream.Writeable; import org.elasticsearch.common.xcontent.XContentParserUtils; import org.elasticsearch.core.Assertions; import org.elasticsearch.core.Nullable; @@ -44,9 +39,7 @@ /** * API key information */ -public final class ApiKey implements ToXContentObject, Writeable { - - public static final TransportVersion CROSS_CLUSTER_KEY_VERSION = TransportVersions.V_8_9_X; +public final class ApiKey implements ToXContentObject { public enum Type { /** @@ -164,47 +157,6 @@ private ApiKey( this.limitedBy = limitedBy; } - public ApiKey(StreamInput in) throws IOException { - if (in.getTransportVersion().onOrAfter(TransportVersions.V_7_5_0)) { - this.name = in.readOptionalString(); - } else { - this.name = in.readString(); - } - this.id = in.readString(); - if (in.getTransportVersion().onOrAfter(CROSS_CLUSTER_KEY_VERSION)) { - this.type = in.readEnum(Type.class); - } else { - // This default is safe because - // 1. ApiKey objects never transfer between nodes - // 2. Creating cross-cluster API keys mandates minimal node version that understands the API key type - this.type = Type.REST; - } - this.creation = in.readInstant(); - this.expiration = in.readOptionalInstant(); - this.invalidated = in.readBoolean(); - if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_12_0)) { - this.invalidation = in.readOptionalInstant(); - } else { - this.invalidation = null; - } - - this.username = in.readString(); - this.realm = in.readString(); - if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_0_0)) { - this.metadata = in.readGenericMap(); - } else { - this.metadata = Map.of(); - } - if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_5_0)) { - final List roleDescriptors = in.readOptionalCollectionAsList(RoleDescriptor::new); - this.roleDescriptors = roleDescriptors != null ? List.copyOf(roleDescriptors) : null; - this.limitedBy = in.readOptionalWriteable(RoleDescriptorsIntersection::new); - } else { - this.roleDescriptors = null; - this.limitedBy = null; - } - } - public String getId() { return id; } @@ -323,34 +275,6 @@ private void buildXContentForCrossClusterApiKeyAccess(XContentBuilder builder, R builder.endObject(); } - @Override - public void writeTo(StreamOutput out) throws IOException { - if (out.getTransportVersion().onOrAfter(TransportVersions.V_7_5_0)) { - out.writeOptionalString(name); - } else { - out.writeString(name); - } - out.writeString(id); - if (out.getTransportVersion().onOrAfter(CROSS_CLUSTER_KEY_VERSION)) { - out.writeEnum(type); - } - out.writeInstant(creation); - out.writeOptionalInstant(expiration); - out.writeBoolean(invalidated); - if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_12_0)) { - out.writeOptionalInstant(invalidation); - } - out.writeString(username); - out.writeString(realm); - if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_0_0)) { - out.writeGenericMap(metadata); - } - if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_5_0)) { - out.writeOptionalCollection(roleDescriptors); - out.writeOptionalWriteable(limitedBy); - } - } - @Override public int hashCode() { return Objects.hash( diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequest.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequest.java index a8b14795e2dd8..5e5773c4296ed 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequest.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequest.java @@ -7,12 +7,10 @@ package org.elasticsearch.xpack.core.security.action.apikey; -import org.elasticsearch.TransportVersion; -import org.elasticsearch.TransportVersions; import org.elasticsearch.action.ActionRequest; import org.elasticsearch.action.ActionRequestValidationException; +import org.elasticsearch.action.support.TransportAction; import org.elasticsearch.common.Strings; -import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.io.stream.StreamOutput; import org.elasticsearch.core.Nullable; @@ -26,8 +24,6 @@ */ public final class GetApiKeyRequest extends ActionRequest { - static TransportVersion API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION = TransportVersions.V_8_10_X; - private final String realmName; private final String userName; private final String apiKeyId; @@ -36,29 +32,6 @@ public final class GetApiKeyRequest extends ActionRequest { private final boolean withLimitedBy; private final boolean activeOnly; - public GetApiKeyRequest(StreamInput in) throws IOException { - super(in); - realmName = textOrNull(in.readOptionalString()); - userName = textOrNull(in.readOptionalString()); - apiKeyId = textOrNull(in.readOptionalString()); - apiKeyName = textOrNull(in.readOptionalString()); - if (in.getTransportVersion().onOrAfter(TransportVersions.V_7_4_0)) { - ownedByAuthenticatedUser = in.readOptionalBoolean(); - } else { - ownedByAuthenticatedUser = false; - } - if (in.getTransportVersion().onOrAfter(TransportVersions.V_8_5_0)) { - withLimitedBy = in.readBoolean(); - } else { - withLimitedBy = false; - } - if (in.getTransportVersion().onOrAfter(API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION)) { - activeOnly = in.readBoolean(); - } else { - activeOnly = false; - } - } - private GetApiKeyRequest( @Nullable String realmName, @Nullable String userName, @@ -136,20 +109,7 @@ public ActionRequestValidationException validate() { @Override public void writeTo(StreamOutput out) throws IOException { - super.writeTo(out); - out.writeOptionalString(realmName); - out.writeOptionalString(userName); - out.writeOptionalString(apiKeyId); - out.writeOptionalString(apiKeyName); - if (out.getTransportVersion().onOrAfter(TransportVersions.V_7_4_0)) { - out.writeOptionalBoolean(ownedByAuthenticatedUser); - } - if (out.getTransportVersion().onOrAfter(TransportVersions.V_8_5_0)) { - out.writeBoolean(withLimitedBy); - } - if (out.getTransportVersion().onOrAfter(API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION)) { - out.writeBoolean(activeOnly); - } + TransportAction.localOnly(); } @Override diff --git a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponse.java b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponse.java index dbe52fb23d056..6e484d5b04426 100644 --- a/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponse.java +++ b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponse.java @@ -8,9 +8,8 @@ package org.elasticsearch.xpack.core.security.action.apikey; import org.elasticsearch.action.ActionResponse; -import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.action.support.TransportAction; import org.elasticsearch.common.io.stream.StreamOutput; -import org.elasticsearch.common.io.stream.Writeable; import org.elasticsearch.xcontent.ConstructingObjectParser; import org.elasticsearch.xcontent.ParseField; import org.elasticsearch.xcontent.ToXContentObject; @@ -19,7 +18,6 @@ import java.io.IOException; import java.util.Collection; -import java.util.Collections; import java.util.List; import java.util.Objects; @@ -29,22 +27,17 @@ * Response for get API keys.
* The result contains information about the API keys that were found. */ -public final class GetApiKeyResponse extends ActionResponse implements ToXContentObject, Writeable { +public final class GetApiKeyResponse extends ActionResponse implements ToXContentObject { private final ApiKey[] foundApiKeysInfo; - public GetApiKeyResponse(StreamInput in) throws IOException { - super(in); - this.foundApiKeysInfo = in.readArray(ApiKey::new, ApiKey[]::new); - } - public GetApiKeyResponse(Collection foundApiKeysInfo) { Objects.requireNonNull(foundApiKeysInfo, "found_api_keys_info must be provided"); this.foundApiKeysInfo = foundApiKeysInfo.toArray(new ApiKey[0]); } public static GetApiKeyResponse emptyResponse() { - return new GetApiKeyResponse(Collections.emptyList()); + return new GetApiKeyResponse(List.of()); } public ApiKey[] getApiKeyInfos() { @@ -59,7 +52,7 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws @Override public void writeTo(StreamOutput out) throws IOException { - out.writeArray(foundApiKeysInfo); + TransportAction.localOnly(); } @SuppressWarnings("unchecked") diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKeySerializationTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKeySerializationTests.java deleted file mode 100644 index 8e6cdb2042811..0000000000000 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKeySerializationTests.java +++ /dev/null @@ -1,81 +0,0 @@ -/* - * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one - * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. - */ - -package org.elasticsearch.xpack.core.security.action.apikey; - -import org.elasticsearch.TransportVersions; -import org.elasticsearch.common.io.stream.NamedWriteableRegistry; -import org.elasticsearch.common.io.stream.Writeable; -import org.elasticsearch.test.AbstractWireSerializingTestCase; -import org.elasticsearch.xpack.core.XPackClientPlugin; - -import java.io.IOException; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -import static org.elasticsearch.xpack.core.security.action.apikey.ApiKeyTests.randomApiKeyInstance; -import static org.hamcrest.Matchers.nullValue; - -public class ApiKeySerializationTests extends AbstractWireSerializingTestCase { - - public void testSerializationBackwardsCompatibility() throws IOException { - ApiKey testInstance = createTestInstance(); - ApiKey deserializedInstance = copyInstance(testInstance, TransportVersions.V_8_11_X); - try { - // Transport is on a version before invalidation was introduced, so should always be null - assertThat(deserializedInstance.getInvalidation(), nullValue()); - } finally { - dispose(deserializedInstance); - } - } - - @Override - protected ApiKey createTestInstance() { - return randomApiKeyInstance(); - } - - @Override - protected ApiKey mutateInstance(ApiKey instance) throws IOException { - ApiKey copyOfInstance = copyInstance(instance); - // Metadata in test instance is mutable, so mutate it instead of the copy (immutable metadata) to make sure they differ - Object metadataNumberValue = instance.getMetadata().getOrDefault("number", Integer.toString(randomInt())); - instance.getMetadata().put("number", Integer.parseInt(metadataNumberValue.toString()) + randomInt()); - return copyOfInstance; - } - - @Override - protected Writeable.Reader instanceReader() { - return ApiKey::new; - } - - @Override - protected NamedWriteableRegistry getNamedWriteableRegistry() { - return new NamedWriteableRegistry(new XPackClientPlugin().getNamedWriteables()); - } - - public static Map randomMetadata() { - Map randomMetadata = randomFrom( - Map.of( - "application", - randomAlphaOfLength(5), - "number", - 1, - "numbers", - List.of(1, 3, 5), - "environment", - Map.of("os", "linux", "level", 42, "category", "trusted") - ), - Map.of(randomAlphaOfLengthBetween(3, 8), randomAlphaOfLengthBetween(3, 8)), - Map.of(), - null - ); - - // Make metadata mutable for testing purposes - return randomMetadata == null ? new HashMap<>() : new HashMap<>(randomMetadata); - } -} diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequestTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequestTests.java index 50369cadaf365..c37e95d4934fc 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequestTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyRequestTests.java @@ -7,26 +7,13 @@ package org.elasticsearch.xpack.core.security.action.apikey; -import org.elasticsearch.TransportVersion; -import org.elasticsearch.TransportVersions; -import org.elasticsearch.action.ActionRequest; import org.elasticsearch.action.ActionRequestValidationException; -import org.elasticsearch.common.io.stream.InputStreamStreamInput; -import org.elasticsearch.common.io.stream.OutputStreamStreamOutput; -import org.elasticsearch.common.io.stream.StreamOutput; import org.elasticsearch.test.ESTestCase; -import org.elasticsearch.test.TransportVersionUtils; -import java.io.ByteArrayInputStream; -import java.io.ByteArrayOutputStream; import java.io.IOException; import java.util.function.Supplier; -import static org.elasticsearch.test.TransportVersionUtils.randomVersionBetween; -import static org.elasticsearch.xpack.core.security.action.apikey.GetApiKeyRequest.API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION; import static org.hamcrest.Matchers.containsInAnyOrder; -import static org.hamcrest.Matchers.equalTo; -import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.nullValue; public class GetApiKeyRequestTests extends ESTestCase { @@ -57,39 +44,6 @@ public void testRequestValidation() { } public void testRequestValidationFailureScenarios() throws IOException { - class Dummy extends ActionRequest { - String realm; - String user; - String apiKeyId; - String apiKeyName; - boolean ownedByAuthenticatedUser; - - Dummy(String[] a) { - realm = a[0]; - user = a[1]; - apiKeyId = a[2]; - apiKeyName = a[3]; - ownedByAuthenticatedUser = Boolean.parseBoolean(a[4]); - } - - @Override - public ActionRequestValidationException validate() { - return null; - } - - @Override - public void writeTo(StreamOutput out) throws IOException { - super.writeTo(out); - out.writeOptionalString(realm); - out.writeOptionalString(user); - out.writeOptionalString(apiKeyId); - out.writeOptionalString(apiKeyName); - out.writeOptionalBoolean(ownedByAuthenticatedUser); - out.writeBoolean(randomBoolean()); - out.writeBoolean(randomBoolean()); - } - } - String[][] inputs = new String[][] { { randomNullOrEmptyString(), "user", "api-kid", "api-kname", "false" }, { "realm", randomNullOrEmptyString(), "api-kid", "api-kname", "false" }, @@ -110,112 +64,17 @@ public void writeTo(StreamOutput out) throws IOException { { "neither username nor realm-name may be specified when retrieving owned API keys" } }; for (int caseNo = 0; caseNo < inputs.length; caseNo++) { - try ( - ByteArrayOutputStream bos = new ByteArrayOutputStream(); - OutputStreamStreamOutput osso = new OutputStreamStreamOutput(bos) - ) { - Dummy d = new Dummy(inputs[caseNo]); - d.writeTo(osso); - - ByteArrayInputStream bis = new ByteArrayInputStream(bos.toByteArray()); - InputStreamStreamInput issi = new InputStreamStreamInput(bis); - - GetApiKeyRequest request = new GetApiKeyRequest(issi); - ActionRequestValidationException ve = request.validate(); - assertNotNull(ve); - assertEquals(expectedErrorMessages[caseNo].length, ve.validationErrors().size()); - assertThat(ve.validationErrors(), containsInAnyOrder(expectedErrorMessages[caseNo])); - } - } - } - - public void testSerialization() throws IOException { - final String apiKeyId = randomAlphaOfLength(5); - { - final GetApiKeyRequest getApiKeyRequest = GetApiKeyRequest.builder().ownedByAuthenticatedUser(true).apiKeyId(apiKeyId).build(); - ByteArrayOutputStream outBuffer = new ByteArrayOutputStream(); - OutputStreamStreamOutput out = new OutputStreamStreamOutput(outBuffer); - out.setTransportVersion(randomVersionBetween(random(), TransportVersions.V_7_0_0, TransportVersions.V_7_3_0)); - getApiKeyRequest.writeTo(out); - - InputStreamStreamInput inputStreamStreamInput = new InputStreamStreamInput(new ByteArrayInputStream(outBuffer.toByteArray())); - inputStreamStreamInput.setTransportVersion( - randomVersionBetween(random(), TransportVersions.V_7_0_0, TransportVersions.V_7_3_0) - ); - GetApiKeyRequest requestFromInputStream = new GetApiKeyRequest(inputStreamStreamInput); - - assertThat(requestFromInputStream.getApiKeyId(), equalTo(getApiKeyRequest.getApiKeyId())); - // old version so the default for `ownedByAuthenticatedUser` is false - assertThat(requestFromInputStream.ownedByAuthenticatedUser(), is(false)); - } - { - final GetApiKeyRequest getApiKeyRequest = GetApiKeyRequest.builder() - .apiKeyId(apiKeyId) - .ownedByAuthenticatedUser(true) - .withLimitedBy(randomBoolean()) - .activeOnly(randomBoolean()) - .build(); - ByteArrayOutputStream outBuffer = new ByteArrayOutputStream(); - OutputStreamStreamOutput out = new OutputStreamStreamOutput(outBuffer); - out.setTransportVersion(randomVersionBetween(random(), TransportVersions.V_7_4_0, TransportVersions.V_8_4_0)); - getApiKeyRequest.writeTo(out); - - InputStreamStreamInput inputStreamStreamInput = new InputStreamStreamInput(new ByteArrayInputStream(outBuffer.toByteArray())); - inputStreamStreamInput.setTransportVersion( - randomVersionBetween(random(), TransportVersions.V_7_4_0, TransportVersions.V_8_4_0) - ); - GetApiKeyRequest requestFromInputStream = new GetApiKeyRequest(inputStreamStreamInput); - - assertThat(requestFromInputStream.getApiKeyId(), equalTo(getApiKeyRequest.getApiKeyId())); - assertThat(requestFromInputStream.ownedByAuthenticatedUser(), is(true)); - // old version so the default for `withLimitedBy` is false - assertThat(requestFromInputStream.withLimitedBy(), is(false)); - // old version so the default for `activeOnly` is false - assertThat(requestFromInputStream.activeOnly(), is(false)); - } - { - final GetApiKeyRequest getApiKeyRequest = GetApiKeyRequest.builder() - .apiKeyId(apiKeyId) - .ownedByAuthenticatedUser(randomBoolean()) - .withLimitedBy(randomBoolean()) - .activeOnly(randomBoolean()) + GetApiKeyRequest request = GetApiKeyRequest.builder() + .realmName(inputs[caseNo][0]) + .userName(inputs[caseNo][1]) + .apiKeyId(inputs[caseNo][2]) + .apiKeyName(inputs[caseNo][3]) + .ownedByAuthenticatedUser(Boolean.parseBoolean(inputs[caseNo][4])) .build(); - ByteArrayOutputStream outBuffer = new ByteArrayOutputStream(); - OutputStreamStreamOutput out = new OutputStreamStreamOutput(outBuffer); - TransportVersion beforeActiveOnly = TransportVersionUtils.getPreviousVersion(API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION); - out.setTransportVersion(randomVersionBetween(random(), TransportVersions.V_8_5_0, beforeActiveOnly)); - getApiKeyRequest.writeTo(out); - - InputStreamStreamInput inputStreamStreamInput = new InputStreamStreamInput(new ByteArrayInputStream(outBuffer.toByteArray())); - inputStreamStreamInput.setTransportVersion(randomVersionBetween(random(), TransportVersions.V_8_5_0, beforeActiveOnly)); - GetApiKeyRequest requestFromInputStream = new GetApiKeyRequest(inputStreamStreamInput); - - assertThat(requestFromInputStream.getApiKeyId(), equalTo(getApiKeyRequest.getApiKeyId())); - assertThat(requestFromInputStream.ownedByAuthenticatedUser(), is(getApiKeyRequest.ownedByAuthenticatedUser())); - assertThat(requestFromInputStream.withLimitedBy(), is(getApiKeyRequest.withLimitedBy())); - // old version so the default for `activeOnly` is false - assertThat(requestFromInputStream.activeOnly(), is(false)); - } - { - final GetApiKeyRequest getApiKeyRequest = GetApiKeyRequest.builder() - .apiKeyId(apiKeyId) - .withLimitedBy(randomBoolean()) - .activeOnly(randomBoolean()) - .build(); - ByteArrayOutputStream outBuffer = new ByteArrayOutputStream(); - OutputStreamStreamOutput out = new OutputStreamStreamOutput(outBuffer); - out.setTransportVersion( - randomVersionBetween(random(), API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION, TransportVersion.current()) - ); - getApiKeyRequest.writeTo(out); - - InputStreamStreamInput inputStreamStreamInput = new InputStreamStreamInput(new ByteArrayInputStream(outBuffer.toByteArray())); - inputStreamStreamInput.setTransportVersion( - randomVersionBetween(random(), API_KEY_ACTIVE_ONLY_PARAM_TRANSPORT_VERSION, TransportVersion.current()) - ); - GetApiKeyRequest requestFromInputStream = new GetApiKeyRequest(inputStreamStreamInput); - - assertThat(requestFromInputStream, equalTo(getApiKeyRequest)); + ActionRequestValidationException ve = request.validate(); + assertNotNull(ve); + assertEquals(expectedErrorMessages[caseNo].length, ve.validationErrors().size()); + assertThat(ve.validationErrors(), containsInAnyOrder(expectedErrorMessages[caseNo])); } } diff --git a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponseTests.java b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponseTests.java index a32ed8f53f5b2..0b287f2fb6329 100644 --- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponseTests.java +++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/action/apikey/GetApiKeyResponseTests.java @@ -8,23 +8,16 @@ package org.elasticsearch.xpack.core.security.action.apikey; import org.elasticsearch.common.Strings; -import org.elasticsearch.common.io.stream.BytesStreamOutput; -import org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput; -import org.elasticsearch.common.io.stream.NamedWriteableRegistry; -import org.elasticsearch.common.io.stream.StreamInput; import org.elasticsearch.common.xcontent.XContentHelper; import org.elasticsearch.test.ESTestCase; import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xpack.core.security.authz.RoleDescriptor; -import org.elasticsearch.xpack.core.security.authz.privilege.ConfigurableClusterPrivilege; -import org.elasticsearch.xpack.core.security.authz.privilege.ConfigurableClusterPrivileges; import java.io.IOException; import java.time.Instant; import java.util.Arrays; -import java.util.Collections; import java.util.List; import java.util.Map; @@ -32,58 +25,10 @@ import static org.elasticsearch.xpack.core.security.action.apikey.CrossClusterApiKeyRoleDescriptorBuilder.CCS_AND_CCR_CLUSTER_PRIVILEGE_NAMES; import static org.elasticsearch.xpack.core.security.action.apikey.CrossClusterApiKeyRoleDescriptorBuilder.CCS_INDICES_PRIVILEGE_NAMES; import static org.elasticsearch.xpack.core.security.action.apikey.CrossClusterApiKeyRoleDescriptorBuilder.ROLE_DESCRIPTOR_NAME; -import static org.elasticsearch.xpack.core.security.authz.RoleDescriptorTests.randomCrossClusterAccessRoleDescriptor; -import static org.elasticsearch.xpack.core.security.authz.RoleDescriptorTests.randomUniquelyNamedRoleDescriptors; import static org.hamcrest.Matchers.equalTo; public class GetApiKeyResponseTests extends ESTestCase { - public void testSerialization() throws IOException { - boolean withApiKeyName = randomBoolean(); - boolean withExpiration = randomBoolean(); - final ApiKey.Type type = randomFrom(ApiKey.Type.values()); - ApiKey apiKeyInfo = createApiKeyInfo( - (withApiKeyName) ? randomAlphaOfLength(4) : null, - randomAlphaOfLength(5), - type, - Instant.now(), - (withExpiration) ? Instant.now() : null, - false, - null, - randomAlphaOfLength(4), - randomAlphaOfLength(5), - randomBoolean() ? null : Map.of(randomAlphaOfLengthBetween(3, 8), randomAlphaOfLengthBetween(3, 8)), - type == ApiKey.Type.CROSS_CLUSTER - ? List.of(randomCrossClusterAccessRoleDescriptor()) - : randomFrom(randomUniquelyNamedRoleDescriptors(0, 3), null), - type == ApiKey.Type.CROSS_CLUSTER ? null : randomUniquelyNamedRoleDescriptors(1, 3) - ); - GetApiKeyResponse response = new GetApiKeyResponse(Collections.singletonList(apiKeyInfo)); - - final NamedWriteableRegistry namedWriteableRegistry = new NamedWriteableRegistry( - List.of( - new NamedWriteableRegistry.Entry( - ConfigurableClusterPrivilege.class, - ConfigurableClusterPrivileges.ManageApplicationPrivileges.WRITEABLE_NAME, - ConfigurableClusterPrivileges.ManageApplicationPrivileges::createFrom - ), - new NamedWriteableRegistry.Entry( - ConfigurableClusterPrivilege.class, - ConfigurableClusterPrivileges.WriteProfileDataPrivileges.WRITEABLE_NAME, - ConfigurableClusterPrivileges.WriteProfileDataPrivileges::createFrom - ) - ) - ); - - try (BytesStreamOutput output = new BytesStreamOutput()) { - response.writeTo(output); - try (StreamInput input = new NamedWriteableAwareStreamInput(output.bytes().streamInput(), namedWriteableRegistry)) { - GetApiKeyResponse serialized = new GetApiKeyResponse(input); - assertThat(serialized.getApiKeyInfos(), equalTo(response.getApiKeyInfos())); - } - } - } - public void testToXContent() throws IOException { final List roleDescriptors = List.of( new RoleDescriptor( diff --git a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/apikey/TransportGetApiKeyAction.java b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/apikey/TransportGetApiKeyAction.java index 2e12d230bcc88..4e79a40af7ec4 100644 --- a/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/apikey/TransportGetApiKeyAction.java +++ b/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/action/apikey/TransportGetApiKeyAction.java @@ -9,10 +9,9 @@ import org.elasticsearch.action.ActionListener; import org.elasticsearch.action.support.ActionFilters; -import org.elasticsearch.action.support.HandledTransportAction; +import org.elasticsearch.action.support.TransportAction; import org.elasticsearch.common.Strings; import org.elasticsearch.common.inject.Inject; -import org.elasticsearch.common.util.concurrent.EsExecutors; import org.elasticsearch.tasks.Task; import org.elasticsearch.transport.TransportService; import org.elasticsearch.xpack.core.security.SecurityContext; @@ -22,7 +21,7 @@ import org.elasticsearch.xpack.core.security.authc.Authentication; import org.elasticsearch.xpack.security.authc.ApiKeyService; -public final class TransportGetApiKeyAction extends HandledTransportAction { +public final class TransportGetApiKeyAction extends TransportAction { private final ApiKeyService apiKeyService; private final SecurityContext securityContext; @@ -34,7 +33,7 @@ public TransportGetApiKeyAction( ApiKeyService apiKeyService, SecurityContext context ) { - super(GetApiKeyAction.NAME, transportService, actionFilters, GetApiKeyRequest::new, EsExecutors.DIRECT_EXECUTOR_SERVICE); + super(GetApiKeyAction.NAME, actionFilters, transportService.getTaskManager()); this.apiKeyService = apiKeyService; this.securityContext = context; } From 9a0eee603bacd6cd98c40fa11e3762a6a85f72f0 Mon Sep 17 00:00:00 2001 From: Niels Bauman <33722607+nielsbauman@users.noreply.github.com> Date: Mon, 19 Feb 2024 15:51:32 +0100 Subject: [PATCH 33/45] Add missing repository integrity docs for Health API (#105555) Follow up on #104614 --- docs/reference/health/health.asciidoc | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/reference/health/health.asciidoc b/docs/reference/health/health.asciidoc index 4baed1fba5edd..9c62bca8b5f10 100644 --- a/docs/reference/health/health.asciidoc +++ b/docs/reference/health/health.asciidoc @@ -73,7 +73,7 @@ for health status set `verbose` to `false` to disable the more expensive analysi `repository_integrity`:: Tracks repository integrity and reports health issues - that arise if repositories become corrupted. + that arise if repositories become corrupted, unknown, or invalid. `slm`:: Reports health issues related to @@ -356,6 +356,14 @@ watermark threshold>>. (Optional, array of strings) If corrupted repositories have been detected in the system, the names of up to ten of them are displayed in this field. If no corrupted repositories are found, this detail is omitted. +`unknown_repositories`:: + (Optional, int) The number of repositories that have been determined to be unknown by at least one node. + If there are no unknown repositories detected, this detail is omitted. + +`invalid_repositories`:: + (Optional, int) The number of repositories that have been determined to be invalid by at least one node. + If there are no invalid repositories detected, this detail is omitted. + [[health-api-response-details-ilm]] ===== ilm From 9e5fe197ca9d35ca9bce99a0fc3b78691388bf27 Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Mon, 19 Feb 2024 16:25:31 +0100 Subject: [PATCH 34/45] [DOCS] Fix sublist syntax (#105625) --- .../search/search-your-data/knn-search.asciidoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/reference/search/search-your-data/knn-search.asciidoc b/docs/reference/search/search-your-data/knn-search.asciidoc index a68cacec8c10c..ab65b834c0ce7 100644 --- a/docs/reference/search/search-your-data/knn-search.asciidoc +++ b/docs/reference/search/search-your-data/knn-search.asciidoc @@ -568,12 +568,12 @@ NOTE: `similarity` is the true <> before it For each configured <>, here is the corresponding inverted `_score` function. This is so if you are wanting to filter from a `_score` perspective, you can do this minor transformation to correctly reject irrelevant results. -- - - `l2_norm`: `sqrt((1 / _score) - 1)` - - `cosine`: `(2 * _score) - 1` - - `dot_product`: `(2 * _score) - 1` - - `max_inner_product`: - - `_score < 1`: `1 - (1 / _score)` - - `_score >= 1`: `_score - 1` +* `l2_norm`: `sqrt((1 / _score) - 1)` +* `cosine`: `(2 * _score) - 1` +* `dot_product`: `(2 * _score) - 1` +* `max_inner_product`: +** `_score < 1`: `1 - (1 / _score)` +** `_score >= 1`: `_score - 1` -- Here is an example. In this example we search for the given `query_vector` for `k` nearest neighbors. However, with From eeff6184519e5479eebacabfaa23cb7bbc01dcdc Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Mon, 19 Feb 2024 21:50:16 +0100 Subject: [PATCH 35/45] Save allocations and copying in TimeSeriesIdFieldMapper#buildTsidHash (#105582) No point in copying the bytes multiple times here. Just presize the array correctly (at most wasting a single byte) and serialize into it. Saving a couple GB of allocations during the TSDB rally track indexing step. --- .../common/io/stream/StreamOutput.java | 2 +- .../index/mapper/TimeSeriesIdFieldMapper.java | 25 +++++++++++++------ 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java b/server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java index b67879510b108..69a5135215eba 100644 --- a/server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java +++ b/server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java @@ -216,7 +216,7 @@ public void writeVInt(int i) throws IOException { writeBytes(buffer, 0, index); } - private static int putVInt(byte[] buffer, int i, int off) { + public static int putVInt(byte[] buffer, int i, int off) { if (Integer.numberOfLeadingZeros(i) >= 25) { buffer[off] = (byte) i; return 1; diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java index fb26debab2acc..1ee7caff497ad 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TimeSeriesIdFieldMapper.java @@ -19,6 +19,7 @@ import org.elasticsearch.common.hash.MurmurHash3; import org.elasticsearch.common.io.stream.BytesStreamOutput; import org.elasticsearch.common.io.stream.StreamInput; +import org.elasticsearch.common.io.stream.StreamOutput; import org.elasticsearch.common.network.NetworkAddress; import org.elasticsearch.common.util.ByteUtils; import org.elasticsearch.core.Nullable; @@ -208,6 +209,12 @@ public BytesReference buildLegacyTsid() throws IOException { } } + private static final int MAX_HASH_LEN_BYTES = 2; + + static { + assert MAX_HASH_LEN_BYTES == StreamOutput.putVInt(new byte[2], tsidHashLen(MAX_DIMENSIONS), 0); + } + /** * Here we build the hash of the tsid using a similarity function so that we have a result * with the following pattern: @@ -219,11 +226,13 @@ public BytesReference buildLegacyTsid() throws IOException { * The idea is to be able to place 'similar' time series close to each other. Two time series * are considered 'similar' if they share the same dimensions (names and values). */ - public BytesReference buildTsidHash() throws IOException { + public BytesReference buildTsidHash() { // NOTE: hash all dimension field names int numberOfDimensions = Math.min(MAX_DIMENSIONS, dimensions.size()); - int tsidHashIndex = 0; - byte[] tsidHash = new byte[16 + 16 + 4 * numberOfDimensions]; + int len = tsidHashLen(numberOfDimensions); + // either one or two bytes are occupied by the vint since we're bounded by #MAX_DIMENSIONS + byte[] tsidHash = new byte[MAX_HASH_LEN_BYTES + len]; + int tsidHashIndex = StreamOutput.putVInt(tsidHash, len, 0); tsidHasher.reset(); for (final Dimension dimension : dimensions) { @@ -258,11 +267,11 @@ public BytesReference buildTsidHash() throws IOException { } tsidHashIndex = writeHash128(tsidHasher.digestHash(), tsidHash, tsidHashIndex); - assert tsidHashIndex == tsidHash.length; - try (BytesStreamOutput out = new BytesStreamOutput(tsidHash.length)) { - out.writeBytesRef(new BytesRef(tsidHash, 0, tsidHash.length)); - return out.bytes(); - } + return new BytesArray(tsidHash, 0, tsidHashIndex); + } + + private static int tsidHashLen(int numberOfDimensions) { + return 16 + 16 + 4 * numberOfDimensions; } private int writeHash128(final MurmurHash3.Hash128 hash128, byte[] buffer, int tsidHashIndex) { From 67e36f176c7c1802be113b1287e81b9dfce63fef Mon Sep 17 00:00:00 2001 From: Armin Braun Date: Mon, 19 Feb 2024 21:52:54 +0100 Subject: [PATCH 36/45] Speedup deserialization in InternalAggregations (#105596) We can use the faster list reader here because the category is a constant. --- .../elasticsearch/search/aggregations/InternalAggregations.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java b/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java index 9467c5b40e83e..b65f6b01de348 100644 --- a/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java +++ b/server/src/main/java/org/elasticsearch/search/aggregations/InternalAggregations.java @@ -165,7 +165,7 @@ public static InternalAggregations from(List aggregations) } public static InternalAggregations readFrom(StreamInput in) throws IOException { - return from(in.readCollectionAsList(stream -> stream.readNamedWriteable(InternalAggregation.class))); + return from(in.readNamedWriteableCollectionAsList(InternalAggregation.class)); } @Override From 61264682f5c05bb97559c60d34ee45ff044637be Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 20 Feb 2024 07:57:44 +0100 Subject: [PATCH 37/45] Fix parsing of flattened fields within subobjects: false (#105373) --- docs/changelog/105373.yaml | 5 ++ .../index/mapper/DocumentParser.java | 3 +- .../index/mapper/DocumentParserTests.java | 33 +++++++++++++ .../index/mapper/DynamicTemplatesTests.java | 47 +++++++++++++++++++ 4 files changed, 87 insertions(+), 1 deletion(-) create mode 100644 docs/changelog/105373.yaml diff --git a/docs/changelog/105373.yaml b/docs/changelog/105373.yaml new file mode 100644 index 0000000000000..f9d3c718f7ae3 --- /dev/null +++ b/docs/changelog/105373.yaml @@ -0,0 +1,5 @@ +pr: 105373 +summary: "Fix parsing of flattened fields within subobjects: false" +area: Mapping +type: bug +issues: [] diff --git a/server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java b/server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java index 58ccd6025013f..9a0e391102708 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java @@ -455,11 +455,12 @@ private static void parseObject(final DocumentParserContext context, String curr private static void doParseObject(DocumentParserContext context, String currentFieldName, Mapper objectMapper) throws IOException { context.path().add(currentFieldName); + boolean withinLeafObject = context.path().isWithinLeafObject(); if (objectMapper instanceof ObjectMapper objMapper && objMapper.subobjects() == false) { context.path().setWithinLeafObject(true); } parseObjectOrField(context, objectMapper); - context.path().setWithinLeafObject(false); + context.path().setWithinLeafObject(withinLeafObject); context.path().remove(); } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/DocumentParserTests.java b/server/src/test/java/org/elasticsearch/index/mapper/DocumentParserTests.java index ed2efb4728b8d..d3dd585788867 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/DocumentParserTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/DocumentParserTests.java @@ -2273,6 +2273,39 @@ public void testSubobjectsFalseParentDynamicFalse() throws Exception { assertNull(doc.dynamicMappingsUpdate()); } + public void testSubobjectsFalseFlattened() throws Exception { + DocumentMapper mapper = createDocumentMapper(mapping(b -> { + b.startObject("attributes"); + { + b.field("dynamic", false); + b.field("subobjects", false); + b.startObject("properties"); + { + b.startObject("simple.attribute"); + b.field("type", "keyword"); + b.endObject(); + b.startObject("complex.attribute"); + b.field("type", "flattened"); + b.endObject(); + } + b.endObject(); + } + b.endObject(); + })); + ParsedDocument doc = mapper.parse(source(""" + { + "attributes": { + "complex.attribute": { + "foo" : "bar" + }, + "simple.attribute": "foo" + } + } + """)); + assertNotNull(doc.rootDoc().getField("attributes.complex.attribute")); + assertNotNull(doc.rootDoc().getField("attributes.simple.attribute")); + } + public void testWriteToFieldAlias() throws Exception { DocumentMapper mapper = createDocumentMapper(mapping(b -> { b.startObject("alias-field"); diff --git a/server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplatesTests.java b/server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplatesTests.java index 07bcc6d564bc7..38960597647e9 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplatesTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplatesTests.java @@ -1807,6 +1807,53 @@ public void testSubobjectsFalseDocWithEmptyObject() throws IOException { assertFalse(leaf.subobjects()); } + public void testSubobjectsFalseFlattened() throws IOException { + String mapping = """ + { + "_doc": { + "properties": { + "attributes": { + "type": "object", + "subobjects": false + } + }, + "dynamic_templates": [ + { + "test": { + "path_match": "attributes.*", + "match_mapping_type": "object", + "mapping": { + "type": "flattened" + } + } + } + ] + } + } + """; + String docJson = """ + { + "attributes": { + "complex.attribute": { + "a": "b" + }, + "foo.bar": "baz" + } + } + """; + + MapperService mapperService = createMapperService(mapping); + ParsedDocument parsedDoc = mapperService.documentMapper().parse(source(docJson)); + merge(mapperService, dynamicMapping(parsedDoc.dynamicMappingsUpdate())); + + Mapper fooBarMapper = mapperService.documentMapper().mappers().getMapper("attributes.foo.bar"); + assertNotNull(fooBarMapper); + assertEquals("text", fooBarMapper.typeName()); + Mapper fooStructuredMapper = mapperService.documentMapper().mappers().getMapper("attributes.complex.attribute"); + assertNotNull(fooStructuredMapper); + assertEquals("flattened", fooStructuredMapper.typeName()); + } + public void testMatchWithArrayOfFieldNames() throws IOException { String mapping = """ { From 369096365cfebfc947874631fc269ea94c8af8cf Mon Sep 17 00:00:00 2001 From: David Turner Date: Tue, 20 Feb 2024 08:43:18 +0000 Subject: [PATCH 38/45] Expand docs about max-shards-per-node (#105607) Adds a little more detail on what sorts of problems may occur if you exceed the default limits. --- .../how-to/size-your-shards.asciidoc | 9 +++ docs/reference/modules/cluster/misc.asciidoc | 64 +++++++++++-------- 2 files changed, 47 insertions(+), 26 deletions(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 8b631dbbaa5ce..4e2e9e0061b31 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -221,6 +221,15 @@ GET _cat/shards?v=true ---- // TEST[setup:my_index] +[discrete] +[[shard-count-per-node-recommendation]] +==== Add enough nodes to stay within the cluster shard limits + +The <> prevent creation of more than +1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen +node. Make sure you have enough nodes of each type in your cluster to handle +the number of shards you need. + [discrete] [[field-count-recommendation]] ==== Allow enough heap for field mappers and overheads diff --git a/docs/reference/modules/cluster/misc.asciidoc b/docs/reference/modules/cluster/misc.asciidoc index acaf2dea056fd..7eb1cf357498f 100644 --- a/docs/reference/modules/cluster/misc.asciidoc +++ b/docs/reference/modules/cluster/misc.asciidoc @@ -24,35 +24,46 @@ API can make the cluster read-write again. [discrete] [[cluster-shard-limit]] -==== Cluster shard limit +==== Cluster shard limits -There is a soft limit on the number of shards in a cluster, based on the number -of nodes in the cluster. This is intended to prevent operations which may -unintentionally destabilize the cluster. +There is a limit on the number of shards in a cluster, based on the number of +nodes in the cluster. This is intended to prevent a runaway process from +creating too many shards which can harm performance and in extreme cases may +destabilize your cluster. -IMPORTANT: This limit is intended as a safety net, not a sizing recommendation. The -exact number of shards your cluster can safely support depends on your hardware -configuration and workload, but should remain well below this limit in almost -all cases, as the default limit is set quite high. +[IMPORTANT] +==== -If an operation, such as creating a new index, restoring a snapshot of an index, -or opening a closed index would lead to the number of shards in the cluster -going over this limit, the operation will fail with an error indicating the -shard limit. +These limits are intended as a safety net to protect against runaway shard +creation and are not a sizing recommendation. The exact number of shards your +cluster can safely support depends on your hardware configuration and workload, +and may be smaller than the default limits. -If the cluster is already over the limit, due to changes in node membership or -setting changes, all operations that create or open indices will fail until -either the limit is increased as described below, or some indices are -<> or <> to bring the -number of shards below the limit. +We do not recommend increasing these limits beyond the defaults. Clusters with +more shards may appear to run well in normal operation, but may take a very +long time to recover from temporary disruptions such as a network partition or +an unexpected node restart, and may encounter problems when performing +maintenance activities such as a rolling restart or upgrade. -The cluster shard limit defaults to 1,000 shards per non-frozen data node for +==== + +If an operation, such as creating a new index, restoring a snapshot of an +index, or opening a closed index would lead to the number of shards in the +cluster going over this limit, the operation will fail with an error indicating +the shard limit. To resolve this, either scale out your cluster by adding +nodes, or <> to bring the number of +shards below the limit. + +If a cluster is already over the limit, perhaps due to changes in node +membership or setting changes, all operations that create or open indices will +fail. + +The cluster shard limit defaults to 1000 shards per non-frozen data node for normal (non-frozen) indices and 3000 shards per frozen data node for frozen -indices. -Both primary and replica shards of all open indices count toward the limit, -including unassigned shards. -For example, an open index with 5 primary shards and 2 replicas counts as 15 shards. -Closed indices do not contribute to the shard count. +indices. Both primary and replica shards of all open indices count toward the +limit, including unassigned shards. For example, an open index with 5 primary +shards and 2 replicas counts as 15 shards. Closed indices do not contribute to +the shard count. You can dynamically adjust the cluster shard limit with the following setting: @@ -99,12 +110,13 @@ For example, a cluster with a `cluster.max_shards_per_node.frozen` setting of `100` and three frozen data nodes has a frozen shard limit of 300. If the cluster already contains 296 shards, {es} rejects any request that adds five or more frozen shards to the cluster. +-- -NOTE: These setting do not limit shards for individual nodes. To limit the -number of shards for each node, use the +NOTE: These limits only apply to actions which create shards and do not limit +the number of shards assigned to each node. To limit the number of shards +assigned to each node, use the <> setting. --- [discrete] [[user-defined-data]] From b6c3cef82936e49854fd652c4ba57cb4e14d17e1 Mon Sep 17 00:00:00 2001 From: Jedr Blaszyk Date: Tue, 20 Feb 2024 11:18:16 +0100 Subject: [PATCH 39/45] [Connector API] Bugfix: support list type in filtering advanced snippet value (#105633) --- docs/changelog/105633.yaml | 6 ++ .../332_connector_update_filtering.yml | 13 +++- .../filtering/FilteringAdvancedSnippet.java | 29 ++++---- .../connector/ConnectorFilteringTests.java | 67 +++++++++++++++++++ .../application/connector/ConnectorTests.java | 18 ++++- 5 files changed, 116 insertions(+), 17 deletions(-) create mode 100644 docs/changelog/105633.yaml diff --git a/docs/changelog/105633.yaml b/docs/changelog/105633.yaml new file mode 100644 index 0000000000000..b19ec67f4602a --- /dev/null +++ b/docs/changelog/105633.yaml @@ -0,0 +1,6 @@ +pr: 105633 +summary: "[Connector API] Bugfix: support list type in filtering advenced snippet\ + \ value" +area: Application +type: bug +issues: [] diff --git a/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/332_connector_update_filtering.yml b/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/332_connector_update_filtering.yml index c5634365db3ec..a693ba5431d4b 100644 --- a/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/332_connector_update_filtering.yml +++ b/x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/332_connector_update_filtering.yml @@ -23,7 +23,10 @@ setup: advanced_snippet: created_at: "2023-05-25T12:30:00.000Z" updated_at: "2023-05-25T12:30:00.000Z" - value: {} + value: + - tables: + - some_table + query: 'SELECT id, st_geohash(coordinates) FROM my_db.some_table;' rules: - created_at: "2023-05-25T12:30:00.000Z" field: _ @@ -41,7 +44,13 @@ setup: advanced_snippet: created_at: "2023-05-25T12:30:00.000Z" updated_at: "2023-05-25T12:30:00.000Z" - value: {} + value: + - tables: + - some_table + query: 'SELECT id, st_geohash(coordinates) FROM my_db.some_table;' + - tables: + - another_table + query: 'SELECT id, st_geohash(coordinates) FROM my_db.another_table;' rules: - created_at: "2023-05-25T12:30:00.000Z" field: _ diff --git a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/filtering/FilteringAdvancedSnippet.java b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/filtering/FilteringAdvancedSnippet.java index 480eaf91bb23b..384fbc7bb5340 100644 --- a/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/filtering/FilteringAdvancedSnippet.java +++ b/x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/connector/filtering/FilteringAdvancedSnippet.java @@ -15,12 +15,12 @@ import org.elasticsearch.xcontent.ParseField; import org.elasticsearch.xcontent.ToXContentObject; import org.elasticsearch.xcontent.XContentBuilder; +import org.elasticsearch.xcontent.XContentParseException; import org.elasticsearch.xcontent.XContentParser; import org.elasticsearch.xpack.application.connector.ConnectorUtils; import java.io.IOException; import java.time.Instant; -import java.util.Map; import java.util.Objects; import static org.elasticsearch.xcontent.ConstructingObjectParser.constructorArg; @@ -34,18 +34,14 @@ public class FilteringAdvancedSnippet implements Writeable, ToXContentObject { private final Instant advancedSnippetCreatedAt; private final Instant advancedSnippetUpdatedAt; - private final Map advancedSnippetValue; + private final Object advancedSnippetValue; /** * @param advancedSnippetCreatedAt The creation timestamp of the advanced snippet. * @param advancedSnippetUpdatedAt The update timestamp of the advanced snippet. * @param advancedSnippetValue The value of the advanced snippet. */ - private FilteringAdvancedSnippet( - Instant advancedSnippetCreatedAt, - Instant advancedSnippetUpdatedAt, - Map advancedSnippetValue - ) { + private FilteringAdvancedSnippet(Instant advancedSnippetCreatedAt, Instant advancedSnippetUpdatedAt, Object advancedSnippetValue) { this.advancedSnippetCreatedAt = advancedSnippetCreatedAt; this.advancedSnippetUpdatedAt = advancedSnippetUpdatedAt; this.advancedSnippetValue = advancedSnippetValue; @@ -54,7 +50,7 @@ private FilteringAdvancedSnippet( public FilteringAdvancedSnippet(StreamInput in) throws IOException { this.advancedSnippetCreatedAt = in.readInstant(); this.advancedSnippetUpdatedAt = in.readInstant(); - this.advancedSnippetValue = in.readMap(StreamInput::readString, StreamInput::readGenericValue); + this.advancedSnippetValue = in.readGenericValue(); } private static final ParseField CREATED_AT_FIELD = new ParseField("created_at"); @@ -67,7 +63,7 @@ public FilteringAdvancedSnippet(StreamInput in) throws IOException { true, args -> new Builder().setAdvancedSnippetCreatedAt((Instant) args[0]) .setAdvancedSnippetUpdatedAt((Instant) args[1]) - .setAdvancedSnippetValue((Map) args[2]) + .setAdvancedSnippetValue(args[2]) .build() ); @@ -84,7 +80,14 @@ public FilteringAdvancedSnippet(StreamInput in) throws IOException { UPDATED_AT_FIELD, ObjectParser.ValueType.STRING ); - PARSER.declareField(constructorArg(), (p, c) -> p.map(), VALUE_FIELD, ObjectParser.ValueType.OBJECT); + PARSER.declareField(constructorArg(), (p, c) -> { + if (p.currentToken() == XContentParser.Token.START_ARRAY) { + return p.list(); + } else if (p.currentToken() == XContentParser.Token.START_OBJECT) { + return p.map(); + } + throw new XContentParseException("Unsupported token [" + p.currentToken() + "]. Expected an array or an object."); + }, VALUE_FIELD, ObjectParser.ValueType.OBJECT_ARRAY); } @Override @@ -107,7 +110,7 @@ public static FilteringAdvancedSnippet fromXContent(XContentParser parser) throw public void writeTo(StreamOutput out) throws IOException { out.writeInstant(advancedSnippetCreatedAt); out.writeInstant(advancedSnippetUpdatedAt); - out.writeMap(advancedSnippetValue, StreamOutput::writeString, StreamOutput::writeGenericValue); + out.writeGenericValue(advancedSnippetValue); } @Override @@ -129,7 +132,7 @@ public static class Builder { private Instant advancedSnippetCreatedAt; private Instant advancedSnippetUpdatedAt; - private Map advancedSnippetValue; + private Object advancedSnippetValue; public Builder setAdvancedSnippetCreatedAt(Instant advancedSnippetCreatedAt) { this.advancedSnippetCreatedAt = advancedSnippetCreatedAt; @@ -141,7 +144,7 @@ public Builder setAdvancedSnippetUpdatedAt(Instant advancedSnippetUpdatedAt) { return this; } - public Builder setAdvancedSnippetValue(Map advancedSnippetValue) { + public Builder setAdvancedSnippetValue(Object advancedSnippetValue) { this.advancedSnippetValue = advancedSnippetValue; return this; } diff --git a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorFilteringTests.java b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorFilteringTests.java index e65236e90d928..20c2200b26f2b 100644 --- a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorFilteringTests.java +++ b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorFilteringTests.java @@ -110,6 +110,73 @@ public void testToXContent() throws IOException { } + public void testToXContent_WithAdvancedSnippetPopulated() throws IOException { + String content = XContentHelper.stripWhitespace(""" + { + "active": { + "advanced_snippet": { + "created_at": "2023-11-09T15:13:08.231Z", + "updated_at": "2023-11-09T15:13:08.231Z", + "value": [ + {"service": "Incident", "query": "user_nameSTARTSWITHa"}, + {"service": "Incident", "query": "user_nameSTARTSWITHj"} + ] + }, + "rules": [ + { + "created_at": "2023-11-09T15:13:08.231Z", + "field": "_", + "id": "DEFAULT", + "order": 0, + "policy": "include", + "rule": "regex", + "updated_at": "2023-11-09T15:13:08.231Z", + "value": ".*" + } + ], + "validation": { + "errors": [], + "state": "valid" + } + }, + "domain": "DEFAULT", + "draft": { + "advanced_snippet": { + "created_at": "2023-11-09T15:13:08.231Z", + "updated_at": "2023-11-09T15:13:08.231Z", + "value": {} + }, + "rules": [ + { + "created_at": "2023-11-09T15:13:08.231Z", + "field": "_", + "id": "DEFAULT", + "order": 0, + "policy": "include", + "rule": "regex", + "updated_at": "2023-11-09T15:13:08.231Z", + "value": ".*" + } + ], + "validation": { + "errors": [], + "state": "valid" + } + } + } + """); + + ConnectorFiltering filtering = ConnectorFiltering.fromXContentBytes(new BytesArray(content), XContentType.JSON); + boolean humanReadable = true; + BytesReference originalBytes = toShuffledXContent(filtering, XContentType.JSON, ToXContent.EMPTY_PARAMS, humanReadable); + ConnectorFiltering parsed; + try (XContentParser parser = createParser(XContentType.JSON.xContent(), originalBytes)) { + parsed = ConnectorFiltering.fromXContent(parser); + } + assertToXContentEquivalent(originalBytes, toXContent(parsed, XContentType.JSON, humanReadable), XContentType.JSON); + + } + private void assertTransportSerialization(ConnectorFiltering testInstance) throws IOException { ConnectorFiltering deserializedInstance = copyInstance(testInstance); assertNotSame(testInstance, deserializedInstance); diff --git a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorTests.java b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorTests.java index 0fd590a4ce106..5525b4694ef04 100644 --- a/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorTests.java +++ b/x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/connector/ConnectorTests.java @@ -147,7 +147,14 @@ public void testToXContent() throws IOException { "advanced_snippet":{ "created_at":"2023-11-09T15:13:08.231Z", "updated_at":"2023-11-09T15:13:08.231Z", - "value":{} + "value":[ + { + "tables": [ + "some_table" + ], + "query": "SELECT id, st_geohash(coordinates) FROM my_db.some_table;" + } + ] }, "rules":[ { @@ -171,7 +178,14 @@ public void testToXContent() throws IOException { "advanced_snippet":{ "created_at":"2023-11-09T15:13:08.231Z", "updated_at":"2023-11-09T15:13:08.231Z", - "value":{} + "value":[ + { + "tables": [ + "some_table" + ], + "query": "SELECT id, st_geohash(coordinates) FROM my_db.some_table;" + } + ] }, "rules":[ { From 1b7f3a0eae0e1f936f3c23e9e61548e5594adcff Mon Sep 17 00:00:00 2001 From: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com> Date: Tue, 20 Feb 2024 12:01:04 +0100 Subject: [PATCH 40/45] Extract common inference ServiceSettings methods (#105553) --- .../inference/SemanticTextModelSettings.java | 91 +++++++++++++++++++ .../inference/ServiceSettings.java | 19 ++++ .../inference}/SimilarityMeasure.java | 7 +- .../inference/services/ServiceUtils.java | 2 +- .../services/cohere/CohereService.java | 2 +- .../cohere/CohereServiceSettings.java | 2 +- .../HuggingFaceServiceSettings.java | 2 +- .../services/openai/OpenAiService.java | 2 +- .../OpenAiEmbeddingsServiceSettings.java | 2 +- ...lingualE5SmallInternalServiceSettings.java | 5 + .../TextEmbeddingInternalService.java | 3 +- .../cohere/CohereServiceSettingsTests.java | 2 +- .../CohereEmbeddingsModelTests.java | 2 +- .../CohereEmbeddingsServiceSettingsTests.java | 2 +- .../HuggingFaceServiceSettingsTests.java | 2 +- .../services/openai/OpenAiServiceTests.java | 2 +- .../OpenAiEmbeddingsModelTests.java | 2 +- .../OpenAiEmbeddingsServiceSettingsTests.java | 2 +- 18 files changed, 133 insertions(+), 18 deletions(-) create mode 100644 server/src/main/java/org/elasticsearch/inference/SemanticTextModelSettings.java rename {x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/common => server/src/main/java/org/elasticsearch/inference}/SimilarityMeasure.java (68%) diff --git a/server/src/main/java/org/elasticsearch/inference/SemanticTextModelSettings.java b/server/src/main/java/org/elasticsearch/inference/SemanticTextModelSettings.java new file mode 100644 index 0000000000000..78773bfb72a95 --- /dev/null +++ b/server/src/main/java/org/elasticsearch/inference/SemanticTextModelSettings.java @@ -0,0 +1,91 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +package org.elasticsearch.inference; + +import org.elasticsearch.core.Nullable; +import org.elasticsearch.xcontent.ConstructingObjectParser; +import org.elasticsearch.xcontent.ParseField; +import org.elasticsearch.xcontent.XContentParser; + +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; + +/** + * Model settings that are interesting for semantic_text inference fields. This class is used to serialize common + * ServiceSettings methods when building inference for semantic_text fields. + * + * @param taskType task type + * @param inferenceId inference id + * @param dimensions number of dimensions. May be null if not applicable + * @param similarity similarity used by the service. May be null if not applicable + */ +public record SemanticTextModelSettings( + TaskType taskType, + String inferenceId, + @Nullable Integer dimensions, + @Nullable SimilarityMeasure similarity +) { + + public static final String NAME = "model_settings"; + private static final ParseField TASK_TYPE_FIELD = new ParseField("task_type"); + private static final ParseField INFERENCE_ID_FIELD = new ParseField("inference_id"); + private static final ParseField DIMENSIONS_FIELD = new ParseField("dimensions"); + private static final ParseField SIMILARITY_FIELD = new ParseField("similarity"); + + public SemanticTextModelSettings(TaskType taskType, String inferenceId, Integer dimensions, SimilarityMeasure similarity) { + Objects.requireNonNull(taskType, "task type must not be null"); + Objects.requireNonNull(inferenceId, "inferenceId must not be null"); + this.taskType = taskType; + this.inferenceId = inferenceId; + this.dimensions = dimensions; + this.similarity = similarity; + } + + public SemanticTextModelSettings(Model model) { + this( + model.getTaskType(), + model.getInferenceEntityId(), + model.getServiceSettings().dimensions(), + model.getServiceSettings().similarity() + ); + } + + public static SemanticTextModelSettings parse(XContentParser parser) throws IOException { + return PARSER.apply(parser, null); + } + + private static final ConstructingObjectParser PARSER = new ConstructingObjectParser<>(NAME, args -> { + TaskType taskType = TaskType.fromString((String) args[0]); + String inferenceId = (String) args[1]; + Integer dimensions = (Integer) args[2]; + SimilarityMeasure similarity = args[3] == null ? null : SimilarityMeasure.fromString((String) args[2]); + return new SemanticTextModelSettings(taskType, inferenceId, dimensions, similarity); + }); + static { + PARSER.declareString(ConstructingObjectParser.constructorArg(), TASK_TYPE_FIELD); + PARSER.declareString(ConstructingObjectParser.constructorArg(), INFERENCE_ID_FIELD); + PARSER.declareInt(ConstructingObjectParser.optionalConstructorArg(), DIMENSIONS_FIELD); + PARSER.declareString(ConstructingObjectParser.optionalConstructorArg(), SIMILARITY_FIELD); + } + + public Map asMap() { + Map attrsMap = new HashMap<>(); + attrsMap.put(TASK_TYPE_FIELD.getPreferredName(), taskType.toString()); + attrsMap.put(INFERENCE_ID_FIELD.getPreferredName(), inferenceId); + if (dimensions != null) { + attrsMap.put(DIMENSIONS_FIELD.getPreferredName(), dimensions); + } + if (similarity != null) { + attrsMap.put(SIMILARITY_FIELD.getPreferredName(), similarity); + } + return Map.of(NAME, attrsMap); + } +} diff --git a/server/src/main/java/org/elasticsearch/inference/ServiceSettings.java b/server/src/main/java/org/elasticsearch/inference/ServiceSettings.java index 6fed8bb7239e5..2e745635d0fd9 100644 --- a/server/src/main/java/org/elasticsearch/inference/ServiceSettings.java +++ b/server/src/main/java/org/elasticsearch/inference/ServiceSettings.java @@ -17,4 +17,23 @@ public interface ServiceSettings extends ToXContentObject, VersionedNamedWriteab * Returns a {@link ToXContentObject} that only writes the exposed fields. Any hidden fields are not written. */ ToXContentObject getFilteredXContentObject(); + + /** + * Similarity used in the service. Will be null if not applicable. + * + * @return similarity + */ + default SimilarityMeasure similarity() { + return null; + } + + /** + * Number of dimensions the service works with. Will be null if not applicable. + * + * @return number of dimensions + */ + default Integer dimensions() { + return null; + } + } diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/common/SimilarityMeasure.java b/server/src/main/java/org/elasticsearch/inference/SimilarityMeasure.java similarity index 68% rename from x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/common/SimilarityMeasure.java rename to server/src/main/java/org/elasticsearch/inference/SimilarityMeasure.java index 3028ecd078597..cd81cc461bd1d 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/common/SimilarityMeasure.java +++ b/server/src/main/java/org/elasticsearch/inference/SimilarityMeasure.java @@ -1,11 +1,12 @@ /* * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one * or more contributor license agreements. Licensed under the Elastic License - * 2.0; you may not use this file except in compliance with the Elastic License - * 2.0. + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. */ -package org.elasticsearch.xpack.inference.common; +package org.elasticsearch.inference; import java.util.Locale; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/ServiceUtils.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/ServiceUtils.java index 532fd2359ac2b..cfbb07cb940e7 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/ServiceUtils.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/ServiceUtils.java @@ -16,11 +16,11 @@ import org.elasticsearch.inference.InferenceService; import org.elasticsearch.inference.InputType; import org.elasticsearch.inference.Model; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.inference.TaskType; import org.elasticsearch.rest.RestStatus; import org.elasticsearch.xpack.core.inference.results.TextEmbedding; import org.elasticsearch.xpack.core.inference.results.TextEmbeddingResults; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import java.net.URI; import java.net.URISyntaxException; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereService.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereService.java index bc511c043fdf3..35b245e9a657a 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereService.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereService.java @@ -20,9 +20,9 @@ import org.elasticsearch.inference.Model; import org.elasticsearch.inference.ModelConfigurations; import org.elasticsearch.inference.ModelSecrets; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.inference.TaskType; import org.elasticsearch.rest.RestStatus; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.external.action.cohere.CohereActionCreator; import org.elasticsearch.xpack.inference.external.http.sender.HttpRequestSenderFactory; import org.elasticsearch.xpack.inference.services.SenderService; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettings.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettings.java index eb6dbc352d36d..97ad1b575caa9 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettings.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettings.java @@ -17,9 +17,9 @@ import org.elasticsearch.core.Nullable; import org.elasticsearch.inference.ModelConfigurations; import org.elasticsearch.inference.ServiceSettings; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.xcontent.ToXContentObject; import org.elasticsearch.xcontent.XContentBuilder; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import java.io.IOException; import java.net.URI; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettings.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettings.java index 92920e0b9224f..f176cf7580567 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettings.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettings.java @@ -15,9 +15,9 @@ import org.elasticsearch.core.Nullable; import org.elasticsearch.inference.ModelConfigurations; import org.elasticsearch.inference.ServiceSettings; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.xcontent.ToXContentObject; import org.elasticsearch.xcontent.XContentBuilder; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import java.io.IOException; import java.net.URI; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/OpenAiService.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/OpenAiService.java index 7010b59990cd3..03781450fc08c 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/OpenAiService.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/OpenAiService.java @@ -21,9 +21,9 @@ import org.elasticsearch.inference.Model; import org.elasticsearch.inference.ModelConfigurations; import org.elasticsearch.inference.ModelSecrets; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.inference.TaskType; import org.elasticsearch.rest.RestStatus; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.external.action.openai.OpenAiActionCreator; import org.elasticsearch.xpack.inference.external.http.sender.HttpRequestSenderFactory; import org.elasticsearch.xpack.inference.services.SenderService; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettings.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettings.java index 229e45a024458..468e82d4f0866 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettings.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettings.java @@ -15,9 +15,9 @@ import org.elasticsearch.core.Nullable; import org.elasticsearch.inference.ModelConfigurations; import org.elasticsearch.inference.ServiceSettings; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.xcontent.ToXContentObject; import org.elasticsearch.xcontent.XContentBuilder; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.ServiceUtils; import org.elasticsearch.xpack.inference.services.openai.OpenAiParseContext; diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/MultilingualE5SmallInternalServiceSettings.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/MultilingualE5SmallInternalServiceSettings.java index cab9d9d863885..aa1de0e0beddc 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/MultilingualE5SmallInternalServiceSettings.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/MultilingualE5SmallInternalServiceSettings.java @@ -103,4 +103,9 @@ public TransportVersion getMinimalSupportedVersion() { public void writeTo(StreamOutput out) throws IOException { super.writeTo(out); } + + @Override + public Integer dimensions() { + return 384; + } } diff --git a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/TextEmbeddingInternalService.java b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/TextEmbeddingInternalService.java index 59228c6dcbddf..06d6545a381bd 100644 --- a/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/TextEmbeddingInternalService.java +++ b/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/textembedding/TextEmbeddingInternalService.java @@ -37,7 +37,6 @@ import org.elasticsearch.xpack.core.ml.inference.TrainedModelConfig; import org.elasticsearch.xpack.core.ml.inference.TrainedModelInput; import org.elasticsearch.xpack.core.ml.inference.trainedmodel.TextEmbeddingConfigUpdate; -import org.elasticsearch.xpack.core.ml.inference.trainedmodel.TextExpansionConfigUpdate; import org.elasticsearch.xpack.core.ml.inference.trainedmodel.TokenizationConfigUpdate; import org.elasticsearch.xpack.inference.services.settings.InternalServiceSettings; @@ -252,7 +251,7 @@ public void chunkedInfer( var configUpdate = chunkingOptions.settingsArePresent() ? new TokenizationConfigUpdate(chunkingOptions.windowSize(), chunkingOptions.span()) - : TextExpansionConfigUpdate.EMPTY_UPDATE; + : TextEmbeddingConfigUpdate.EMPTY_INSTANCE; var request = InferTrainedModelDeploymentAction.Request.forTextInput( model.getConfigurations().getInferenceEntityId(), diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettingsTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettingsTests.java index 321567dfa32e8..bbee8aa1de577 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettingsTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/CohereServiceSettingsTests.java @@ -11,11 +11,11 @@ import org.elasticsearch.common.ValidationException; import org.elasticsearch.common.io.stream.Writeable; import org.elasticsearch.core.Nullable; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.test.AbstractWireSerializingTestCase; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentType; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.ServiceFields; import org.elasticsearch.xpack.inference.services.ServiceUtils; import org.hamcrest.CoreMatchers; diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsModelTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsModelTests.java index 5570731dbe8d9..ec36ac5ce58d5 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsModelTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsModelTests.java @@ -10,9 +10,9 @@ import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.core.Nullable; import org.elasticsearch.inference.InputType; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.inference.TaskType; import org.elasticsearch.test.ESTestCase; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.cohere.CohereServiceSettings; import org.elasticsearch.xpack.inference.services.settings.DefaultSecretSettings; import org.hamcrest.MatcherAssert; diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsServiceSettingsTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsServiceSettingsTests.java index 39aba9c281a0c..2f5eba676a314 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsServiceSettingsTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/cohere/embeddings/CohereEmbeddingsServiceSettingsTests.java @@ -13,10 +13,10 @@ import org.elasticsearch.common.io.stream.NamedWriteableRegistry; import org.elasticsearch.common.io.stream.Writeable; import org.elasticsearch.core.Nullable; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.test.AbstractWireSerializingTestCase; import org.elasticsearch.xpack.core.ml.inference.MlInferenceNamedXContentProvider; import org.elasticsearch.xpack.inference.InferenceNamedWriteablesProvider; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.ServiceFields; import org.elasticsearch.xpack.inference.services.ServiceUtils; import org.elasticsearch.xpack.inference.services.cohere.CohereServiceSettings; diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettingsTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettingsTests.java index 7e2a333685321..f32fafd493395 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettingsTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/huggingface/HuggingFaceServiceSettingsTests.java @@ -10,8 +10,8 @@ import org.elasticsearch.common.Strings; import org.elasticsearch.common.ValidationException; import org.elasticsearch.common.io.stream.Writeable; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.test.AbstractWireSerializingTestCase; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.ServiceFields; import org.elasticsearch.xpack.inference.services.ServiceUtils; diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/OpenAiServiceTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/OpenAiServiceTests.java index 3fd4d17d7a6e4..b3d9a98bad189 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/OpenAiServiceTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/OpenAiServiceTests.java @@ -22,13 +22,13 @@ import org.elasticsearch.inference.Model; import org.elasticsearch.inference.ModelConfigurations; import org.elasticsearch.inference.ModelSecrets; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.inference.TaskType; import org.elasticsearch.test.ESTestCase; import org.elasticsearch.test.http.MockResponse; import org.elasticsearch.test.http.MockWebServer; import org.elasticsearch.threadpool.ThreadPool; import org.elasticsearch.xcontent.XContentType; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.external.http.HttpClientManager; import org.elasticsearch.xpack.inference.external.http.sender.HttpRequestSenderFactory; import org.elasticsearch.xpack.inference.external.http.sender.Sender; diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsModelTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsModelTests.java index 60ed5a13d9c58..01b60fdb896d0 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsModelTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsModelTests.java @@ -9,9 +9,9 @@ import org.elasticsearch.common.settings.SecureString; import org.elasticsearch.core.Nullable; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.inference.TaskType; import org.elasticsearch.test.ESTestCase; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.settings.DefaultSecretSettings; import java.util.Map; diff --git a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettingsTests.java b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettingsTests.java index 51069b46afb94..18b5ab44f59ca 100644 --- a/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettingsTests.java +++ b/x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/services/openai/embeddings/OpenAiEmbeddingsServiceSettingsTests.java @@ -11,11 +11,11 @@ import org.elasticsearch.common.ValidationException; import org.elasticsearch.common.io.stream.Writeable; import org.elasticsearch.core.Nullable; +import org.elasticsearch.inference.SimilarityMeasure; import org.elasticsearch.test.AbstractWireSerializingTestCase; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentType; -import org.elasticsearch.xpack.inference.common.SimilarityMeasure; import org.elasticsearch.xpack.inference.services.ServiceFields; import org.elasticsearch.xpack.inference.services.ServiceUtils; import org.elasticsearch.xpack.inference.services.openai.OpenAiParseContext; From b8dc5c3041be8085f45cb2ff3cb7e01d5e65aa2c Mon Sep 17 00:00:00 2001 From: Panagiotis Bailis Date: Tue, 20 Feb 2024 13:50:02 +0200 Subject: [PATCH 41/45] Fix for SearchServiceTests#testWaitOnRefreshFailsIfCheckpointNotIndexed - increasing timeout for randomly failing test (#105395) --- .../java/org/elasticsearch/search/SearchServiceTests.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/server/src/test/java/org/elasticsearch/search/SearchServiceTests.java b/server/src/test/java/org/elasticsearch/search/SearchServiceTests.java index 551874a2d271a..b0c4ef00230d5 100644 --- a/server/src/test/java/org/elasticsearch/search/SearchServiceTests.java +++ b/server/src/test/java/org/elasticsearch/search/SearchServiceTests.java @@ -1767,7 +1767,9 @@ public void testWaitOnRefreshFailsIfCheckpointNotIndexed() { final IndexService indexService = indicesService.indexServiceSafe(resolveIndex("index")); final IndexShard indexShard = indexService.getShard(0); SearchRequest searchRequest = new SearchRequest().allowPartialSearchResults(true); - searchRequest.setWaitForCheckpointsTimeout(TimeValue.timeValueMillis(randomIntBetween(10, 100))); + // Increased timeout to avoid cancelling the search task prior to its completion, + // as we expect to raise an Exception. Timeout itself is tested on the following `testWaitOnRefreshTimeout` test. + searchRequest.setWaitForCheckpointsTimeout(TimeValue.timeValueMillis(randomIntBetween(200, 300))); searchRequest.setWaitForCheckpoints(Collections.singletonMap("index", new long[] { 1 })); final DocWriteResponse response = prepareIndex("index").setSource("id", "1").get(); From 410efb6fb6d71433bc242d3aed5d3fa51a5acf29 Mon Sep 17 00:00:00 2001 From: Panagiotis Bailis Date: Tue, 20 Feb 2024 15:10:22 +0200 Subject: [PATCH 42/45] Fixing NPE when requesting [_none_] for stored_fields (#104711) --- docs/changelog/104711.yaml | 5 ++++ .../search/builder/SearchSourceBuilder.java | 8 +++++-- .../builder/SearchSourceBuilderTests.java | 23 +++++++++++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-) create mode 100644 docs/changelog/104711.yaml diff --git a/docs/changelog/104711.yaml b/docs/changelog/104711.yaml new file mode 100644 index 0000000000000..f0f9bf7f10e45 --- /dev/null +++ b/docs/changelog/104711.yaml @@ -0,0 +1,5 @@ +pr: 104711 +summary: "Fixing NPE when requesting [_none_] for `stored_fields`" +area: Search +type: bug +issues: [] diff --git a/server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java b/server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java index 649c40c856fe8..72fd84cda760b 100644 --- a/server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java +++ b/server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java @@ -1337,7 +1337,10 @@ private SearchSourceBuilder parseXContent( SearchSourceBuilder.STORED_FIELDS_FIELD.getPreferredName(), parser ); - searchUsage.trackSectionUsage(STORED_FIELDS_FIELD.getPreferredName()); + if (storedFieldsContext.fetchFields() == false + || (storedFieldsContext.fieldNames() != null && storedFieldsContext.fieldNames().size() > 0)) { + searchUsage.trackSectionUsage(STORED_FIELDS_FIELD.getPreferredName()); + } } else if (SORT_FIELD.match(currentFieldName, parser.getDeprecationHandler())) { sort(parser.text()); } else if (PROFILE_FIELD.match(currentFieldName, parser.getDeprecationHandler())) { @@ -1493,7 +1496,8 @@ private SearchSourceBuilder parseXContent( } else if (token == XContentParser.Token.START_ARRAY) { if (STORED_FIELDS_FIELD.match(currentFieldName, parser.getDeprecationHandler())) { storedFieldsContext = StoredFieldsContext.fromXContent(STORED_FIELDS_FIELD.getPreferredName(), parser); - if (storedFieldsContext.fieldNames().size() > 0 || storedFieldsContext.fetchFields() == false) { + if (storedFieldsContext.fetchFields() == false + || (storedFieldsContext.fieldNames() != null && storedFieldsContext.fieldNames().size() > 0)) { searchUsage.trackSectionUsage(STORED_FIELDS_FIELD.getPreferredName()); } } else if (DOCVALUE_FIELDS_FIELD.match(currentFieldName, parser.getDeprecationHandler())) { diff --git a/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java b/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java index 8ee1c64ddbb22..26eefe850fc8f 100644 --- a/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java +++ b/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java @@ -65,6 +65,7 @@ import java.util.Collections; import java.util.List; import java.util.Map; +import java.util.Set; import java.util.concurrent.TimeUnit; import java.util.function.Supplier; import java.util.function.ToLongFunction; @@ -592,6 +593,28 @@ public void testNegativeTrackTotalHits() throws IOException { } } + public void testStoredFieldsUsage() throws IOException { + Set storedFieldRestVariations = Set.of( + "{\"stored_fields\" : [\"_none_\"]}", + "{\"stored_fields\" : \"_none_\"}", + "{\"stored_fields\" : [\"field\"]}", + "{\"stored_fields\" : \"field\"}" + ); + for (String storedFieldRest : storedFieldRestVariations) { + SearchUsageHolder searchUsageHolder = new UsageService().getSearchUsageHolder(); + try (XContentParser parser = createParser(JsonXContent.jsonXContent, storedFieldRest)) { + new SearchSourceBuilder().parseXContent(parser, true, searchUsageHolder); + SearchUsageStats searchUsageStats = searchUsageHolder.getSearchUsageStats(); + Map sectionsUsage = searchUsageStats.getSectionsUsage(); + assertEquals( + "Failed to correctly parse and record usage of '" + storedFieldRest + "'", + 1L, + sectionsUsage.get("stored_fields").longValue() + ); + } + } + } + public void testEmptySectionsAreNotTracked() throws IOException { SearchUsageHolder searchUsageHolder = new UsageService().getSearchUsageHolder(); From 7fb4b74b95da14e947c47d6ec3c2542e3bc1f1be Mon Sep 17 00:00:00 2001 From: Pat Whelan Date: Tue, 20 Feb 2024 08:22:27 -0500 Subject: [PATCH 43/45] [Transform] Test waits for next iteration (#105560) Currently, the `testStopWaitForCheckpoint` only verifies that the transform state is `stopped`, which might be the previous iteration's state. There is a small chance that we may exit the loop before the transform starts and stops for that iteration, where the test might fail the final `stopped` check. Now, we check the `trigger_count` to verify that the transform has at least had a chance to move from the `STARTED` state into the `INDEXING` and eventually `STOPPED` state before we finish the iteration. Fix #105388 Co-authored-by: Elastic Machine --- .../transform/integration/TransformIT.java | 34 +++++++++++++------ 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/x-pack/plugin/transform/qa/multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/transform/integration/TransformIT.java b/x-pack/plugin/transform/qa/multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/transform/integration/TransformIT.java index 394732742e528..073f604e608da 100644 --- a/x-pack/plugin/transform/qa/multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/transform/integration/TransformIT.java +++ b/x-pack/plugin/transform/qa/multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/transform/integration/TransformIT.java @@ -37,7 +37,9 @@ import java.util.HashMap; import java.util.Map; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import static org.elasticsearch.core.Strings.format; import static org.elasticsearch.xcontent.XContentFactory.jsonBuilder; import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; @@ -375,7 +377,7 @@ public void testStopWaitForCheckpoint() throws Exception { // wait until transform has been triggered and indexed at least 1 document assertBusy(() -> { - var stateAndStats = getBasicTransformStats(config.getId()); + var stateAndStats = getBasicTransformStats(transformId); assertThat((Integer) XContentMapValues.extractValue("stats.documents_indexed", stateAndStats), greaterThan(1)); }); @@ -384,39 +386,51 @@ public void testStopWaitForCheckpoint() throws Exception { // Wait until the first checkpoint waitUntilCheckpoint(config.getId(), 1L); + var previousTriggerCount = new AtomicInteger(0); // Even though we are continuous, we should be stopped now as we needed to stop at the first checkpoint assertBusy(() -> { - var stateAndStats = getBasicTransformStats(config.getId()); + var stateAndStats = getBasicTransformStats(transformId); assertThat(stateAndStats.get("state"), equalTo("stopped")); assertThat((Integer) XContentMapValues.extractValue("stats.documents_indexed", stateAndStats), equalTo(1000)); + previousTriggerCount.set((int) XContentMapValues.extractValue("stats.trigger_count", stateAndStats)); }); + // Create N additional runs of starting and stopping int additionalRuns = randomIntBetween(1, 10); for (int i = 0; i < additionalRuns; ++i) { + var testFailureMessage = format("Can't determine if Transform ran for iteration number [%d] out of [%d].", i, additionalRuns); // index some more docs using a new user - long timeStamp = Instant.now().toEpochMilli() - 1_000; - long user = 42 + i; + var timeStamp = Instant.now().toEpochMilli() - 1_000; + var user = 42 + i; indexMoreDocs(timeStamp, user, indexName); - startTransformWithRetryOnConflict(config.getId(), RequestOptions.DEFAULT); + startTransformWithRetryOnConflict(transformId, RequestOptions.DEFAULT); - boolean waitForCompletion = randomBoolean(); - stopTransform(transformId, waitForCompletion, null, true); + assertBusy(() -> { + var stateAndStats = getBasicTransformStats(transformId); + var currentTriggerCount = (int) XContentMapValues.extractValue("stats.trigger_count", stateAndStats); + // We should verify that we are retrieving the stats *after* this run had been started. + // If the trigger_count has increased, we know we have started this test iteration. + assertThat(testFailureMessage, previousTriggerCount.get(), lessThan(currentTriggerCount)); + }); + var waitForCompletion = randomBoolean(); + stopTransform(transformId, waitForCompletion, null, true); assertBusy(() -> { - var stateAndStats = getBasicTransformStats(config.getId()); + var stateAndStats = getBasicTransformStats(transformId); assertThat(stateAndStats.get("state"), equalTo("stopped")); + previousTriggerCount.set((int) XContentMapValues.extractValue("stats.trigger_count", stateAndStats)); }); } - var stateAndStats = getBasicTransformStats(config.getId()); + var stateAndStats = getBasicTransformStats(transformId); assertThat(stateAndStats.get("state"), equalTo("stopped")); // Despite indexing new documents into the source index, the number of documents in the destination index stays the same. assertThat((Integer) XContentMapValues.extractValue("stats.documents_indexed", stateAndStats), equalTo(1000)); stopTransform(transformId); - deleteTransform(config.getId()); + deleteTransform(transformId); } public void testContinuousTransformRethrottle() throws Exception { From 0d0b319bf9714395aa7439b6fe8cea5d52a2bceb Mon Sep 17 00:00:00 2001 From: Panagiotis Bailis Date: Tue, 20 Feb 2024 16:28:27 +0200 Subject: [PATCH 44/45] Fixing compilation error in SearchSourceBuilderTests#testStoredFieldsUsage (#105656) --- .../elasticsearch/search/builder/SearchSourceBuilderTests.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java b/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java index 26eefe850fc8f..7b67bc5b94f7f 100644 --- a/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java +++ b/server/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java @@ -603,7 +603,7 @@ public void testStoredFieldsUsage() throws IOException { for (String storedFieldRest : storedFieldRestVariations) { SearchUsageHolder searchUsageHolder = new UsageService().getSearchUsageHolder(); try (XContentParser parser = createParser(JsonXContent.jsonXContent, storedFieldRest)) { - new SearchSourceBuilder().parseXContent(parser, true, searchUsageHolder); + new SearchSourceBuilder().parseXContent(parser, true, searchUsageHolder, nf -> false); SearchUsageStats searchUsageStats = searchUsageHolder.getSearchUsageStats(); Map sectionsUsage = searchUsageStats.getSectionsUsage(); assertEquals( From 5920c917aa933bf8078e3c88f43c217957bf9dd0 Mon Sep 17 00:00:00 2001 From: Felix Barnsteiner Date: Tue, 20 Feb 2024 15:53:14 +0100 Subject: [PATCH 45/45] Encapsulate Mapper.Builder#name and make it private (#105648) This is in preparation to make the field mutable, which is needed in the context of https://github.com/elastic/elasticsearch/pull/103542 --- .../legacygeo/mapper/LegacyGeoShapeFieldMapper.java | 4 ++-- .../index/mapper/extras/MatchOnlyTextFieldMapper.java | 4 ++-- .../index/mapper/extras/RankFeatureFieldMapper.java | 4 ++-- .../index/mapper/extras/RankFeaturesFieldMapper.java | 4 ++-- .../index/mapper/extras/ScaledFloatFieldMapper.java | 4 ++-- .../mapper/extras/SearchAsYouTypeFieldMapper.java | 8 ++++---- .../index/mapper/extras/TokenCountFieldMapper.java | 6 +++--- .../join/mapper/ParentJoinFieldMapper.java | 8 ++++---- .../percolator/PercolatorFieldMapper.java | 4 ++-- .../analysis/icu/ICUCollationKeywordFieldMapper.java | 4 ++-- .../mapper/annotatedtext/AnnotatedTextFieldMapper.java | 6 +++--- .../index/mapper/murmur3/Murmur3FieldMapper.java | 4 ++-- .../elasticsearch/index/mapper/BinaryFieldMapper.java | 4 ++-- .../elasticsearch/index/mapper/BooleanFieldMapper.java | 6 +++--- .../index/mapper/CompletionFieldMapper.java | 4 ++-- .../elasticsearch/index/mapper/DateFieldMapper.java | 4 ++-- .../index/mapper/GeoPointFieldMapper.java | 10 +++++----- .../index/mapper/GeoShapeFieldMapper.java | 6 +++--- .../org/elasticsearch/index/mapper/IpFieldMapper.java | 6 +++--- .../elasticsearch/index/mapper/KeywordFieldMapper.java | 8 ++++---- .../java/org/elasticsearch/index/mapper/Mapper.java | 2 +- .../elasticsearch/index/mapper/NestedObjectMapper.java | 6 +++--- .../elasticsearch/index/mapper/NumberFieldMapper.java | 6 +++--- .../org/elasticsearch/index/mapper/ObjectMapper.java | 6 +++--- .../index/mapper/PassThroughObjectMapper.java | 6 +++--- .../index/mapper/PlaceHolderFieldMapper.java | 4 ++-- .../elasticsearch/index/mapper/RangeFieldMapper.java | 4 ++-- .../elasticsearch/index/mapper/RootObjectMapper.java | 2 +- .../elasticsearch/index/mapper/TextFieldMapper.java | 10 +++++----- .../index/mapper/flattened/FlattenedFieldMapper.java | 8 ++++---- .../index/mapper/vectors/DenseVectorFieldMapper.java | 4 ++-- .../index/mapper/vectors/SparseVectorFieldMapper.java | 4 ++-- .../index/mapper/ParametrizedMapperTests.java | 2 +- .../xpack/analytics/mapper/HistogramFieldMapper.java | 4 ++-- .../mapper/AggregateDoubleMetricFieldMapper.java | 6 +++--- .../mapper/ConstantKeywordFieldMapper.java | 4 ++-- .../countedkeyword/CountedKeywordFieldMapper.java | 6 +++--- .../xpack/unsignedlong/UnsignedLongFieldMapper.java | 4 ++-- .../xpack/versionfield/VersionStringFieldMapper.java | 4 ++-- .../index/mapper/GeoShapeWithDocValuesFieldMapper.java | 8 ++++---- .../xpack/spatial/index/mapper/PointFieldMapper.java | 6 +++--- .../xpack/spatial/index/mapper/ShapeFieldMapper.java | 4 ++-- .../xpack/wildcard/mapper/WildcardFieldMapper.java | 4 ++-- 43 files changed, 111 insertions(+), 111 deletions(-) diff --git a/modules/legacy-geo/src/main/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapper.java b/modules/legacy-geo/src/main/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapper.java index afd969cc17ad4..4ef2b2e07bb26 100644 --- a/modules/legacy-geo/src/main/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapper.java +++ b/modules/legacy-geo/src/main/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapper.java @@ -324,7 +324,7 @@ private static void setupPrefixTrees(GeoShapeFieldType ft) { private GeoShapeFieldType buildFieldType(LegacyGeoShapeParser parser, MapperBuilderContext context) { GeoShapeFieldType ft = new GeoShapeFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get(), orientation.get().value(), parser, @@ -353,7 +353,7 @@ private static int getLevels(int treeLevels, double precisionInMeters, int defau public LegacyGeoShapeFieldMapper build(MapperBuilderContext context) { LegacyGeoShapeParser parser = new LegacyGeoShapeParser(); GeoShapeFieldType ft = buildFieldType(parser, context); - return new LegacyGeoShapeFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, parser, this); + return new LegacyGeoShapeFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, parser, this); } } diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapper.java index fa83e2600de9b..a965b9a2bbce4 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapper.java @@ -127,7 +127,7 @@ private MatchOnlyTextFieldType buildFieldType(MapperBuilderContext context) { NamedAnalyzer indexAnalyzer = analyzers.getIndexAnalyzer(); TextSearchInfo tsi = new TextSearchInfo(Defaults.FIELD_TYPE, null, searchAnalyzer, searchQuoteAnalyzer); MatchOnlyTextFieldType ft = new MatchOnlyTextFieldType( - context.buildFullName(name), + context.buildFullName(name()), tsi, indexAnalyzer, context.isSourceSynthetic(), @@ -140,7 +140,7 @@ private MatchOnlyTextFieldType buildFieldType(MapperBuilderContext context) { public MatchOnlyTextFieldMapper build(MapperBuilderContext context) { MatchOnlyTextFieldType tft = buildFieldType(context); MultiFields multiFields = multiFieldsBuilder.build(this, context); - return new MatchOnlyTextFieldMapper(name, Defaults.FIELD_TYPE, tft, multiFields, copyTo, context.isSourceSynthetic(), this); + return new MatchOnlyTextFieldMapper(name(), Defaults.FIELD_TYPE, tft, multiFields, copyTo, context.isSourceSynthetic(), this); } } diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapper.java index b5a5ce87d5096..f63f290bf58fc 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapper.java @@ -92,9 +92,9 @@ protected Parameter[] getParameters() { @Override public RankFeatureFieldMapper build(MapperBuilderContext context) { return new RankFeatureFieldMapper( - name, + name(), new RankFeatureFieldType( - context.buildFullName(name), + context.buildFullName(name()), meta.getValue(), positiveScoreImpact.getValue(), nullValue.getValue() diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapper.java index f36dfb5605633..5f0d44d1fb796 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapper.java @@ -64,8 +64,8 @@ protected Parameter[] getParameters() { @Override public RankFeaturesFieldMapper build(MapperBuilderContext context) { return new RankFeaturesFieldMapper( - name, - new RankFeaturesFieldType(context.buildFullName(name), meta.getValue(), positiveScoreImpact.getValue()), + name(), + new RankFeaturesFieldType(context.buildFullName(name()), meta.getValue(), positiveScoreImpact.getValue()), multiFieldsBuilder.build(this, context), copyTo, positiveScoreImpact.getValue() diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java index cc2ceb3c017ba..e2b932b01a516 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java @@ -186,7 +186,7 @@ protected Parameter[] getParameters() { @Override public ScaledFloatFieldMapper build(MapperBuilderContext context) { ScaledFloatFieldType type = new ScaledFloatFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.getValue(), stored.getValue(), hasDocValues.getValue(), @@ -196,7 +196,7 @@ public ScaledFloatFieldMapper build(MapperBuilderContext context) { metric.getValue(), indexMode ); - return new ScaledFloatFieldMapper(name, type, multiFieldsBuilder.build(this, context), copyTo, this); + return new ScaledFloatFieldMapper(name(), type, multiFieldsBuilder.build(this, context), copyTo, this); } } diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapper.java index ca8231c46736f..a5e011d5772f0 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapper.java @@ -187,7 +187,7 @@ public SearchAsYouTypeFieldMapper build(MapperBuilderContext context) { NamedAnalyzer searchAnalyzer = analyzers.getSearchAnalyzer(); SearchAsYouTypeFieldType ft = new SearchAsYouTypeFieldType( - context.buildFullName(name), + context.buildFullName(name()), fieldType, similarity.getValue(), analyzers.getSearchAnalyzer(), @@ -202,7 +202,7 @@ public SearchAsYouTypeFieldMapper build(MapperBuilderContext context) { prefixft.setIndexOptions(fieldType.indexOptions()); prefixft.setOmitNorms(true); prefixft.setStored(false); - final String fullName = context.buildFullName(name); + final String fullName = context.buildFullName(name()); // wrap the root field's index analyzer with shingles and edge ngrams final Analyzer prefixIndexWrapper = SearchAsYouTypeAnalyzer.withShingleAndPrefix( indexAnalyzer.analyzer(), @@ -228,7 +228,7 @@ public SearchAsYouTypeFieldMapper build(MapperBuilderContext context) { final int shingleSize = i + 2; FieldType shingleft = new FieldType(fieldType); shingleft.setStored(false); - String fieldName = getShingleFieldName(context.buildFullName(name), shingleSize); + String fieldName = getShingleFieldName(context.buildFullName(name()), shingleSize); // wrap the root field's index, search, and search quote analyzers with shingles final SearchAsYouTypeAnalyzer shingleIndexWrapper = SearchAsYouTypeAnalyzer.withShingle( indexAnalyzer.analyzer(), @@ -260,7 +260,7 @@ public SearchAsYouTypeFieldMapper build(MapperBuilderContext context) { ft.setPrefixField(prefixFieldType); ft.setShingleFields(shingleFieldTypes); return new SearchAsYouTypeFieldMapper( - name, + name(), ft, copyTo, indexAnalyzers, diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapper.java index 4d04e83361252..831306a8e8594 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapper.java @@ -77,17 +77,17 @@ protected Parameter[] getParameters() { @Override public TokenCountFieldMapper build(MapperBuilderContext context) { if (analyzer.getValue() == null) { - throw new MapperParsingException("Analyzer must be set for field [" + name + "] but wasn't."); + throw new MapperParsingException("Analyzer must be set for field [" + name() + "] but wasn't."); } MappedFieldType ft = new TokenCountFieldType( - context.buildFullName(name), + context.buildFullName(name()), index.getValue(), store.getValue(), hasDocValues.getValue(), nullValue.getValue(), meta.getValue() ); - return new TokenCountFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, this); + return new TokenCountFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, this); } } diff --git a/modules/parent-join/src/main/java/org/elasticsearch/join/mapper/ParentJoinFieldMapper.java b/modules/parent-join/src/main/java/org/elasticsearch/join/mapper/ParentJoinFieldMapper.java index 2bbd5e81444b7..d6b7ccad4f3c5 100644 --- a/modules/parent-join/src/main/java/org/elasticsearch/join/mapper/ParentJoinFieldMapper.java +++ b/modules/parent-join/src/main/java/org/elasticsearch/join/mapper/ParentJoinFieldMapper.java @@ -112,16 +112,16 @@ protected Parameter[] getParameters() { @Override public ParentJoinFieldMapper build(MapperBuilderContext context) { - checkObjectOrNested(context, name); + checkObjectOrNested(context, name()); final Map parentIdFields = new HashMap<>(); relations.get() .stream() - .map(relation -> new ParentIdFieldMapper(name + "#" + relation.parent(), eagerGlobalOrdinals.get())) + .map(relation -> new ParentIdFieldMapper(name() + "#" + relation.parent(), eagerGlobalOrdinals.get())) .forEach(mapper -> parentIdFields.put(mapper.name(), mapper)); Joiner joiner = new Joiner(name(), relations.get()); return new ParentJoinFieldMapper( - name, - new JoinFieldType(context.buildFullName(name), joiner, meta.get()), + name(), + new JoinFieldType(context.buildFullName(name()), joiner, meta.get()), Collections.unmodifiableMap(parentIdFields), eagerGlobalOrdinals.get(), relations.get() diff --git a/modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java b/modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java index be8d342254afd..7ba83f9ce71b5 100644 --- a/modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java +++ b/modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java @@ -135,10 +135,10 @@ protected Parameter[] getParameters() { @Override public PercolatorFieldMapper build(MapperBuilderContext context) { - PercolatorFieldType fieldType = new PercolatorFieldType(context.buildFullName(name), meta.getValue()); + PercolatorFieldType fieldType = new PercolatorFieldType(context.buildFullName(name()), meta.getValue()); // TODO should percolator even allow multifields? MultiFields multiFields = multiFieldsBuilder.build(this, context); - context = context.createChildContext(name); + context = context.createChildContext(name()); KeywordFieldMapper extractedTermsField = createExtractQueryFieldBuilder( EXTRACTED_TERMS_FIELD_NAME, context, diff --git a/plugins/analysis-icu/src/main/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapper.java b/plugins/analysis-icu/src/main/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapper.java index 19f1d0455630d..1da274ff236da 100644 --- a/plugins/analysis-icu/src/main/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapper.java +++ b/plugins/analysis-icu/src/main/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapper.java @@ -327,7 +327,7 @@ public ICUCollationKeywordFieldMapper build(MapperBuilderContext context) { final CollatorParams params = collatorParams(); final Collator collator = params.buildCollator(); CollationFieldType ft = new CollationFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.getValue(), stored.getValue(), hasDocValues.getValue(), @@ -337,7 +337,7 @@ public ICUCollationKeywordFieldMapper build(MapperBuilderContext context) { meta.getValue() ); return new ICUCollationKeywordFieldMapper( - name, + name(), buildFieldType(), ft, multiFieldsBuilder.build(this, context), diff --git a/plugins/mapper-annotated-text/src/main/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapper.java b/plugins/mapper-annotated-text/src/main/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapper.java index 7153fcf4d46b3..fae2ab19aee39 100644 --- a/plugins/mapper-annotated-text/src/main/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapper.java +++ b/plugins/mapper-annotated-text/src/main/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapper.java @@ -122,7 +122,7 @@ private AnnotatedTextFieldType buildFieldType(FieldType fieldType, MapperBuilder wrapAnalyzer(analyzers.getSearchQuoteAnalyzer()) ); return new AnnotatedTextFieldType( - context.buildFullName(name), + context.buildFullName(name()), store.getValue(), tsi, context.isSourceSynthetic(), @@ -139,12 +139,12 @@ public AnnotatedTextFieldMapper build(MapperBuilderContext context) { if (analyzers.positionIncrementGap.isConfigured()) { if (fieldType.indexOptions().compareTo(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS) < 0) { throw new IllegalArgumentException( - "Cannot set position_increment_gap on field [" + name + "] without positions enabled" + "Cannot set position_increment_gap on field [" + name() + "] without positions enabled" ); } } return new AnnotatedTextFieldMapper( - name, + name(), fieldType, buildFieldType(fieldType, context), multiFieldsBuilder.build(this, context), diff --git a/plugins/mapper-murmur3/src/main/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapper.java b/plugins/mapper-murmur3/src/main/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapper.java index c1e2888c47c62..08a133bcb69c8 100644 --- a/plugins/mapper-murmur3/src/main/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapper.java +++ b/plugins/mapper-murmur3/src/main/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapper.java @@ -55,8 +55,8 @@ protected Parameter[] getParameters() { @Override public Murmur3FieldMapper build(MapperBuilderContext context) { return new Murmur3FieldMapper( - name, - new Murmur3FieldType(context.buildFullName(name), stored.getValue(), meta.getValue()), + name(), + new Murmur3FieldType(context.buildFullName(name()), stored.getValue(), meta.getValue()), multiFieldsBuilder.build(this, context), copyTo ); diff --git a/server/src/main/java/org/elasticsearch/index/mapper/BinaryFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/BinaryFieldMapper.java index 403156c95540e..948baf0dff830 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/BinaryFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/BinaryFieldMapper.java @@ -64,8 +64,8 @@ public Parameter[] getParameters() { @Override public BinaryFieldMapper build(MapperBuilderContext context) { return new BinaryFieldMapper( - name, - new BinaryFieldType(context.buildFullName(name), stored.getValue(), hasDocValues.getValue(), meta.getValue()), + name(), + new BinaryFieldType(context.buildFullName(name()), stored.getValue(), hasDocValues.getValue(), meta.getValue()), multiFieldsBuilder.build(this, context), copyTo, this diff --git a/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java index 43e6e662dc8f2..cc01a487ad7b8 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java @@ -115,7 +115,7 @@ protected Parameter[] getParameters() { @Override public BooleanFieldMapper build(MapperBuilderContext context) { MappedFieldType ft = new BooleanFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.getValue() && indexCreatedVersion.isLegacyIndexVersion() == false, stored.getValue(), docValues.getValue(), @@ -123,7 +123,7 @@ public BooleanFieldMapper build(MapperBuilderContext context) { scriptValues(), meta.getValue() ); - return new BooleanFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, context.isSourceSynthetic(), this); + return new BooleanFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, context.isSourceSynthetic(), this); } private FieldValues scriptValues() { @@ -133,7 +133,7 @@ private FieldValues scriptValues() { BooleanFieldScript.Factory scriptFactory = scriptCompiler.compile(script.get(), BooleanFieldScript.CONTEXT); return scriptFactory == null ? null - : (lookup, ctx, doc, consumer) -> scriptFactory.newFactory(name, script.get().getParams(), lookup, OnScriptError.FAIL) + : (lookup, ctx, doc, consumer) -> scriptFactory.newFactory(name(), script.get().getParams(), lookup, OnScriptError.FAIL) .newInstance(ctx) .runForDoc(doc, consumer); } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/CompletionFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/CompletionFieldMapper.java index 94b937c534491..5d5ef076852a8 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/CompletionFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/CompletionFieldMapper.java @@ -205,9 +205,9 @@ public CompletionFieldMapper build(MapperBuilderContext context) { new CompletionAnalyzer(this.searchAnalyzer.getValue(), preserveSeparators.getValue(), preservePosInc.getValue()) ); - CompletionFieldType ft = new CompletionFieldType(context.buildFullName(name), completionAnalyzer, meta.getValue()); + CompletionFieldType ft = new CompletionFieldType(context.buildFullName(name()), completionAnalyzer, meta.getValue()); ft.setContextMappings(contexts.getValue()); - return new CompletionFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, this); + return new CompletionFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, this); } private void checkCompletionContextsLimit() { diff --git a/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java index 0c54b58aae0e3..1b926734c1713 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java @@ -306,7 +306,7 @@ private FieldValues scriptValues() { return factory == null ? null : (lookup, ctx, doc, consumer) -> factory.newFactory( - name, + name(), script.get().getParams(), lookup, buildFormatter(), @@ -364,7 +364,7 @@ public DateFieldMapper build(MapperBuilderContext context) { && ignoreMalformed.isConfigured() == false) { ignoreMalformed.setValue(false); } - return new DateFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, nullTimestamp, resolution, this); + return new DateFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, nullTimestamp, resolution, this); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java index 4effc380646ff..85a9b8377e6f0 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java @@ -186,7 +186,7 @@ private FieldValues scriptValues() { GeoPointFieldScript.Factory factory = scriptCompiler.compile(this.script.get(), GeoPointFieldScript.CONTEXT); return factory == null ? null - : (lookup, ctx, doc, consumer) -> factory.newFactory(name, script.get().getParams(), lookup, OnScriptError.FAIL) + : (lookup, ctx, doc, consumer) -> factory.newFactory(name(), script.get().getParams(), lookup, OnScriptError.FAIL) .newInstance(ctx) .runForDoc(doc, consumer); } @@ -194,7 +194,7 @@ private FieldValues scriptValues() { @Override public FieldMapper build(MapperBuilderContext context) { Parser geoParser = new GeoPointParser( - name, + name(), (parser) -> GeoUtils.parseGeoPoint(parser, ignoreZValue.get().value()), nullValue.get(), ignoreZValue.get().value(), @@ -202,7 +202,7 @@ public FieldMapper build(MapperBuilderContext context) { metric.get() != TimeSeriesParams.MetricType.POSITION ); GeoPointFieldType ft = new GeoPointFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get() && indexCreatedVersion.isLegacyIndexVersion() == false, stored.get(), hasDocValues.get(), @@ -214,9 +214,9 @@ public FieldMapper build(MapperBuilderContext context) { indexMode ); if (this.script.get() == null) { - return new GeoPointFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, geoParser, this); + return new GeoPointFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, geoParser, this); } - return new GeoPointFieldMapper(name, ft, geoParser, this); + return new GeoPointFieldMapper(name(), ft, geoParser, this); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/GeoShapeFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/GeoShapeFieldMapper.java index e39684705e26a..541538f65a550 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/GeoShapeFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/GeoShapeFieldMapper.java @@ -99,18 +99,18 @@ public GeoShapeFieldMapper build(MapperBuilderContext context) { ); GeoShapeParser geoShapeParser = new GeoShapeParser(geometryParser, orientation.get().value()); GeoShapeFieldType ft = new GeoShapeFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get(), orientation.get().value(), geoShapeParser, meta.get() ); return new GeoShapeFieldMapper( - name, + name(), ft, multiFieldsBuilder.build(this, context), copyTo, - new GeoShapeIndexer(orientation.get().value(), context.buildFullName(name)), + new GeoShapeIndexer(orientation.get().value(), context.buildFullName(name())), geoShapeParser, this ); diff --git a/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java index 8ce726b49ff66..355b38d4dcb96 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java @@ -154,7 +154,7 @@ private FieldValues scriptValues() { IpFieldScript.Factory factory = scriptCompiler.compile(this.script.get(), IpFieldScript.CONTEXT); return factory == null ? null - : (lookup, ctx, doc, consumer) -> factory.newFactory(name, script.get().getParams(), lookup, OnScriptError.FAIL) + : (lookup, ctx, doc, consumer) -> factory.newFactory(name(), script.get().getParams(), lookup, OnScriptError.FAIL) .newInstance(ctx) .runForDoc(doc, consumer); } @@ -170,9 +170,9 @@ public IpFieldMapper build(MapperBuilderContext context) { dimension.setValue(true); } return new IpFieldMapper( - name, + name(), new IpFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.getValue() && indexCreatedVersion.isLegacyIndexVersion() == false, stored.getValue(), hasDocValues.getValue(), diff --git a/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java index a5a571fb82d85..06e689784b087 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java @@ -254,7 +254,7 @@ private FieldValues scriptValues() { StringFieldScript.Factory scriptFactory = scriptCompiler.compile(script.get(), StringFieldScript.CONTEXT); return scriptFactory == null ? null - : (lookup, ctx, doc, consumer) -> scriptFactory.newFactory(name, script.get().getParams(), lookup, OnScriptError.FAIL) + : (lookup, ctx, doc, consumer) -> scriptFactory.newFactory(name(), script.get().getParams(), lookup, OnScriptError.FAIL) .newInstance(ctx) .runForDoc(doc, consumer); } @@ -294,7 +294,7 @@ private KeywordFieldType buildFieldType(MapperBuilderContext context, FieldType ); normalizer = Lucene.KEYWORD_ANALYZER; } else { - throw new MapperParsingException("normalizer [" + normalizerName + "] not found for field [" + name + "]"); + throw new MapperParsingException("normalizer [" + normalizerName + "] not found for field [" + name() + "]"); } } searchAnalyzer = quoteAnalyzer = normalizer; @@ -308,7 +308,7 @@ private KeywordFieldType buildFieldType(MapperBuilderContext context, FieldType dimension(true); } return new KeywordFieldType( - context.buildFullName(name), + context.buildFullName(name()), fieldType, normalizer, searchAnalyzer, @@ -330,7 +330,7 @@ public KeywordFieldMapper build(MapperBuilderContext context) { fieldtype = Defaults.FIELD_TYPE; } return new KeywordFieldMapper( - name, + name(), fieldtype, buildFieldType(context, fieldtype), multiFieldsBuilder.build(this, context), diff --git a/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java b/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java index 397f99f63030c..cf4025150584f 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java @@ -24,7 +24,7 @@ public abstract class Mapper implements ToXContentFragment, Iterable { public abstract static class Builder { - protected final String name; + private final String name; protected Builder(String name) { this.name = internFieldName(name); diff --git a/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java index a654819811621..1216618b1e986 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java @@ -62,8 +62,8 @@ public NestedObjectMapper build(MapperBuilderContext context) { this.includeInRoot = Explicit.IMPLICIT_FALSE; } } - NestedMapperBuilderContext nestedContext = new NestedMapperBuilderContext(context.buildFullName(name), parentIncludedInRoot); - final String fullPath = context.buildFullName(name); + NestedMapperBuilderContext nestedContext = new NestedMapperBuilderContext(context.buildFullName(name()), parentIncludedInRoot); + final String fullPath = context.buildFullName(name()); final String nestedTypePath; if (indexCreatedVersion.before(IndexVersions.V_8_0_0)) { nestedTypePath = "__" + fullPath; @@ -71,7 +71,7 @@ public NestedObjectMapper build(MapperBuilderContext context) { nestedTypePath = fullPath; } return new NestedObjectMapper( - name, + name(), fullPath, buildMappers(nestedContext), enabled, diff --git a/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java index 5935eaf2c3d14..2245e527c2aa2 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java @@ -227,7 +227,7 @@ private FieldValues scriptValues() { if (this.script.get() == null) { return null; } - return type.compile(name, script.get(), scriptCompiler); + return type.compile(name(), script.get(), scriptCompiler); } public Builder dimension(boolean dimension) { @@ -271,8 +271,8 @@ public NumberFieldMapper build(MapperBuilderContext context) { dimension.setValue(true); } - MappedFieldType ft = new NumberFieldType(context.buildFullName(name), this); - return new NumberFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, context.isSourceSynthetic(), this); + MappedFieldType ft = new NumberFieldType(context.buildFullName(name()), this); + return new NumberFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, context.isSourceSynthetic(), this); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java index 7a807f767611b..a9de4bdd1467a 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java @@ -172,12 +172,12 @@ protected final Map buildMappers(MapperBuilderContext mapperBuil @Override public ObjectMapper build(MapperBuilderContext context) { return new ObjectMapper( - name, - context.buildFullName(name), + name(), + context.buildFullName(name()), enabled, subobjects, dynamic, - buildMappers(context.createChildContext(name)) + buildMappers(context.createChildContext(name())) ); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/PassThroughObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/PassThroughObjectMapper.java index b49c9328fcc79..4ce7f51ed7386 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/PassThroughObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/PassThroughObjectMapper.java @@ -56,11 +56,11 @@ public PassThroughObjectMapper.Builder setContainsDimensions() { @Override public PassThroughObjectMapper build(MapperBuilderContext context) { return new PassThroughObjectMapper( - name, - context.buildFullName(name), + name(), + context.buildFullName(name()), enabled, dynamic, - buildMappers(context.createChildContext(name)), + buildMappers(context.createChildContext(name())), timeSeriesDimensionSubFields ); } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/PlaceHolderFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/PlaceHolderFieldMapper.java index 98f8f21be704a..67260273bc5a5 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/PlaceHolderFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/PlaceHolderFieldMapper.java @@ -90,8 +90,8 @@ protected Parameter[] getParameters() { @Override public PlaceHolderFieldMapper build(MapperBuilderContext context) { - PlaceHolderFieldType mappedFieldType = new PlaceHolderFieldType(context.buildFullName(name), type, Map.of()); - return new PlaceHolderFieldMapper(name, mappedFieldType, multiFieldsBuilder.build(this, context), copyTo, unknownParams); + PlaceHolderFieldType mappedFieldType = new PlaceHolderFieldType(context.buildFullName(name()), type, Map.of()); + return new PlaceHolderFieldMapper(name(), mappedFieldType, multiFieldsBuilder.build(this, context), copyTo, unknownParams); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/RangeFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/RangeFieldMapper.java index fcd2a425a6625..3836915e65753 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/RangeFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/RangeFieldMapper.java @@ -116,7 +116,7 @@ protected Parameter[] getParameters() { } protected RangeFieldType setupFieldType(MapperBuilderContext context) { - String fullName = context.buildFullName(name); + String fullName = context.buildFullName(name()); if (format.isConfigured()) { if (type != RangeType.DATE) { throw new IllegalArgumentException( @@ -163,7 +163,7 @@ protected RangeFieldType setupFieldType(MapperBuilderContext context) { @Override public RangeFieldMapper build(MapperBuilderContext context) { RangeFieldType ft = setupFieldType(context); - return new RangeFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, type, this); + return new RangeFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, type, this); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java index d7cc9e8f7e71f..a730d8c2da89e 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java @@ -114,7 +114,7 @@ public RootObjectMapper build(MapperBuilderContext context) { Map mappers = buildMappers(context); mappers.putAll(getAliasMappers(mappers, context)); return new RootObjectMapper( - name, + name(), enabled, subobjects, dynamic, diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java index 1885869073711..faa840dacc732 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java @@ -355,18 +355,18 @@ private TextFieldType buildFieldType( if (analyzers.positionIncrementGap.isConfigured()) { if (fieldType.indexOptions().compareTo(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS) < 0) { throw new IllegalArgumentException( - "Cannot set position_increment_gap on field [" + name + "] without positions enabled" + "Cannot set position_increment_gap on field [" + name() + "] without positions enabled" ); } } TextSearchInfo tsi = new TextSearchInfo(fieldType, similarity.getValue(), searchAnalyzer, searchQuoteAnalyzer); TextFieldType ft; if (indexCreatedVersion.isLegacyIndexVersion()) { - ft = new LegacyTextFieldType(context.buildFullName(name), index.getValue(), store.getValue(), tsi, meta.getValue()); + ft = new LegacyTextFieldType(context.buildFullName(name()), index.getValue(), store.getValue(), tsi, meta.getValue()); // ignore fieldData and eagerGlobalOrdinals } else { ft = new TextFieldType( - context.buildFullName(name), + context.buildFullName(name()), index.getValue(), store.getValue(), tsi, @@ -412,7 +412,7 @@ private SubFieldInfo buildPrefixInfo(MapperBuilderContext context, FieldType fie * or a multi-field). This way search will continue to work on old indices and new indices * will use the expected full name. */ - String fullName = indexCreatedVersion.before(IndexVersions.V_7_2_1) ? name() : context.buildFullName(name); + String fullName = indexCreatedVersion.before(IndexVersions.V_7_2_1) ? name() : context.buildFullName(name()); // Copy the index options of the main field to allow phrase queries on // the prefix field. FieldType pft = new FieldType(fieldType); @@ -476,7 +476,7 @@ public TextFieldMapper build(MapperBuilderContext context) { throw new MapperParsingException("Cannot use reserved field name [" + mapper.name() + "]"); } } - return new TextFieldMapper(name, fieldType, tft, prefixFieldInfo, phraseFieldInfo, multiFields, copyTo, this); + return new TextFieldMapper(name(), fieldType, tft, prefixFieldInfo, phraseFieldInfo, multiFields, copyTo, this); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapper.java index c15adfb3be116..5a8efb6c8ed59 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapper.java @@ -202,13 +202,13 @@ protected Parameter[] getParameters() { public FlattenedFieldMapper build(MapperBuilderContext context) { MultiFields multiFields = multiFieldsBuilder.build(this, context); if (multiFields.iterator().hasNext()) { - throw new IllegalArgumentException(CONTENT_TYPE + " field [" + name + "] does not support [fields]"); + throw new IllegalArgumentException(CONTENT_TYPE + " field [" + name() + "] does not support [fields]"); } if (copyTo.copyToFields().isEmpty() == false) { - throw new IllegalArgumentException(CONTENT_TYPE + " field [" + name + "] does not support [copy_to]"); + throw new IllegalArgumentException(CONTENT_TYPE + " field [" + name() + "] does not support [copy_to]"); } MappedFieldType ft = new RootFlattenedFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get(), hasDocValues.get(), meta.get(), @@ -216,7 +216,7 @@ public FlattenedFieldMapper build(MapperBuilderContext context) { eagerGlobalOrdinals.get(), dimensions.get() ); - return new FlattenedFieldMapper(name, ft, this); + return new FlattenedFieldMapper(name(), ft, this); } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java index d36ca9e0b25c1..598a6383bfdaa 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java @@ -213,9 +213,9 @@ protected Parameter[] getParameters() { @Override public DenseVectorFieldMapper build(MapperBuilderContext context) { return new DenseVectorFieldMapper( - name, + name(), new DenseVectorFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexVersionCreated, elementType.getValue(), dims.getValue(), diff --git a/server/src/main/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapper.java index 3b892fc1647b6..6532abed19044 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/vectors/SparseVectorFieldMapper.java @@ -66,8 +66,8 @@ protected Parameter[] getParameters() { @Override public SparseVectorFieldMapper build(MapperBuilderContext context) { return new SparseVectorFieldMapper( - name, - new SparseVectorFieldType(context.buildFullName(name), meta.getValue()), + name(), + new SparseVectorFieldType(context.buildFullName(name()), meta.getValue()), multiFieldsBuilder.build(this, context), copyTo ); diff --git a/server/src/test/java/org/elasticsearch/index/mapper/ParametrizedMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/ParametrizedMapperTests.java index 562a30ba4f389..b1b7f80ba865f 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/ParametrizedMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/ParametrizedMapperTests.java @@ -175,7 +175,7 @@ protected Parameter[] getParameters() { @Override public FieldMapper build(MapperBuilderContext context) { - return new TestMapper(name(), context.buildFullName(name), multiFieldsBuilder.build(this, context), copyTo, this); + return new TestMapper(name(), context.buildFullName(name()), multiFieldsBuilder.build(this, context), copyTo, this); } } diff --git a/x-pack/plugin/analytics/src/main/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapper.java b/x-pack/plugin/analytics/src/main/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapper.java index 421973723837d..b8e4f77f7da7b 100644 --- a/x-pack/plugin/analytics/src/main/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapper.java +++ b/x-pack/plugin/analytics/src/main/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapper.java @@ -94,8 +94,8 @@ protected Parameter[] getParameters() { @Override public HistogramFieldMapper build(MapperBuilderContext context) { return new HistogramFieldMapper( - name, - new HistogramFieldType(context.buildFullName(name), meta.getValue()), + name(), + new HistogramFieldType(context.buildFullName(name()), meta.getValue()), multiFieldsBuilder.build(this, context), copyTo, this diff --git a/x-pack/plugin/mapper-aggregate-metric/src/main/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java b/x-pack/plugin/mapper-aggregate-metric/src/main/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java index b5c35e758a65c..1581803920cdc 100644 --- a/x-pack/plugin/mapper-aggregate-metric/src/main/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java +++ b/x-pack/plugin/mapper-aggregate-metric/src/main/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java @@ -209,7 +209,7 @@ public AggregateDoubleMetricFieldMapper build(MapperBuilderContext context) { EnumMap metricMappers = new EnumMap<>(Metric.class); // Instantiate one NumberFieldMapper instance for each metric for (Metric m : this.metrics.getValue()) { - String fieldName = subfieldName(name, m); + String fieldName = subfieldName(name(), m); NumberFieldMapper.Builder builder; if (m == Metric.value_count) { @@ -245,14 +245,14 @@ public AggregateDoubleMetricFieldMapper build(MapperBuilderContext context) { }, () -> new EnumMap<>(Metric.class))); AggregateDoubleMetricFieldType metricFieldType = new AggregateDoubleMetricFieldType( - context.buildFullName(name), + context.buildFullName(name()), meta.getValue(), timeSeriesMetric.getValue() ); metricFieldType.setMetricFields(metricFields); metricFieldType.setDefaultMetric(defaultMetric.getValue()); - return new AggregateDoubleMetricFieldMapper(name, metricFieldType, metricMappers, this); + return new AggregateDoubleMetricFieldMapper(name(), metricFieldType, metricMappers, this); } } diff --git a/x-pack/plugin/mapper-constant-keyword/src/main/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapper.java b/x-pack/plugin/mapper-constant-keyword/src/main/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapper.java index cee397d906149..f2b1f013212db 100644 --- a/x-pack/plugin/mapper-constant-keyword/src/main/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapper.java +++ b/x-pack/plugin/mapper-constant-keyword/src/main/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapper.java @@ -99,8 +99,8 @@ protected Parameter[] getParameters() { @Override public ConstantKeywordFieldMapper build(MapperBuilderContext context) { return new ConstantKeywordFieldMapper( - name, - new ConstantKeywordFieldType(context.buildFullName(name), value.getValue(), meta.getValue()) + name(), + new ConstantKeywordFieldType(context.buildFullName(name()), value.getValue(), meta.getValue()) ); } } diff --git a/x-pack/plugin/mapper-counted-keyword/src/main/java/org/elasticsearch/xpack/countedkeyword/CountedKeywordFieldMapper.java b/x-pack/plugin/mapper-counted-keyword/src/main/java/org/elasticsearch/xpack/countedkeyword/CountedKeywordFieldMapper.java index ad5e224efd5db..878a949a69841 100644 --- a/x-pack/plugin/mapper-counted-keyword/src/main/java/org/elasticsearch/xpack/countedkeyword/CountedKeywordFieldMapper.java +++ b/x-pack/plugin/mapper-counted-keyword/src/main/java/org/elasticsearch/xpack/countedkeyword/CountedKeywordFieldMapper.java @@ -289,14 +289,14 @@ protected Parameter[] getParameters() { @Override public FieldMapper build(MapperBuilderContext context) { - BinaryFieldMapper countFieldMapper = new BinaryFieldMapper.Builder(name + COUNT_FIELD_NAME_SUFFIX, true).build(context); + BinaryFieldMapper countFieldMapper = new BinaryFieldMapper.Builder(name() + COUNT_FIELD_NAME_SUFFIX, true).build(context); boolean isIndexed = indexed.getValue(); FieldType ft = isIndexed ? FIELD_TYPE_INDEXED : FIELD_TYPE_NOT_INDEXED; return new CountedKeywordFieldMapper( - name, + name(), ft, new CountedKeywordFieldType( - context.buildFullName(name), + context.buildFullName(name()), isIndexed, false, true, diff --git a/x-pack/plugin/mapper-unsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapper.java b/x-pack/plugin/mapper-unsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapper.java index c468d7bcd6718..955d658b01bab 100644 --- a/x-pack/plugin/mapper-unsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapper.java +++ b/x-pack/plugin/mapper-unsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapper.java @@ -199,7 +199,7 @@ public UnsignedLongFieldMapper build(MapperBuilderContext context) { dimension.setValue(true); } UnsignedLongFieldType fieldType = new UnsignedLongFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.getValue(), stored.getValue(), hasDocValues.getValue(), @@ -209,7 +209,7 @@ public UnsignedLongFieldMapper build(MapperBuilderContext context) { metric.getValue(), indexMode ); - return new UnsignedLongFieldMapper(name, fieldType, multiFieldsBuilder.build(this, context), copyTo, this); + return new UnsignedLongFieldMapper(name(), fieldType, multiFieldsBuilder.build(this, context), copyTo, this); } } diff --git a/x-pack/plugin/mapper-version/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java b/x-pack/plugin/mapper-version/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java index e233df2af3fbd..40b8bcf208a2d 100644 --- a/x-pack/plugin/mapper-version/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java +++ b/x-pack/plugin/mapper-version/src/main/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapper.java @@ -112,14 +112,14 @@ static class Builder extends FieldMapper.Builder { } private VersionStringFieldType buildFieldType(MapperBuilderContext context, FieldType fieldtype) { - return new VersionStringFieldType(context.buildFullName(name), fieldtype, meta.getValue()); + return new VersionStringFieldType(context.buildFullName(name()), fieldtype, meta.getValue()); } @Override public VersionStringFieldMapper build(MapperBuilderContext context) { FieldType fieldtype = new FieldType(Defaults.FIELD_TYPE); return new VersionStringFieldMapper( - name, + name(), fieldtype, buildFieldType(context, fieldtype), multiFieldsBuilder.build(this, context), diff --git a/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapper.java b/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapper.java index 71fb9b0f3126a..a8f437f476ada 100644 --- a/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapper.java +++ b/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapper.java @@ -173,7 +173,7 @@ private FieldValues scriptValues() { GeometryFieldScript.Factory factory = scriptCompiler.compile(this.script.get(), GeometryFieldScript.CONTEXT); return factory == null ? null - : (lookup, ctx, doc, consumer) -> factory.newFactory(name, script.get().getParams(), lookup, OnScriptError.FAIL) + : (lookup, ctx, doc, consumer) -> factory.newFactory(name(), script.get().getParams(), lookup, OnScriptError.FAIL) .newInstance(ctx) .runForDoc(doc, consumer); } @@ -194,7 +194,7 @@ public GeoShapeWithDocValuesFieldMapper build(MapperBuilderContext context) { ); GeoShapeParser parser = new GeoShapeParser(geometryParser, orientation.get().value()); GeoShapeWithDocValuesFieldType ft = new GeoShapeWithDocValuesFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get(), hasDocValues.get(), stored.get(), @@ -206,7 +206,7 @@ public GeoShapeWithDocValuesFieldMapper build(MapperBuilderContext context) { ); if (script.get() == null) { return new GeoShapeWithDocValuesFieldMapper( - name, + name(), ft, multiFieldsBuilder.build(this, context), copyTo, @@ -216,7 +216,7 @@ public GeoShapeWithDocValuesFieldMapper build(MapperBuilderContext context) { ); } return new GeoShapeWithDocValuesFieldMapper( - name, + name(), ft, multiFieldsBuilder.build(this, context), copyTo, diff --git a/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapper.java b/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapper.java index 01a2b5f0e5598..1657a3bf7fbce 100644 --- a/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapper.java +++ b/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapper.java @@ -105,14 +105,14 @@ public FieldMapper build(MapperBuilderContext context) { ); } CartesianPointParser parser = new CartesianPointParser( - name, + name(), p -> CartesianPoint.parsePoint(p, ignoreZValue.get().value()), nullValue.get(), ignoreZValue.get().value(), ignoreMalformed.get().value() ); PointFieldType ft = new PointFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get(), stored.get(), hasDocValues.get(), @@ -120,7 +120,7 @@ public FieldMapper build(MapperBuilderContext context) { nullValue.get(), meta.get() ); - return new PointFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, parser, this); + return new PointFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, parser, this); } } diff --git a/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapper.java b/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapper.java index 0a1c0278d88d7..83e434f829591 100644 --- a/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapper.java +++ b/x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapper.java @@ -118,14 +118,14 @@ public ShapeFieldMapper build(MapperBuilderContext context) { ); Parser parser = new ShapeParser(geometryParser); ShapeFieldType ft = new ShapeFieldType( - context.buildFullName(name), + context.buildFullName(name()), indexed.get(), hasDocValues.get(), orientation.get().value(), parser, meta.get() ); - return new ShapeFieldMapper(name, ft, multiFieldsBuilder.build(this, context), copyTo, parser, this); + return new ShapeFieldMapper(name(), ft, multiFieldsBuilder.build(this, context), copyTo, parser, this); } } diff --git a/x-pack/plugin/wildcard/src/main/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapper.java b/x-pack/plugin/wildcard/src/main/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapper.java index 1954e291b1a7f..62306a18d946b 100644 --- a/x-pack/plugin/wildcard/src/main/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapper.java +++ b/x-pack/plugin/wildcard/src/main/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapper.java @@ -240,8 +240,8 @@ Builder nullValue(String nullValue) { @Override public WildcardFieldMapper build(MapperBuilderContext context) { return new WildcardFieldMapper( - name, - new WildcardFieldType(context.buildFullName(name), nullValue.get(), ignoreAbove.get(), indexVersionCreated, meta.get()), + name(), + new WildcardFieldType(context.buildFullName(name()), nullValue.get(), ignoreAbove.get(), indexVersionCreated, meta.get()), ignoreAbove.get(), context.isSourceSynthetic(), multiFieldsBuilder.build(this, context),