From a7d4567b9b5296d919a8dbe818f711d3ea4d8844 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 9 Apr 2024 15:20:41 -0600 Subject: [PATCH 001/166] Add mapping as part of content gap initiative Signed-off-by: Melissa Vagi --- _mapping/index.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 _mapping/index.md diff --git a/_mapping/index.md b/_mapping/index.md new file mode 100644 index 0000000000..e69de29bb2 From 7b9aa58215caffe686c0816f12ed222bda99a48f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 9 Apr 2024 15:26:29 -0600 Subject: [PATCH 002/166] Update mapping as part of content gap initiative Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- _mapping/index.md | 0 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 _mapping/index.md diff --git a/_field-types/index.md b/_field-types/index.md index 7a7e816ada..94ef9e268d 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -14,7 +14,7 @@ redirect_from: You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/). -If you're just starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings. +If you're starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings. For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers. diff --git a/_mapping/index.md b/_mapping/index.md deleted file mode 100644 index e69de29bb2..0000000000 From 3ad5c25432c6c495b295d4bb6c0d16ba81bb49fd Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 9 Apr 2024 18:04:11 -0600 Subject: [PATCH 003/166] Add content to address feedback Signed-off-by: Melissa Vagi --- _field-types/index.md | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 94ef9e268d..4473604e9c 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -18,14 +18,8 @@ If you're starting to build out your cluster and data, you may not know exactly For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers. -This section provides an example for how to create an index mapping and how to add a document to it that will get ip_range validated. +This documentation provides an example for how to create an index mapping and how to add a document to it that will get `ip_range` validated. -#### Table of contents -1. TOC -{:toc} - - ---- ## Dynamic mapping When you index a document, OpenSearch adds fields automatically with dynamic mapping. You can also explicitly add fields to an index mapping. @@ -88,6 +82,41 @@ POST sample-index1/_mapping You cannot change the mapping of an existing field, you can only modify the field's mapping parameters. {: .note} +## Mapping parameters + +Mapping parameters are used to configure the behavior of fields in an index. The following table lists commonly used mapping parameters. + +Parameter | Description +:--- | :--- +`analyzer` | Specifies the analyzer used to analyze string fields. +`boost` | Specifies a field-level query time to boost. +`coerce` | Tries to convert the value to the specified data type. +`copy_to` | Copies the values of this field to another field. +`doc_values` | Specifies whether the field should be stored on disk to make sorting and aggregation faster. +`dynamic` | Determines whether new fields should be added dynamically. +`enabled` | Specifies whether the field is enabled or disabled. +`format` | Specifies the date format for date fields. +`ignore_above` | Skips indexing values that are longer than the specified length. +`ignore_malformed` | Specifies whether malformed values should be ignored. +`index` | Specifies whether the field should be indexed. +`index_options` | Specifies what information should be stored in the index for scoring purposes. + +## Mapping limit settings + +OpenSearch may have certain limits or settings related to mappings, such as the maximum number of fields allowed in an index or the maximum depth of nested objects. These settings can be configured based on your requirements. <_Need SME input about mapping limits settings, their default values, and how to configure them if needed._> + +## Removal of mapping types + +<_Need SME input. Does OpenSearch allow for defining mapping types within an index? If so, SME needs to provide information to include in documentation. Do we have guidance on how to structure data without relying on mapping types?_> + +## Runtime fields + +You can define fields at query time, rather than at index time, by using runtime fields. this can be useful for creating fields based on the values of other fields, or for performing transformations on data during the query process. Runtime fields are defined in the query itself and do not affect the underlying data in the index. + +## Metadata fields + +OpenSearch has several built-in metadata fields that provide information about the documents in an index. These fields include `_doc_count`, `field_names`, `_ignored`, `_id`, `_index`, `_meta`, `_routing`, `_source`, `_tier`, and `_type`. These fields can be accessed or used in queries as needed. + --- ## Mapping example usage From 48d3a6ee608b66efdfed00e7de8e6135b8dea7a6 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 15 Apr 2024 15:14:45 -0600 Subject: [PATCH 004/166] Add delete mapping section Signed-off-by: Melissa Vagi --- _field-types/index.md | 47 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 4473604e9c..1934a6840e 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -24,7 +24,7 @@ This documentation provides an example for how to create an index mapping and ho When you index a document, OpenSearch adds fields automatically with dynamic mapping. You can also explicitly add fields to an index mapping. -#### Dynamic mapping types +### Dynamic mapping types Type | Description :--- | :--- @@ -57,7 +57,7 @@ PUT sample-index1 } ``` -### Response +#### Response ```json { "acknowledged": true, @@ -105,7 +105,7 @@ Parameter | Description OpenSearch may have certain limits or settings related to mappings, such as the maximum number of fields allowed in an index or the maximum depth of nested objects. These settings can be configured based on your requirements. <_Need SME input about mapping limits settings, their default values, and how to configure them if needed._> -## Removal of mapping types +## Removing mapping types <_Need SME input. Does OpenSearch allow for defining mapping types within an index? If so, SME needs to provide information to include in documentation. Do we have guidance on how to structure data without relying on mapping types?_> @@ -115,7 +115,20 @@ You can define fields at query time, rather than at index time, by using runtime ## Metadata fields -OpenSearch has several built-in metadata fields that provide information about the documents in an index. These fields include `_doc_count`, `field_names`, `_ignored`, `_id`, `_index`, `_meta`, `_routing`, `_source`, `_tier`, and `_type`. These fields can be accessed or used in queries as needed. +OpenSearch has built-in metadata fields that provide information about the documents in an index. These fields can be accessed or used in queries as needed. + +Metadata fields | Description +:--- | :--- +`_doc_count` | +`field_names` | +`_ignored` | +`_id` | +`_index` | +`_meta` | +`_routing` | +`_source` | +`_tier` | +`_type` | --- ## Mapping example usage @@ -200,7 +213,7 @@ To get all mappings for one or more indexes, use the following request: GET /_mapping ``` -In the above request, `` may be an index name or a comma-separated list of index names. +In the previous request, `` may be an index name or a comma-separated list of index names. To get all mappings for all indexes, use the following request: @@ -249,3 +262,27 @@ The response contains the specified fields: } } ``` + +## Delete a mapping + +The syntax for deleting a mapping depends on whether you want to delete the entire mapping for an index or the mapping for a specific field. The syntax for deleting a mapping is as follows: + +```json +DELETE //_mapping +DELETE / +``` + +For example, to delete the entire mapping for the `sample-index1` index, you can use the following commands: + +```json + +``` + +If you want to delete the mapping for a specific field, you can For example, to delete the mapping for the `year` field, use the following command: + +```json + +``` + +Deleting a field mapping will remove the mapping definition for that field across all indexes or the specified index. It will not delete the actual data stored in those fields. +{: .note} From 8722b151c4a7b1b273264e053b4b922bc642524e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 15 Apr 2024 15:57:33 -0600 Subject: [PATCH 005/166] Update index.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/index.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 1934a6840e..e570a22008 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -103,7 +103,9 @@ Parameter | Description ## Mapping limit settings -OpenSearch may have certain limits or settings related to mappings, such as the maximum number of fields allowed in an index or the maximum depth of nested objects. These settings can be configured based on your requirements. <_Need SME input about mapping limits settings, their default values, and how to configure them if needed._> +OpenSearch may have certain limits or settings related to mappings, such as the maximum number of fields allowed in an index or the maximum depth of nested objects. These settings can be configured based on your requirements. + +<_Need SME input about mapping limits settings, their default values, and how to configure them if needed._> ## Removing mapping types @@ -119,16 +121,13 @@ OpenSearch has built-in metadata fields that provide information about the docum Metadata fields | Description :--- | :--- -`_doc_count` | -`field_names` | +`field_names` | `_ignored` | `_id` | `_index` | `_meta` | `_routing` | `_source` | -`_tier` | -`_type` | --- ## Mapping example usage From b2dfc7aa43fcfdcba08fa59c22e65bcb909c5310 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 15 Apr 2024 16:57:32 -0600 Subject: [PATCH 006/166] Add metadata fields index page Signed-off-by: Melissa Vagi --- _field-types/index.md | 14 -------------- _field-types/metadata-fields/index.md | 21 +++++++++++++++++++++ 2 files changed, 21 insertions(+), 14 deletions(-) create mode 100644 _field-types/metadata-fields/index.md diff --git a/_field-types/index.md b/_field-types/index.md index e570a22008..94e0f36a01 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -115,20 +115,6 @@ OpenSearch may have certain limits or settings related to mappings, such as the You can define fields at query time, rather than at index time, by using runtime fields. this can be useful for creating fields based on the values of other fields, or for performing transformations on data during the query process. Runtime fields are defined in the query itself and do not affect the underlying data in the index. -## Metadata fields - -OpenSearch has built-in metadata fields that provide information about the documents in an index. These fields can be accessed or used in queries as needed. - -Metadata fields | Description -:--- | :--- -`field_names` | -`_ignored` | -`_id` | -`_index` | -`_meta` | -`_routing` | -`_source` | - --- ## Mapping example usage diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md new file mode 100644 index 0000000000..ac7a6b6e32 --- /dev/null +++ b/_field-types/metadata-fields/index.md @@ -0,0 +1,21 @@ +-- +layout: default +title: Metadata fields +nav_order: 90 +has_children: true +has_toc: false +--- + +# Metadata fields + +OpenSearch has built-in metadata fields that provide information about the documents in an index. These fields can be accessed or used in queries as needed. + +Metadata fields | Description +:--- | :--- +`field_names` | The fields within the document that hold non-empty or non-null values. +`_ignored` | The fields in the document that were disregarded during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. +`_id` | The unique identifier assigned to each individual document. +`_index` | The specific index within the OpenSearch database where the document is stored and organized. +`_meta` | Stores custom metadata or additional information specific to the application or use case. +`_routing` | Allows you to specify a custom value that determines the shard assignment for the document within the OpenSearch cluster. +`_source` | Contains the original JSON representation of the document's data. From cf4a80c385076a840340bcef202041b3a7376c46 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Apr 2024 10:07:45 -0600 Subject: [PATCH 007/166] Add setting descriptions Signed-off-by: Melissa Vagi --- _field-types/index.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 94e0f36a01..44d79f4470 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -103,9 +103,16 @@ Parameter | Description ## Mapping limit settings -OpenSearch may have certain limits or settings related to mappings, such as the maximum number of fields allowed in an index or the maximum depth of nested objects. These settings can be configured based on your requirements. - -<_Need SME input about mapping limits settings, their default values, and how to configure them if needed._> +OpenSearch has certain limits or settings related to mappings, such as the settings listed in the following table. Settings can be configured based on your requirements. + +| Setting | Default value | Allowed value | Type | Description | +|-|-|-|-|-| +| index.mapping.nested_fields.limit | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. | +| index.mapping.nested_objects.limit | 10000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created within a single document. | +| index.mapping.total_fields.limit | 1000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | +| index.mapping.depth.limit | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | +| index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | +| index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | ## Removing mapping types From 061cc805998105a01250bbee4e0ce0ec5b8b1aa0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Apr 2024 10:11:34 -0600 Subject: [PATCH 008/166] Delete remove mappings type section Signed-off-by: Melissa Vagi --- _field-types/index.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 44d79f4470..6740e8650b 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -114,10 +114,6 @@ OpenSearch has certain limits or settings related to mappings, such as the setti | index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | | index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | -## Removing mapping types - -<_Need SME input. Does OpenSearch allow for defining mapping types within an index? If so, SME needs to provide information to include in documentation. Do we have guidance on how to structure data without relying on mapping types?_> - ## Runtime fields You can define fields at query time, rather than at index time, by using runtime fields. this can be useful for creating fields based on the values of other fields, or for performing transformations on data during the query process. Runtime fields are defined in the query itself and do not affect the underlying data in the index. From a047b4c9fdbc10731e9aaf3fe316ad4e00c3d035 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Apr 2024 10:54:29 -0600 Subject: [PATCH 009/166] Add individual metadata field docs Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 9 +++++++++ _field-types/metadata-fields/id.md | 7 +++++++ _field-types/metadata-fields/ignored.md | 9 +++++++++ _field-types/metadata-fields/index-metadata.md | 7 +++++++ _field-types/metadata-fields/meta.md | 7 +++++++ _field-types/metadata-fields/routing.md | 7 +++++++ _field-types/metadata-fields/source.md | 8 ++++++++ 7 files changed, 54 insertions(+) create mode 100644 _field-types/metadata-fields/field-names.md create mode 100644 _field-types/metadata-fields/id.md create mode 100644 _field-types/metadata-fields/ignored.md create mode 100644 _field-types/metadata-fields/index-metadata.md create mode 100644 _field-types/metadata-fields/meta.md create mode 100644 _field-types/metadata-fields/routing.md create mode 100644 _field-types/metadata-fields/source.md diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md new file mode 100644 index 0000000000..ca843b6b7c --- /dev/null +++ b/_field-types/metadata-fields/field-names.md @@ -0,0 +1,9 @@ +--- +layout: default +title: Field names +nav_order: 10 +has_children: false +parent: Metadata fields +--- + +# Field names diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md new file mode 100644 index 0000000000..78fbf6cf4d --- /dev/null +++ b/_field-types/metadata-fields/id.md @@ -0,0 +1,7 @@ +--- +layout: default +title: Ignored +nav_order: 20 +has_children: false +parent: Metadata fields +--- \ No newline at end of file diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md new file mode 100644 index 0000000000..d57dfe0be0 --- /dev/null +++ b/_field-types/metadata-fields/ignored.md @@ -0,0 +1,9 @@ +--- +layout: default +title: Ignored +nav_order: 15 +has_children: false +parent: Metadata fields +--- + +# Ignored diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md new file mode 100644 index 0000000000..58c75ce273 --- /dev/null +++ b/_field-types/metadata-fields/index-metadata.md @@ -0,0 +1,7 @@ +--- +layout: default +title: Ignored +nav_order: 25 +has_children: false +parent: Metadata fields +--- \ No newline at end of file diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md new file mode 100644 index 0000000000..527b5436f4 --- /dev/null +++ b/_field-types/metadata-fields/meta.md @@ -0,0 +1,7 @@ +--- +layout: default +title: Ignored +nav_order: 30 +has_children: false +parent: Metadata fields +--- \ No newline at end of file diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md new file mode 100644 index 0000000000..161f5a4382 --- /dev/null +++ b/_field-types/metadata-fields/routing.md @@ -0,0 +1,7 @@ +--- +layout: default +title: Ignored +nav_order: 35 +has_children: false +parent: Metadata fields +--- \ No newline at end of file diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md new file mode 100644 index 0000000000..824e3fa82f --- /dev/null +++ b/_field-types/metadata-fields/source.md @@ -0,0 +1,8 @@ +--- +layout: default +title: Ignored +nav_order: 40 +has_children: false +parent: Metadata fields +--- + From 0b4bf5efcfdad967d211b5c597b4a11cb011dca1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Apr 2024 13:14:22 -0600 Subject: [PATCH 010/166] Added documentation for field names and ignored Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 9 ++++ _field-types/metadata-fields/ignored.md | 60 +++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index ca843b6b7c..acded5205f 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -7,3 +7,12 @@ parent: Metadata fields --- # Field names + +The `field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. + +The `field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `field_names`. + +## Mapping example + + + diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index d57dfe0be0..8ee73751d1 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -7,3 +7,63 @@ parent: Metadata fields --- # Ignored + +The `_ignored` field indexes and stores the name of fields within a document that were ignored during the indexing process due to being malformed. This functionality is enabled with the `ignore_malformed` setting is turned on in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/#mapping-example-usage). + +The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understadning issues related to malformed data in your documents. + +You can query the `_ignored` field using `term`, `terms`, and `exists` queries, and the results will be in the search hits. + +The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated. +{: .note} + +For example, the following query will retrieve all documents that have at least one field that was ignored during indexing: + +```json +GET _search +{ + "query": { + "exists": { + "field": "_ignored" + } + } +} +``` +{% include copy-curl.html %} + +Similarly, you can use a term query to find documents where a specific field, such as created_at, was ignored: + +```json +GET _search +{ + "query": { + "term": { + "_ignored": "created_at" + } + } +} +``` +{% include copy-curl.html %} + +#### Reponse + +```json +{ + "took": 51, + "timed_out": false, + "_shards": { + "total": 45, + "successful": 45, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 0, + "relation": "eq" + }, + "max_score": null, + "hits": [] + } +} +``` \ No newline at end of file From bd3b85c7b68b240285ebde9b7e0e630a83bf4071 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Apr 2024 13:36:16 -0600 Subject: [PATCH 011/166] Add id field doc Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 80 +++++++++++++++++++++++++++++- 1 file changed, 78 insertions(+), 2 deletions(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 78fbf6cf4d..8ef19922ba 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -1,7 +1,83 @@ --- layout: default -title: Ignored +title: ID nav_order: 20 has_children: false parent: Metadata fields ---- \ No newline at end of file +--- + +# ID + +Each document has an `_id` field that uniquely identifies it. This field is indexed, allowing documents to be retrieved either through the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). + +The following examples creates an index `test-index1` and add two documents with different `_id` values: + +```json +PUT test-index1/_doc/1 +{ + "text": "Document with ID 1" +} + +PUT test-index1/_doc/2?refresh=true +{ + "text": "Document with ID 2" +} +``` +{% include copy-curl.html %} + +Now, you can query the documents using the `_id` field: + +```json +GET test-index1/_search +{ + "query": { + "terms": { + "_id": ["1", "2"] + } + } +} +``` +{% include copy-curl.html %} + +The following response shows that this query returns both documents with `_id` values of `1` and `2`. + +```json +{ + "took": 10, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "test-index1", + "_id": "1", + "_score": 1, + "_source": { + "text": "Document with ID 1" + } + }, + { + "_index": "test-index1", + "_id": "2", + "_score": 1, + "_source": { + "text": "Document with ID 2" + } + } + ] + } + ``` + +## Querying on the `_id` field + +While the `_id` field is accessible in various queries, it is restricted from use in aggregations, sorting, and scripting. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the content of the `_id` field into another field that has `doc_values` enabled. From d6cadd13c73584660c6b452960dcac59a2f213ea Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Apr 2024 17:32:39 -0600 Subject: [PATCH 012/166] Add field docs Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 1 + .../metadata-fields/index-metadata.md | 71 ++++++++++++++++++- 2 files changed, 70 insertions(+), 2 deletions(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 8ef19922ba..a242c87604 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -77,6 +77,7 @@ The following response shows that this query returns both documents with `_id` v ] } ``` + {% include copy-curl.html %} ## Querying on the `_id` field diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 58c75ce273..f23f2464f4 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -1,7 +1,74 @@ --- layout: default -title: Ignored +title: Index nav_order: 25 has_children: false parent: Metadata fields ---- \ No newline at end of file +--- + +# Index + +When querying across multiple indexes, you may need to filter results based on the index a document was indexed into. The `index` field matches documents based on their index. + +The following example creates two indexes, `products` and `customers` and adds a document to each index: + +```json +PUT products/_doc/1 +{ + "name": "Widget X" +} + +PUT customers/_doc/2?refresh=true +{ + "name": "John Doe" +} +``` +{% include copy-curl.html %} + +Now, you can query both indexes and filter the results using the `_index` field: + +```json +GET products,customers/_search +{ + "query": { + "terms": { + "_index": ["products", "customers"] + } + }, + "aggs": { + "index_groups": { + "terms": { + "field": "_index", + "size": 10 + } + } + }, + "sort": [ + { + "_index": { + "order": "desc" + } + } + ], + "script_fields": { + "index_name": { + "script": { + "lang": "painless", + "source": "doc['_index'].value" + } + } + } +} +``` +{% include copy-curl.html %} + +In this example: + +- The `query` section uses a `terms` query to match documents from the `products` and `customers` indexes. +- The `aggs` section performs a `terms` aggregation on the `_index` field, grouping the results by index. +- The `sort` section sorts the results by the `_index` field in ascending order. +- The `script_fields` section adds a new field `index_name` to the search results that contains the value of the `_index` field for each document. + +## Querying on the `_index` field + + From 4b6f1d11f80c0aa3700a15d6587238f042764578 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Apr 2024 13:45:29 -0600 Subject: [PATCH 013/166] Add new docs Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 5 +++-- _field-types/metadata-fields/routing.md | 2 +- _field-types/metadata-fields/source.md | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 527b5436f4..05f4dfeae7 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -1,7 +1,8 @@ --- layout: default -title: Ignored +title: Meta nav_order: 30 has_children: false parent: Metadata fields ---- \ No newline at end of file +--- + diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 161f5a4382..05ccff4894 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -1,6 +1,6 @@ --- layout: default -title: Ignored +title: Routing nav_order: 35 has_children: false parent: Metadata fields diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 824e3fa82f..4a9d9f9c74 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -1,6 +1,6 @@ --- layout: default -title: Ignored +title: Source nav_order: 40 has_children: false parent: Metadata fields From a9f6f4f9c7faeb456a600093b79799f28027b3fc Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 12:37:34 -0600 Subject: [PATCH 014/166] Add default and allowed values Signed-off-by: Melissa Vagi --- _field-types/index.md | 57 ++++++++++++------------------------------- 1 file changed, 15 insertions(+), 42 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 6740e8650b..b66a9661d9 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -29,12 +29,12 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. -boolean | OpenSearch accepts `true` and `false` as boolean values. An empty string is equal to `false.` +boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. integer | A signed 32-bit number. object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. -array | Arrays in OpenSearch can only store values of one type, such as an array of just integers or strings. Empty arrays are treated as though they are fields with no values. +array | Arrays in OpenSearch can only store values of one type, such as an array of only integers or strings. Empty arrays are treated as though they are fields with no values. text | A string sequence of characters that represent full-text values. keyword | A string sequence of structured characters, such as an email address or ZIP code. date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date. @@ -88,18 +88,18 @@ Mapping parameters are used to configure the behavior of fields in an index. The Parameter | Description :--- | :--- -`analyzer` | Specifies the analyzer used to analyze string fields. -`boost` | Specifies a field-level query time to boost. -`coerce` | Tries to convert the value to the specified data type. -`copy_to` | Copies the values of this field to another field. -`doc_values` | Specifies whether the field should be stored on disk to make sorting and aggregation faster. -`dynamic` | Determines whether new fields should be added dynamically. -`enabled` | Specifies whether the field is enabled or disabled. -`format` | Specifies the date format for date fields. -`ignore_above` | Skips indexing values that are longer than the specified length. -`ignore_malformed` | Specifies whether malformed values should be ignored. -`index` | Specifies whether the field should be indexed. -`index_options` | Specifies what information should be stored in the index for scoring purposes. +`analyzer` | Specifies the analyzer used to analyze string fields. Default is the `standard` analyzer, which is a general-purpose analyzer that splits text on white space and punctuation, converts to lowercase, and removes stop words. Allowed values are `standard`, `simple`, `whitespace`, and . +`boost` | Specifies a field-level query time to boost. Default boost value is `1.0`, which means no boost is applied. Allowed values are any floating-point number. +`coerce` | Tries to convert the value to the specified data type. Default value is `true`, which means OpenSearch tries to coerce the value to the expected value type. Allowed values are `true` or `false`. +`copy_to` | Copies the values of this field to another field. There is no default value for this parameter. It is an optional parameter that allows you to copy the value of a field to another field. +`doc_values` | Specifies whether the field should be stored on disk to make sorting and aggregation faster. Default value is `true`, which means the doc values are enabled. Allowed values are a single field name or a list of field names. Allowed values are `true` or `false`. +`dynamic` | Determines whether new fields should be added dynamically. Default value is `true`, which means new fields can be added dynamically. Allowed values are `true`, `false`, or `strict`. +`enabled` | Specifies whether the field is enabled or disabled. Default value is `true`, which means the field is enabled. Allowed values are `true` or `false`. +`format` | Specifies the date format for date fields. There is no default value for this parameter. It is used for date fields to specify the date format. Allowed values are any valid date format string, such as `yyyy-MM-dd` or `epoch_millis`. +`ignore_above` | Skips indexing values that are longer than the specified length. Default value is `2147483647`, which means there is no limit on the length of the field value. Allowed values are any positive integer. +`ignore_malformed` | Specifies whether malformed values should be ignored. Default value is `false`, which means malformed values are not ignored. . Allowed values are `true` or `false`. +`index` | Specifies whether the field should be indexed. Default value is `true`, which means the field is indexed. Allowed values are `true`, `false`, or `not_analyzed`. +`index_options` | Specifies what information should be stored in the index for scoring purposes. Default value `docs`, which means only the document numbers are stored in the index. is Allowed values are `docs`, `freqs`, `positions`, or `offsets`. ## Mapping limit settings @@ -114,11 +114,8 @@ OpenSearch has certain limits or settings related to mappings, such as the setti | index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | | index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | -## Runtime fields - -You can define fields at query time, rather than at index time, by using runtime fields. this can be useful for creating fields based on the values of other fields, or for performing transformations on data during the query process. Runtime fields are defined in the query itself and do not affect the underlying data in the index. - --- + ## Mapping example usage The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`. @@ -250,27 +247,3 @@ The response contains the specified fields: } } ``` - -## Delete a mapping - -The syntax for deleting a mapping depends on whether you want to delete the entire mapping for an index or the mapping for a specific field. The syntax for deleting a mapping is as follows: - -```json -DELETE //_mapping -DELETE / -``` - -For example, to delete the entire mapping for the `sample-index1` index, you can use the following commands: - -```json - -``` - -If you want to delete the mapping for a specific field, you can For example, to delete the mapping for the `year` field, use the following command: - -```json - -``` - -Deleting a field mapping will remove the mapping definition for that field across all indexes or the specified index. It will not delete the actual data stored in those fields. -{: .note} From 5b7ab8bb37ff3ba301cc9504195e5f49b5f7fac9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 12:51:44 -0600 Subject: [PATCH 015/166] Add default and allowed values Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index b66a9661d9..f0e5852bf7 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -99,7 +99,7 @@ Parameter | Description `ignore_above` | Skips indexing values that are longer than the specified length. Default value is `2147483647`, which means there is no limit on the length of the field value. Allowed values are any positive integer. `ignore_malformed` | Specifies whether malformed values should be ignored. Default value is `false`, which means malformed values are not ignored. . Allowed values are `true` or `false`. `index` | Specifies whether the field should be indexed. Default value is `true`, which means the field is indexed. Allowed values are `true`, `false`, or `not_analyzed`. -`index_options` | Specifies what information should be stored in the index for scoring purposes. Default value `docs`, which means only the document numbers are stored in the index. is Allowed values are `docs`, `freqs`, `positions`, or `offsets`. +`index_options` | Specifies what information should be stored in the index for scoring purposes. Default value `docs`, which means only the document numbers are stored in the index. Allowed values are `docs`, `freqs`, `positions`, or `offsets`. ## Mapping limit settings From 2b4db59ba69f6bc701ddf3a05b64677af75c75a1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 13:20:53 -0600 Subject: [PATCH 016/166] Update field-names.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 27 +++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index acded5205f..40621da738 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -14,5 +14,28 @@ The `field_names` only indexes field names when both `doc_values` and `norms` ar ## Mapping example - - +{ + "mappings": { + "properties": { + "field_names": { + "type": "keyword" + }, + "title": { + "type": "text", + "doc_values": false, + "norms": false + }, + "description": { + "type": "text", + "doc_values": true, + "norms": false + }, + "price": { + "type": "float", + "doc_values": false, + "norms": true + } + } + } +} +{% include copy-curl.html %} From edd60c6cc6d237fd3e5589de0629ed965e3accd7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 13:29:34 -0600 Subject: [PATCH 017/166] Update index-metadata.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index f23f2464f4..f7f15154b2 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -18,7 +18,7 @@ PUT products/_doc/1 "name": "Widget X" } -PUT customers/_doc/2?refresh=true +PUT customers/_doc/2 { "name": "John Doe" } From a5938362431d67b7acb0950f2337d1b9fa3683d2 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 13:58:55 -0600 Subject: [PATCH 018/166] Update index-metadata.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- .../metadata-fields/index-metadata.md | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index f7f15154b2..286a4dfa48 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -71,4 +71,26 @@ In this example: ## Querying on the `_index` field - +The `_index` field is a special field in OpenSearch that represents the index a document was indexed into. You can use this field in your queries to filter results based on the index. + +Some common use cases for querying on the `_index` field include: + +- Filtering search results to only include documents from specific indexes. +- Performing aggregations to get counts or statistics for each index. +- Sorting search results by the index they belong to. +- Retrieving the index name for each document in the search results. + +The `_index` field is automatically added to every document, so you can use it in your queries just like any other field. For example, you can use the `terms` query to match documents from multiple indexes: + +```json + { + "query": { + "terms": { + "_index": ["products", "customers"] + } + } +} +``` +{% include copy-curl.html %} + +This query returns all documents from the `products` and `customers` indexes. From 0e41a240a6202efff29a5ea32df38bfd5a8c6e55 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 14:32:18 -0600 Subject: [PATCH 019/166] Update meta.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 87 ++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 05f4dfeae7..bc9d0edf52 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -6,3 +6,90 @@ has_children: false parent: Metadata fields --- +# Meta + +The `_meta` field is a mapping property that allows you to associate custom metadata with your index mappings. This metadata is not used by OpenSearch itself, but can be used by your application to store information that is relevant to your use case. The `meta` field can be useful for a variety of scenarios, such as the following: + +- **Versioning**: Store the version of your application or index schema, making it easier to manage changes over time. +- **Ownership**: Identify the owner or author of an index, which can be helpful for maintenance and collaboration. +- **Categorization**: Classify indexes based on the application or domain they belong to, making it easier to organize and manage your data. +- **Auditing**: Track changes to the index metadata over time, providing a history of modifications. + +## Usage + +You can define the `_meta` field when creating a new index or updating an existing index's mapping: + +```json +PUT my-index +{ + "mappings": { + "_meta": { + "application": "MyApp", + "version": "1.2.3", + "author": "John Doe" + }, + "properties": { + "title": { + "type": "text" + }, + "description": { + "type": "text" + } + } + } +} + +``` +{% include copy-curl.html %} + +In this example, three custom metadata fields are added: application, version, and author. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the version of the application, or the author of the index. + +You can update the `_meta` field using the [PUT mapping API](/api-reference/index-apis/put-mapping/): + +```json +PUT my-index/_mapping +{ + "_meta": { + "application": "MyApp", + "version": "1.3.0", + "author": "Jane Smith" + } +} +``` +{% include copy-curl.html %} + +## Retrieving meta information + +You can retrieve the `_meta` information for an index using the GET mapping API: + +```json +GET my-index/_mapping +``` +{% include copy-curl.html %} + +This returns the full mapping for the index, including the `_meta` field, as shown in the following example: + +```json +{ + "my-index": { + "mappings": { + "_meta": { + "application": "MyApp", + "version": "1.3.0", + "author": "Jane Smith" + }, + "properties": { + "description": { + "type": "text" + }, + "title": { + "type": "text" + } + } + } + } +} +``` +{% include copy-curl.html %} + + From 6c0c5c83ccc316d30d62b16e7512e16b512f058e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 14:35:39 -0600 Subject: [PATCH 020/166] Update meta.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index bc9d0edf52..d9807c0753 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -44,7 +44,7 @@ PUT my-index In this example, three custom metadata fields are added: application, version, and author. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the version of the application, or the author of the index. -You can update the `_meta` field using the [PUT mapping API](/api-reference/index-apis/put-mapping/): +You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation: ```json PUT my-index/_mapping @@ -60,7 +60,7 @@ PUT my-index/_mapping ## Retrieving meta information -You can retrieve the `_meta` information for an index using the GET mapping API: +You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation: ```json GET my-index/_mapping @@ -91,5 +91,3 @@ This returns the full mapping for the index, including the `_meta` field, as sho } ``` {% include copy-curl.html %} - - From 5e11c4aff5053c17150cf7b4d0de89f4cbfa24c3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 14:38:18 -0600 Subject: [PATCH 021/166] Update index-metadata.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 286a4dfa48..574eb0eb41 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -10,7 +10,7 @@ parent: Metadata fields When querying across multiple indexes, you may need to filter results based on the index a document was indexed into. The `index` field matches documents based on their index. -The following example creates two indexes, `products` and `customers` and adds a document to each index: +The following example creates two indexes, `products` and `customers`, and adds a document to each index: ```json PUT products/_doc/1 @@ -25,7 +25,7 @@ PUT customers/_doc/2 ``` {% include copy-curl.html %} -Now, you can query both indexes and filter the results using the `_index` field: +Now, you can query both indexes and filter the results using the `_index` field, as shown in the following example: ```json GET products,customers/_search @@ -80,7 +80,7 @@ Some common use cases for querying on the `_index` field include: - Sorting search results by the index they belong to. - Retrieving the index name for each document in the search results. -The `_index` field is automatically added to every document, so you can use it in your queries just like any other field. For example, you can use the `terms` query to match documents from multiple indexes: +The `_index` field is automatically added to every document, so you can use it in your queries just like any other field. For instance, you can use the `terms` query to match documents from multiple indexes, as shown in the following example: ```json { From 3dcfd50b58b609cb0bf094c5af23cc1c99278508 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 14:39:30 -0600 Subject: [PATCH 022/166] Update meta.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index d9807c0753..ea01b2ba23 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -17,7 +17,7 @@ The `_meta` field is a mapping property that allows you to associate custom meta ## Usage -You can define the `_meta` field when creating a new index or updating an existing index's mapping: +You can define the `_meta` field when creating a new index or updating an existing index's mapping, as shown in the following example: ```json PUT my-index @@ -44,7 +44,7 @@ PUT my-index In this example, three custom metadata fields are added: application, version, and author. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the version of the application, or the author of the index. -You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation: +You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, as shown in the following example: ```json PUT my-index/_mapping @@ -60,7 +60,7 @@ PUT my-index/_mapping ## Retrieving meta information -You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation: +You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, as shown in the following example: ```json GET my-index/_mapping From 9fcc2eb5f760f5b5dadfcdd7390a21e467804b17 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 14:43:13 -0600 Subject: [PATCH 023/166] Update index-metadata.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 574eb0eb41..b41f2aae85 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -80,7 +80,7 @@ Some common use cases for querying on the `_index` field include: - Sorting search results by the index they belong to. - Retrieving the index name for each document in the search results. -The `_index` field is automatically added to every document, so you can use it in your queries just like any other field. For instance, you can use the `terms` query to match documents from multiple indexes, as shown in the following example: +The `_index` field is automatically added to every document, so you can use it in your queries like any other field. For instance, you can use the `terms` query to match documents from multiple indexes, as shown in the following example: ```json { From 3d30dfb4eb89eb00c708d5ef9bfd305ab8b82941 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 15:37:33 -0600 Subject: [PATCH 024/166] Update routing.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 91 ++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 05ccff4894..9713b95dc0 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -4,4 +4,93 @@ title: Routing nav_order: 35 has_children: false parent: Metadata fields ---- \ No newline at end of file +--- + +# Routing + +Documents are routed to specific shards within an index using a hashing algorithm. By default, the document's `_id` field is used as the routing value, but you can also specify a custom routing value for each document. + +## Default routing + +The following formula is the default routing formula used in OpenSearch. The `_routing` value is the document's `_id`. + +```json +shard_num = hash(_routing) % num_primary_shards +``` + +## Custom routing + +You can specify a custom routing value when indexing a document, as shown in the following example: + +```json +PUT my-index-001/_doc/1?routing=JohnDoe1 +{ + "title": "This is a document" +} +``` +{% include copy-curl.html %} + +In this example, the document is routed using the value `JohnDoe1` instead of the default `_id`. + +You must provide the same routing value when retrieving, deleting, or updating the document, as shown in the following example: + +```json +GET my-index-001/_doc/1?routing=JohnDoe1 +``` +{% include copy-curl.html %} + +## Querying by routing + +You can query documents based in their routing value using the `_routing` field, as shown in the following example. This query only searches the shard(s) associated with the `JohnDoe1` routing value. + +```json +GET my-index-001/_search +{ + "query": { + "terms": { + "_routing": [ "JohnDoe1" ] + } + } +} +``` +{% include copy-curl.html %} + +## Required routing + +You can make custom routing a required field for all CRUD operations on an index, as shown in the following example. If you try to index a document without providing a routing value, OpenSearch throws an exception. +. + +```json +PUT my-index-002 +{ + "mappings": { + "_routing": { + "required": true + } + } +} +``` +{% include copy-curl.html %} + +## Routing to index partitions + +You can configure an index to route documents to a subset of shards, rather than a single shard. This is done using the `index.routing_partition_size` setting, as shown in the following example: + +```json +PUT my-index-003 +{ + "settings": { + "index.routing_partition_size": 4 + }, + "mappings": { + "_routing": { + "required": true + } + } +} +``` +{% include copy-curl.html %} + +In this example, the documents are routed to one of four partitions within the index, based on a combination of the `_routing` and `_id` fields. + +Partitioned indexes have some limitations, such as not supporting `join` field relationships in the mappings. From 56991f377f9df55abd6098c5727134ea1e55024e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 15:55:23 -0600 Subject: [PATCH 025/166] Update routing.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 9713b95dc0..efec265eb9 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -23,7 +23,7 @@ shard_num = hash(_routing) % num_primary_shards You can specify a custom routing value when indexing a document, as shown in the following example: ```json -PUT my-index-001/_doc/1?routing=JohnDoe1 +PUT sample-index1/_doc/1?routing=JohnDoe1 { "title": "This is a document" } @@ -35,7 +35,7 @@ In this example, the document is routed using the value `JohnDoe1` instead of th You must provide the same routing value when retrieving, deleting, or updating the document, as shown in the following example: ```json -GET my-index-001/_doc/1?routing=JohnDoe1 +GET sample-index1/_doc/1?routing=JohnDoe1 ``` {% include copy-curl.html %} @@ -44,7 +44,7 @@ GET my-index-001/_doc/1?routing=JohnDoe1 You can query documents based in their routing value using the `_routing` field, as shown in the following example. This query only searches the shard(s) associated with the `JohnDoe1` routing value. ```json -GET my-index-001/_search +GET sample-index1/_search { "query": { "terms": { @@ -61,7 +61,7 @@ You can make custom routing a required field for all CRUD operations on an index . ```json -PUT my-index-002 +PUT sample-index2 { "mappings": { "_routing": { @@ -77,7 +77,7 @@ PUT my-index-002 You can configure an index to route documents to a subset of shards, rather than a single shard. This is done using the `index.routing_partition_size` setting, as shown in the following example: ```json -PUT my-index-003 +PUT sample-index3 { "settings": { "index.routing_partition_size": 4 From a7f39c0ce2488e895c24df1953221b07e133decf Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 29 Apr 2024 16:35:23 -0600 Subject: [PATCH 026/166] Update source.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 47 ++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 4a9d9f9c74..775ff6e66e 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -6,3 +6,50 @@ has_children: false parent: Metadata fields --- +# Source + +The `_source` field contains the original JSON document body that was passed at index time. This field is not indexed and is not searchable but is stored so that it can be returned when executing fethc requests such as `get` and `search`. + +## Disabling the field + +You can disable the `_source` field as shown in the following example: + +```json +PUT sample-index1 +{ + "mappings": { + "_source": { + "enabled": false + } + } +} +``` +{% include copy-curl.html %} + +Disabling the `_source` field can cause certain featues to become not supported, such as `update`, `update_by_query`, and `reindex` APIs and the ability to debug queries or aggregations by using the the original document at index time. +{: .warning} + +## Including or excluding fields + +You can prune contents of the `_source` field after the document has been indexed but before the field is stored. You can use the `includes` and excludes` parameters to control which field are included in the stored `_source` field, as shown in the following example: + +```json +PUT logs +{ + "mappings": { + "_source": { + "includes": [ + "*.count", + "meta.*" + ], + "excludes": [ + "meta.description", + "meta.other.*" + ] + } + } +} +``` +{% include copy-curl.html %} + +Although these fields are not stored in the `_source`, you can still search on them, as the data is still indexed. From 078c49ff1a49f51de0f383f216ce9920bc1f7bf2 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 8 May 2024 11:39:17 -0600 Subject: [PATCH 027/166] Update index.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/index.md | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index f0e5852bf7..6008e59de9 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -84,22 +84,7 @@ You cannot change the mapping of an existing field, you can only modify the fiel ## Mapping parameters -Mapping parameters are used to configure the behavior of fields in an index. The following table lists commonly used mapping parameters. - -Parameter | Description -:--- | :--- -`analyzer` | Specifies the analyzer used to analyze string fields. Default is the `standard` analyzer, which is a general-purpose analyzer that splits text on white space and punctuation, converts to lowercase, and removes stop words. Allowed values are `standard`, `simple`, `whitespace`, and . -`boost` | Specifies a field-level query time to boost. Default boost value is `1.0`, which means no boost is applied. Allowed values are any floating-point number. -`coerce` | Tries to convert the value to the specified data type. Default value is `true`, which means OpenSearch tries to coerce the value to the expected value type. Allowed values are `true` or `false`. -`copy_to` | Copies the values of this field to another field. There is no default value for this parameter. It is an optional parameter that allows you to copy the value of a field to another field. -`doc_values` | Specifies whether the field should be stored on disk to make sorting and aggregation faster. Default value is `true`, which means the doc values are enabled. Allowed values are a single field name or a list of field names. Allowed values are `true` or `false`. -`dynamic` | Determines whether new fields should be added dynamically. Default value is `true`, which means new fields can be added dynamically. Allowed values are `true`, `false`, or `strict`. -`enabled` | Specifies whether the field is enabled or disabled. Default value is `true`, which means the field is enabled. Allowed values are `true` or `false`. -`format` | Specifies the date format for date fields. There is no default value for this parameter. It is used for date fields to specify the date format. Allowed values are any valid date format string, such as `yyyy-MM-dd` or `epoch_millis`. -`ignore_above` | Skips indexing values that are longer than the specified length. Default value is `2147483647`, which means there is no limit on the length of the field value. Allowed values are any positive integer. -`ignore_malformed` | Specifies whether malformed values should be ignored. Default value is `false`, which means malformed values are not ignored. . Allowed values are `true` or `false`. -`index` | Specifies whether the field should be indexed. Default value is `true`, which means the field is indexed. Allowed values are `true`, `false`, or `not_analyzed`. -`index_options` | Specifies what information should be stored in the index for scoring purposes. Default value `docs`, which means only the document numbers are stored in the index. Allowed values are `docs`, `freqs`, `positions`, or `offsets`. +Mapping parameters are used to configure the behavior of fields in an index. See the [Mapping parameters](inert-link-to-page) page for more information. ## Mapping limit settings From 415ef50d0214b25cd3deed9164575f54761eb34c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 15 May 2024 14:49:30 -0600 Subject: [PATCH 028/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 6008e59de9..0b3f132355 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -29,7 +29,7 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. -boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` +Boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. integer | A signed 32-bit number. From 11e1a1ce8bfc63fe8319bca6a6f780a7c083401b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 15 May 2024 14:54:45 -0600 Subject: [PATCH 029/166] Update _field-types/metadata-fields/id.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index a242c87604..88ac404e2c 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -81,4 +81,4 @@ The following response shows that this query returns both documents with `_id` v ## Querying on the `_id` field -While the `_id` field is accessible in various queries, it is restricted from use in aggregations, sorting, and scripting. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the content of the `_id` field into another field that has `doc_values` enabled. +While the `_id` field is accessible in various queries, it is restricted from use in aggregations, sorting, and scripting. See [IDs query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/) for an example of the field's usage. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the content of the `_id` field into another field that has `doc_values` enabled. From ffd96dce72da5517979cf999fc5f109a99596304 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 3 Jun 2024 09:40:01 -0600 Subject: [PATCH 030/166] Add dynamic templates section and examples Signed-off-by: Melissa Vagi --- _field-types/index.md | 69 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/_field-types/index.md b/_field-types/index.md index 0b3f132355..28dbdb0d62 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -40,6 +40,10 @@ keyword | A string sequence of structured characters, such as an email address o date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date. numeric detection string | If disabled, OpenSearch may automatically process numeric values as strings when they should be processed as numbers. When enabled, OpenSearch can process strings into `long`, `integer`, `short`, `byte`, `double`, `float`, `half_float`, `scaled_float`, and `unsigned_long`. Default is disabled. +### Dynamic templates + +Dynamic templates are used to define custom mappings for dynamically added fields based on data type, field name, or field path. They allow you to define a flexible schema for your data, which can automatically adapt to changes in the structure or format of the input data. + ## Explicit mapping If you know exactly what your field data types need to be, you can specify them in your request body when creating your index. @@ -56,6 +60,7 @@ PUT sample-index1 } } ``` +{% include copy-curl.html %} #### Response ```json @@ -65,6 +70,7 @@ PUT sample-index1 "index": "sample-index1" } ``` +{% include copy-curl.html %} To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method: @@ -78,6 +84,7 @@ POST sample-index1/_mapping } } ``` +{% include copy-curl.html %} You cannot change the mapping of an existing field, you can only modify the field's mapping parameters. {: .note} @@ -122,6 +129,7 @@ PUT /test-index } } ``` +{% include copy-curl.html %} You can add a document that has a malformed IP address to your index: @@ -131,6 +139,7 @@ PUT /test-index/_doc/1 "ip_address" : "malformed ip address" } ``` +{% include copy-curl.html %} This indexed IP address does not throw an error because `ignore_malformed` is set to true. @@ -139,6 +148,7 @@ You can query the index using the following request: ```json GET /test-index/_search ``` +{% include copy-curl.html %} The response shows that the `ip_address` field is ignored in the indexed document: @@ -174,6 +184,7 @@ The response shows that the `ip_address` field is ignored in the indexed documen } } ``` +{% include copy-curl.html %} ## Get a mapping @@ -182,6 +193,7 @@ To get all mappings for one or more indexes, use the following request: ```json GET /_mapping ``` +{% include copy-curl.html %} In the previous request, `` may be an index name or a comma-separated list of index names. @@ -190,6 +202,7 @@ To get all mappings for all indexes, use the following request: ```json GET _mapping ``` +{% include copy-curl.html %} To get a mapping for a specific field, provide the index name and the field name: @@ -197,6 +210,7 @@ To get a mapping for a specific field, provide the index name and the field name GET _mapping/field/ GET //_mapping/field/ ``` +{% include copy-curl.html %} Both `` and `` can be specified as one value or a comma-separated list. @@ -205,6 +219,7 @@ For example, the following request retrieves the mapping for the `year` and `age ```json GET sample-index1/_mapping/field/year,age ``` +{% include copy-curl.html %} The response contains the specified fields: @@ -232,3 +247,57 @@ The response contains the specified fields: } } ``` +{% include copy-curl.html %} + +## Map null fields to a custom value + +The following example maps all null fields to a custom `N/A` value: + +```json +{ + "dynamic_templates": [ + { + "null_fields": { + "match_mapping_type": "null", + "mapping": { + "type": "text", + "null_value": "N/A" + } + } + } + ] +} +``` +{% include copy-curl.html %} + +## Map numeric fields + +The following example ensures that all float, double, and integer fields are mapped to the appropriate numeric types, rather than being treated as strings: + +```json +{ + "dynamic_templates": [ + { + "floats": { + "match_mapping_type": "float", + "mapping": { + "type": "float" + } + }, + "doubles": { + "match_mapping_type": "double", + "mapping": { + "type": "double" + } + }, + "integers": { + "match_mapping_type": "integer", + "mapping": { + "type": "integer" + } + } + } + ] +} +``` +{% include copy-curl.html %} \ No newline at end of file From 5c64fd6b0f733c0574dd6dcc1a10a7fbd8b77530 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Mon, 3 Jun 2024 10:25:45 -0600 Subject: [PATCH 031/166] Add dynamic templates code snippet Signed-off-by: Melissa Vagi --- _field-types/index.md | 83 +++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 43 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 28dbdb0d62..8be0facc18 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -44,6 +44,27 @@ numeric detection string | If disabled, OpenSearch may automatically process num Dynamic templates are used to define custom mappings for dynamically added fields based on data type, field name, or field path. They allow you to define a flexible schema for your data, which can automatically adapt to changes in the structure or format of the input data. +The syntax for defining a dynamic mapping template in OpenSearch looks like the following: + +```json + "dynamic_templates": [ + { + "template_name": { + ... match conditions ... + "mapping": { ... } + } + }, + ... + ] +``` +{% include copy-curl.html %} + +Note the following: + +- The template name can be any string value. +- The match conditions can include any of the following: `match_mapping_type`, `match`, `match_pattern`, `unmatch`, `path_match`, or `path_unmatch`. +- Specify the `mapping` to be used for the matched field. + ## Explicit mapping If you know exactly what your field data types need to be, you can specify them in your request body when creating your index. @@ -249,55 +270,31 @@ The response contains the specified fields: ``` {% include copy-curl.html %} -## Map null fields to a custom value +## Map string fields to `text` and `keyword` types -The following example maps all null fields to a custom `N/A` value: +This request creates an index named `movies1` with a dynamic template that maps all string fields to both `text` and `keyword` types. ```json +PUT movies1 { - "dynamic_templates": [ - { - "null_fields": { - "match_mapping_type": "null", - "mapping": { - "type": "text", - "null_value": "N/A" + "mappings": { + "dynamic_templates": [ + { + "strings": { + "match_mapping_type": "string", + "mapping": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } } } - } - ] + ] + } } ``` {% include copy-curl.html %} - -## Map numeric fields - -The following example ensures that all float, double, and integer fields are mapped to the appropriate numeric types, rather than being treated as strings: - -```json -{ - "dynamic_templates": [ - { - "floats": { - "match_mapping_type": "float", - "mapping": { - "type": "float" - } - }, - "doubles": { - "match_mapping_type": "double", - "mapping": { - "type": "double" - } - }, - "integers": { - "match_mapping_type": "integer", - "mapping": { - "type": "integer" - } - } - } - ] -} -``` -{% include copy-curl.html %} \ No newline at end of file From 8e7a288d6c1d5b3c4dd19d7b76439dabb74c20f3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:04:17 -0600 Subject: [PATCH 032/166] Update _field-types/metadata-fields/source.md Co-authored-by: Ralph Ursprung <39383228+rursprung@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 775ff6e66e..3ecff61b6d 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -31,7 +31,7 @@ Disabling the `_source` field can cause certain featues to become not supported, ## Including or excluding fields -You can prune contents of the `_source` field after the document has been indexed but before the field is stored. You can use the `includes` and excludes` parameters to control which field are included in the stored `_source` field, as shown in the following example: +You can prune contents of the `_source` field after the document has been indexed but before the field is stored. You can use the `includes` and `excludes` parameters to control which field are included in the stored `_source` field, as shown in the following example: ```json PUT logs From 459f3d0abedb4cb487f788fed9965d75fba0a5ab Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:13:13 -0600 Subject: [PATCH 033/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 8be0facc18..ab91efe1cb 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -28,7 +28,7 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- -null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. +`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. Boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. From 0c451a5f13f6ec31dd4ff4d42a32bea0a0893dd5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:13:26 -0600 Subject: [PATCH 034/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index ab91efe1cb..a35197695d 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -29,7 +29,7 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- `null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. -Boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` +`boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. integer | A signed 32-bit number. From 8eefb2ba95ee15e78488deeccaa969df2778443c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:14:12 -0600 Subject: [PATCH 035/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index a35197695d..97f557e55b 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -28,7 +28,7 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- -`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. +null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. `boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. From 709d4b31d4969fd52ad981f73ef5ea2a7e3c77b7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:14:33 -0600 Subject: [PATCH 036/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 97f557e55b..510215e50a 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -29,7 +29,7 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. -`boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` +boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. integer | A signed 32-bit number. From ced88265a840fab3720376e7c54af22e083e452e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:22:24 -0600 Subject: [PATCH 037/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 510215e50a..33b2a0e45f 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -112,7 +112,7 @@ You cannot change the mapping of an existing field, you can only modify the fiel ## Mapping parameters -Mapping parameters are used to configure the behavior of fields in an index. See the [Mapping parameters](inert-link-to-page) page for more information. +Mapping parameters are used to configure the behavior of fields in an index. See the [Mappings and field types]({{site.url}}{{site.baseurl}}//field-types/) page for more information. ## Mapping limit settings From 23943c88727f389b944ad9ca62c160e8a90f829e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:37:04 -0600 Subject: [PATCH 038/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 33b2a0e45f..c73091caf4 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -112,7 +112,7 @@ You cannot change the mapping of an existing field, you can only modify the fiel ## Mapping parameters -Mapping parameters are used to configure the behavior of fields in an index. See the [Mappings and field types]({{site.url}}{{site.baseurl}}//field-types/) page for more information. +Mapping parameters are used to configure the behavior of fields in an index. See the [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) page for more information. ## Mapping limit settings From 91d0f1858e3a1c9dccd7b9a011557fa37bb4fda2 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:39:22 -0600 Subject: [PATCH 039/166] Update _field-types/metadata-fields/field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 40621da738..6445ad27ce 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -1,6 +1,6 @@ --- layout: default -title: Field names +title: _field_names nav_order: 10 has_children: false parent: Metadata fields From 8485d22844a1ec823233bbc32017343b828d9a04 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 16:53:12 -0600 Subject: [PATCH 040/166] Update field-names.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 6445ad27ce..6b87d84f7f 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -15,10 +15,11 @@ The `field_names` only indexes field names when both `doc_values` and `norms` ar ## Mapping example { - "mappings": { + "mappings": { + "_field_names": { + "enabled": "true" + }, "properties": { - "field_names": { - "type": "keyword" }, "title": { "type": "text", From a5ccbd9c8b6116452b36214856edd2a4d19ea238 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 26 Jun 2024 17:43:08 -0600 Subject: [PATCH 041/166] Update _field-types/metadata-fields/id.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 88ac404e2c..2663638ee5 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -78,6 +78,7 @@ The following response shows that this query returns both documents with `_id` v } ``` {% include copy-curl.html %} + ## Querying on the `_id` field From f4780406e001e15ecf57efd9009598160344c261 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 11 Jul 2024 09:18:29 -0600 Subject: [PATCH 042/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index c73091caf4..e8001ea38f 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -14,7 +14,11 @@ redirect_from: You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/). -If you're starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings. +If you're starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, it is important to note that dynamic mappings can have performance implications, especially for large datasets or high-volume indexing operations. + +While dynamic mappings provide convenience and flexibility, it is generally recommended to use explicit mappings whenever possible. With explicit mappings, you define the exact structure and data types of your fields upfront, enforcing a consistent standard across your data. This approach not only ensures data integrity but also optimizes performance by allowing OpenSearch to efficiently store and query your data. + +If you know exactly what types your data falls under and want to enforce that standard, then you should use explicit mappings. Even if you are unsure about your data structure initially, it is advisable to switch to explicit mappings once you have a better understanding of your data to avoid potential performance issues and maintain data consistency. For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers. From f3337a63815f22142446136d4a36c7aeabe8da95 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 11 Jul 2024 09:18:46 -0600 Subject: [PATCH 043/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index e8001ea38f..13825f6759 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -38,7 +38,7 @@ float | A single-precision 32-bit floating point number. double | A double-precision 64-bit floating point number. integer | A signed 32-bit number. object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. -array | Arrays in OpenSearch can only store values of one type, such as an array of only integers or strings. Empty arrays are treated as though they are fields with no values. +array | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements, not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. text | A string sequence of characters that represent full-text values. keyword | A string sequence of structured characters, such as an email address or ZIP code. date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date. From e21c9af68b86b75b950d94c72f6928d805dac681 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 16 Jul 2024 13:31:40 -0600 Subject: [PATCH 044/166] Writing in progress Signed-off-by: Melissa Vagi --- _field-types/index.md | 187 ++++++++--------------------- _field-types/mappings-use-cases.md | 123 +++++++++++++++++++ 2 files changed, 170 insertions(+), 140 deletions(-) create mode 100644 _field-types/mappings-use-cases.md diff --git a/_field-types/index.md b/_field-types/index.md index 13825f6759..007e158cc5 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -12,17 +12,11 @@ redirect_from: # Mappings and field types -You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/). +Mappings define how documents and their fields are stored and indexed in OpenSearch. You can specify field types (for example, `year` as `date`) to ensure efficient storage and querying. While dynamic mappings automatically add data and fields, explicit mappings are recommended for enforcing consistent data structures and optimizing performance, especially for large datasets or high-volume indexing operations. -If you're starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, it is important to note that dynamic mappings can have performance implications, especially for large datasets or high-volume indexing operations. +Explicit mappings allow you to define the exact structure and data types of your fields upfront, ensuring data integrity and efficient storage and querying. Even if you are unsure about your data structure initially, it is advisable to switch to explicit mappings once you have a better understanding of your data to avoid potential performance issues and maintain consistency. -While dynamic mappings provide convenience and flexibility, it is generally recommended to use explicit mappings whenever possible. With explicit mappings, you define the exact structure and data types of your fields upfront, enforcing a consistent standard across your data. This approach not only ensures data integrity but also optimizes performance by allowing OpenSearch to efficiently store and query your data. - -If you know exactly what types your data falls under and want to enforce that standard, then you should use explicit mappings. Even if you are unsure about your data structure initially, it is advisable to switch to explicit mappings once you have a better understanding of your data to avoid potential performance issues and maintain data consistency. - -For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers. - -This documentation provides an example for how to create an index mapping and how to add a document to it that will get `ip_range` validated. +For example, with explicit mappings, you can ensure that year is treated as text and age as an integer, preventing OpenSearch from interpreting both as integers through dynamic mapping. ## Dynamic mapping @@ -32,15 +26,15 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- -null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. -boolean | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` -float | A single-precision 32-bit floating point number. -double | A double-precision 64-bit floating point number. -integer | A signed 32-bit number. -object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. -array | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements, not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. -text | A string sequence of characters that represent full-text values. -keyword | A string sequence of structured characters, such as an email address or ZIP code. +`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. +`boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` +`float` | A single-precision 32-bit floating point number. +`double` | A double-precision 64-bit floating point number. +`integer` | A signed 32-bit number. +`object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. +`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements, not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. +`text` | A string sequence of characters that represent full-text values. +`keyword` | A string sequence of structured characters, such as an email address or ZIP code. date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date. numeric detection string | If disabled, OpenSearch may automatically process numeric values as strings when they should be processed as numbers. When enabled, OpenSearch can process strings into `long`, `integer`, `short`, `byte`, `double`, `float`, `half_float`, `scaled_float`, and `unsigned_long`. Default is disabled. @@ -48,26 +42,43 @@ numeric detection string | If disabled, OpenSearch may automatically process num Dynamic templates are used to define custom mappings for dynamically added fields based on data type, field name, or field path. They allow you to define a flexible schema for your data, which can automatically adapt to changes in the structure or format of the input data. -The syntax for defining a dynamic mapping template in OpenSearch looks like the following: +You can use the following syntax to define a dynamic mapping template: ```json - "dynamic_templates": [ - { - "template_name": { - ... match conditions ... - "mapping": { ... } - } - }, - ... - ] +PUT index +{ + "mappings": { + "dynamic_templates": [ + { + "fields": { + "mapping": { + "type": "short" + }, + "match_mapping_type": "string", + "path_match": "status*" + } + } + ] + } +} ``` {% include copy-curl.html %} -Note the following: +This mapping configuration dynamically maps any field with a name starting with `status` (for example, `status_code`) to the `short` data type if the initial value provided during indexing is a string. + +### Dynamic mapping parameters -- The template name can be any string value. -- The match conditions can include any of the following: `match_mapping_type`, `match`, `match_pattern`, `unmatch`, `path_match`, or `path_unmatch`. -- Specify the `mapping` to be used for the matched field. +The `dynamic_templates` support the following parameters for matching conditions and mapping rules: + +Parameter | Description | +----------|-------------| +`match_mapping_type` | Specifies the JSON data type (for example, string, long, double, object, binary, Boolean, date) that triggers the mapping. +`match` | A regular expression to match field names and apply the mapping. +`unmatch` | A regular expression to exclude field names from the mapping. +`match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`. +`path_match` | Allows matching nested field paths using a regular expression. +`path_unmatch` | Excludes nested field paths from the mapping using a regular expression. +`mapping` | The mapping configuration to apply for matching fields. ## Explicit mapping @@ -131,85 +142,8 @@ OpenSearch has certain limits or settings related to mappings, such as the setti | index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | | index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | ---- - -## Mapping example usage - -The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`. - -### Create an index with an `ip` mapping - -To create an index, use a PUT request: - -```json -PUT /test-index -{ - "mappings" : { - "properties" : { - "ip_address" : { - "type" : "ip", - "ignore_malformed": true - } - } - } -} -``` -{% include copy-curl.html %} - -You can add a document that has a malformed IP address to your index: - -```json -PUT /test-index/_doc/1 -{ - "ip_address" : "malformed ip address" -} -``` -{% include copy-curl.html %} - -This indexed IP address does not throw an error because `ignore_malformed` is set to true. - -You can query the index using the following request: - -```json -GET /test-index/_search -``` -{% include copy-curl.html %} - -The response shows that the `ip_address` field is ignored in the indexed document: -```json -{ - "took": 14, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 1, - "hits": [ - { - "_index": "test-index", - "_id": "1", - "_score": 1, - "_ignored": [ - "ip_address" - ], - "_source": { - "ip_address": "malformed ip address" - } - } - ] - } -} -``` -{% include copy-curl.html %} +--- ## Get a mapping @@ -237,9 +171,7 @@ GET //_mapping/field/ ``` {% include copy-curl.html %} -Both `` and `` can be specified as one value or a comma-separated list. - -For example, the following request retrieves the mapping for the `year` and `age` fields in `sample-index1`: +Both `` and `` can be specified as one value or a comma-separated list. For example, the following request retrieves the mapping for the `year` and `age` fields in `sample-index1`: ```json GET sample-index1/_mapping/field/year,age @@ -274,31 +206,6 @@ The response contains the specified fields: ``` {% include copy-curl.html %} -## Map string fields to `text` and `keyword` types - -This request creates an index named `movies1` with a dynamic template that maps all string fields to both `text` and `keyword` types. +## Mappings use cases -```json -PUT movies1 -{ - "mappings": { - "dynamic_templates": [ - { - "strings": { - "match_mapping_type": "string", - "mapping": { - "type": "text", - "fields": { - "keyword": { - "type": "keyword", - "ignore_above": 256 - } - } - } - } - } - ] - } -} -``` -{% include copy-curl.html %} +See use case examples, including mapping string fields and ignoring malformed IP addresses, in [Mappings use cases](). diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md new file mode 100644 index 0000000000..650905f534 --- /dev/null +++ b/_field-types/mappings-use-cases.md @@ -0,0 +1,123 @@ +--- +layout: default +title: Mappings use cases +parent: Mappings and fields types +nav_order: 5 +nav_exclude: true +--- + +# Mappings use cases + +Mappings in OpenSearch provide control over how data is indexed and queried, enabling optimized performance and efficient storage for a range of use cases. + +--- + +## Example: Ignoring malformed IP addresses + +The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`. + +### Create an index with an `ip` mapping + +To create an index, use a PUT request: + +```json +PUT /test-index +{ + "mappings" : { + "properties" : { + "ip_address" : { + "type" : "ip", + "ignore_malformed": true + } + } + } +} +``` +{% include copy-curl.html %} + +You can add a document that has a malformed IP address to your index: + +```json +PUT /test-index/_doc/1 +{ + "ip_address" : "malformed ip address" +} +``` +{% include copy-curl.html %} + +This indexed IP address does not throw an error because `ignore_malformed` is set to true. + +You can query the index using the following request: + +```json +GET /test-index/_search +``` +{% include copy-curl.html %} + +The response shows that the `ip_address` field is ignored in the indexed document: + +```json +{ + "took": 14, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "test-index", + "_id": "1", + "_score": 1, + "_ignored": [ + "ip_address" + ], + "_source": { + "ip_address": "malformed ip address" + } + } + ] + } +} +``` +{% include copy-curl.html %} + + +--- + +## Mapping string fields to `text` and `keyword` types + +This request creates an index named `movies1` with a dynamic template that maps all string fields to both `text` and `keyword` types. + +```json +PUT movies1 +{ + "mappings": { + "dynamic_templates": [ + { + "strings": { + "match_mapping_type": "string", + "mapping": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } + } + } + ] + } +} +``` +{% include copy-curl.html %} From f2516c91ab17f166008979520d067132b1714a19 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Jul 2024 12:15:56 -0600 Subject: [PATCH 045/166] Update _field-types/metadata-fields/id.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 2663638ee5..940c02b6b5 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -79,7 +79,8 @@ The following response shows that this query returns both documents with `_id` v ``` {% include copy-curl.html %} - +A unique `_id` is automatically generated and assigned if you do not provide one. +{: .note} ## Querying on the `_id` field While the `_id` field is accessible in various queries, it is restricted from use in aggregations, sorting, and scripting. See [IDs query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/) for an example of the field's usage. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the content of the `_id` field into another field that has `doc_values` enabled. From 365b14521e68c23da540b7ee927f41238e1741aa Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Jul 2024 12:19:16 -0600 Subject: [PATCH 046/166] Update _field-types/metadata-fields/field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 6b87d84f7f..040cb2b4b5 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -8,7 +8,7 @@ parent: Metadata fields # Field names -The `field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. +The `_field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. The `field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `field_names`. From 2292abc3de6f1af7db41657ddcc7368e0f3231ed Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Jul 2024 12:19:45 -0600 Subject: [PATCH 047/166] Update _field-types/metadata-fields/field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 040cb2b4b5..42a30f0a9a 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -10,7 +10,7 @@ parent: Metadata fields The `_field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. -The `field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `field_names`. +`_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `field_names`. ## Mapping example From a114b6e0ab3c43f8f6fbcf54582f1b08a98aebc9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Jul 2024 12:20:17 -0600 Subject: [PATCH 048/166] Update _field-types/metadata-fields/field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 42a30f0a9a..c3092c2e51 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -10,7 +10,7 @@ parent: Metadata fields The `_field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. -`_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `field_names`. +`_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `_field_names`. ## Mapping example From 47a1bf9daa66d505923c9d5c1878dc463b3a41b6 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Jul 2024 12:20:56 -0600 Subject: [PATCH 049/166] Update _field-types/metadata-fields/index.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index ac7a6b6e32..7823e233d9 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -12,7 +12,7 @@ OpenSearch has built-in metadata fields that provide information about the docum Metadata fields | Description :--- | :--- -`field_names` | The fields within the document that hold non-empty or non-null values. +`_field_names` | The fields within the document that hold non-empty or non-null values. `_ignored` | The fields in the document that were disregarded during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. `_id` | The unique identifier assigned to each individual document. `_index` | The specific index within the OpenSearch database where the document is stored and organized. From 00fc9c1bc239152dbcd6e0281b40d91e0f6047d5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 17 Jul 2024 12:38:42 -0600 Subject: [PATCH 050/166] Update ignored.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 85 ++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 3 deletions(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 8ee73751d1..04b0fe42f3 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -17,7 +17,7 @@ You can query the `_ignored` field using `term`, `terms`, and `exists` queries, The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated. {: .note} -For example, the following query will retrieve all documents that have at least one field that was ignored during indexing: +The basic syntax for a request with `_ignored` is as follows: ```json GET _search @@ -31,7 +31,86 @@ GET _search ``` {% include copy-curl.html %} -Similarly, you can use a term query to find documents where a specific field, such as created_at, was ignored: +--- + +#### Example indexing request with `_ignored` field + +The following example request adds a new document to the `test-ignored` index, with `ignore_malformed` set to `true` so that no error is thrown during indexing: + +```json +PUT test-ignored +{ + "mappings": { + "properties": { + "title": { + "type": "text" + }, + "length": { + "type": "long", + "ignore_malformed": true + } + } + } +} + +POST test-ignored/_doc +{ + "title": "correct text", + "length": "not a number" +} + +GET test-ignored/_search +{ + "query": { + "exists": { + "field": "_ignored" + } + } +} +``` +{% include copy-curl.html %} + +#### Example reponse + +```json +{ + "took": 42, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "test-ignored", + "_id": "qcf0wZABpEYH7Rw9OT7F", + "_score": 1, + "_ignored": [ + "length" + ], + "_source": { + "title": "correct text", + "length": "not a number" + } + } + ] + } +} +``` + +--- + +## Ignoring a specified field + +Similarly, you can use a term query to find documents where a specific field, such as created_at, was ignored, as shown in the following example request: ```json GET _search @@ -66,4 +145,4 @@ GET _search "hits": [] } } -``` \ No newline at end of file +``` From 61a9f6bc0ce2524a03b26259681c3dc7fdff557c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 08:47:22 -0600 Subject: [PATCH 051/166] Update index.md address final tech review feedback Signed-off-by: Melissa Vagi --- _field-types/index.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index 007e158cc5..bb6798e5bd 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -12,11 +12,11 @@ redirect_from: # Mappings and field types -Mappings define how documents and their fields are stored and indexed in OpenSearch. You can specify field types (for example, `year` as `date`) to ensure efficient storage and querying. While dynamic mappings automatically add data and fields, explicit mappings are recommended for enforcing consistent data structures and optimizing performance, especially for large datasets or high-volume indexing operations. +Mappings tell OpenSearch how to store and index your documents and their fields. You can specify the data type for each field (for example, `year` as `date`) to make storage and querying more efficient. -Explicit mappings allow you to define the exact structure and data types of your fields upfront, ensuring data integrity and efficient storage and querying. Even if you are unsure about your data structure initially, it is advisable to switch to explicit mappings once you have a better understanding of your data to avoid potential performance issues and maintain consistency. +While dynamic mappings automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. -For example, with explicit mappings, you can ensure that year is treated as text and age as an integer, preventing OpenSearch from interpreting both as integers through dynamic mapping. +Even if you are unsure about your data structure at first, switching to explicit mappings later is a good idea once you understand your data better. This helps avoid potential performance problems and keeps your data consistent. For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. ## Dynamic mapping @@ -68,7 +68,7 @@ This mapping configuration dynamically maps any field with a name starting with ### Dynamic mapping parameters -The `dynamic_templates` support the following parameters for matching conditions and mapping rules: +The `dynamic_templates` support the following parameters for matching conditions and mapping rules. The default vaule is `null`. Parameter | Description | ----------|-------------| @@ -82,7 +82,7 @@ Parameter | Description | ## Explicit mapping -If you know exactly what your field data types need to be, you can specify them in your request body when creating your index. +If you know exactly what your field data types need to be, you can specify them in your request body when creating your index, for example, as shown in the following request: ```json PUT sample-index1 @@ -108,7 +108,7 @@ PUT sample-index1 ``` {% include copy-curl.html %} -To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method: +To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method, for example, as shown in the following request: ```json POST sample-index1/_mapping @@ -142,7 +142,6 @@ OpenSearch has certain limits or settings related to mappings, such as the setti | index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | | index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | - --- ## Get a mapping From 40d9b4345c7de699941fb52f97fa7e52e59be8d6 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 08:58:06 -0600 Subject: [PATCH 052/166] Update index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/_field-types/index.md b/_field-types/index.md index bb6798e5bd..cc3f47288f 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -16,7 +16,7 @@ Mappings tell OpenSearch how to store and index your documents and their fields. While dynamic mappings automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. -Even if you are unsure about your data structure at first, switching to explicit mappings later is a good idea once you understand your data better. This helps avoid potential performance problems and keeps your data consistent. For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. +Even if you are unsure about your data structure at first, switching to explicit mappings later is recommended once you understand your data better. This helps avoid potential performance problems and keeps your data consistent. For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. ## Dynamic mapping @@ -82,7 +82,7 @@ Parameter | Description | ## Explicit mapping -If you know exactly what your field data types need to be, you can specify them in your request body when creating your index, for example, as shown in the following request: +If you know exactly what your field data types need to be, then you can specify them in your request body when creating your index, for example, as shown in the following request: ```json PUT sample-index1 @@ -127,7 +127,7 @@ You cannot change the mapping of an existing field, you can only modify the fiel ## Mapping parameters -Mapping parameters are used to configure the behavior of fields in an index. See the [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) page for more information. +Mapping parameters are used to configure the behavior of fields in an index. See [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) for more information. ## Mapping limit settings @@ -135,12 +135,12 @@ OpenSearch has certain limits or settings related to mappings, such as the setti | Setting | Default value | Allowed value | Type | Description | |-|-|-|-|-| -| index.mapping.nested_fields.limit | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. | -| index.mapping.nested_objects.limit | 10000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created within a single document. | -| index.mapping.total_fields.limit | 1000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | -| index.mapping.depth.limit | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | -| index.mapping.field_name_length.limit | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | -| index.mapper.dynamic | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | +| `index.mapping.nested_fields.limit` | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. | +| `index.mapping.nested_objects.limit` | 10000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created within a single document. | +| `index.mapping.total_fields.limit` | 1000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | +| `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | +| `index.mapping.field_name_length.limit` | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | +| `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | --- From c7d958588ebc7d9fcb4e9f3c58ff7bf0cc95a462 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 10:16:00 -0600 Subject: [PATCH 053/166] Update _field-types/mappings-use-cases.md Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index 650905f534..ad612843ba 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -45,7 +45,7 @@ PUT /test-index/_doc/1 ``` {% include copy-curl.html %} -This indexed IP address does not throw an error because `ignore_malformed` is set to true. +This indexed IP address does not throw an error because `ignore_malformed` is set to `true`. You can query the index using the following request: From dc2792efcb5a2c185aeca3d2d2aaf69b670718e8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 12:29:28 -0600 Subject: [PATCH 054/166] Update mappings-use-cases.md address tech review comments Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index ad612843ba..a0cd3d8692 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -18,7 +18,7 @@ The following example shows how to create a mapping to specify that OpenSearch s ### Create an index with an `ip` mapping -To create an index, use a PUT request: +To create an index with an `ip` mapping, use a PUT request, like the following request: ```json PUT /test-index @@ -35,7 +35,7 @@ PUT /test-index ``` {% include copy-curl.html %} -You can add a document that has a malformed IP address to your index: +You can then add a document with a malformed IP address, like the following request: ```json PUT /test-index/_doc/1 @@ -45,16 +45,14 @@ PUT /test-index/_doc/1 ``` {% include copy-curl.html %} -This indexed IP address does not throw an error because `ignore_malformed` is set to `true`. - -You can query the index using the following request: +When you query the index, the `ip_address` field will be ignored. You can query the index using the following request: ```json GET /test-index/_search ``` {% include copy-curl.html %} -The response shows that the `ip_address` field is ignored in the indexed document: +#### Response ```json { @@ -90,12 +88,11 @@ The response shows that the `ip_address` field is ignored in the indexed documen ``` {% include copy-curl.html %} - --- ## Mapping string fields to `text` and `keyword` types -This request creates an index named `movies1` with a dynamic template that maps all string fields to both `text` and `keyword` types. +To create an index named `movies1` with a dynamic template that maps all string fields to both `text` and `keyword` types, you can use the following request: ```json PUT movies1 @@ -121,3 +118,5 @@ PUT movies1 } ``` {% include copy-curl.html %} + +This dynamic template ensures that any string fields in your documents will be indexed as both a full-text `text` type and a `keyword` type. From 83bfc2594dd3acd0f4399afb068952244d1b2f62 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 12:33:08 -0600 Subject: [PATCH 055/166] Update field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index c3092c2e51..f410204779 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -8,9 +8,9 @@ parent: Metadata fields # Field names -The `_field_names` field indexes the names of fields within a document that contain non-null values. This field support the `exists` query, which identifies documents with or without non-null values for a specified field. +The `_field_names` field indexes the names of fields within a document that contain non-null values. This enables the use of the `exists` query, which can identify documents that have or do not have non-null values for a specified field. -`_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query remains functional but does not rely on `_field_names`. +However, `_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query still functions but will not rely on the `_field_names` field. ## Mapping example From 43890935dfdb453fbe886b5a98cd5bc231c47757 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 12:40:21 -0600 Subject: [PATCH 056/166] Update id.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 940c02b6b5..64da3a017b 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -8,9 +8,9 @@ parent: Metadata fields # ID -Each document has an `_id` field that uniquely identifies it. This field is indexed, allowing documents to be retrieved either through the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). +Each document in OpenSearch has a unique `_id` field that identifies it. This field is indexed, allowing you to retrieve documents using the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). -The following examples creates an index `test-index1` and add two documents with different `_id` values: +The following example creates an index `test-index1` and adds two documents with different `_id` values: ```json PUT test-index1/_doc/1 @@ -25,7 +25,7 @@ PUT test-index1/_doc/2?refresh=true ``` {% include copy-curl.html %} -Now, you can query the documents using the `_id` field: +You can then query the documents using the `_id` field: ```json GET test-index1/_search @@ -39,7 +39,7 @@ GET test-index1/_search ``` {% include copy-curl.html %} -The following response shows that this query returns both documents with `_id` values of `1` and `2`. +This returns returns both documents with `_id` values of `1` and `2`. ```json { @@ -79,8 +79,9 @@ The following response shows that this query returns both documents with `_id` v ``` {% include copy-curl.html %} -A unique `_id` is automatically generated and assigned if you do not provide one. +If you do not provide an `_id` value, then OpenSearch automatically generates a unique one for the document. {: .note} -## Querying on the `_id` field -While the `_id` field is accessible in various queries, it is restricted from use in aggregations, sorting, and scripting. See [IDs query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/) for an example of the field's usage. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the content of the `_id` field into another field that has `doc_values` enabled. +## Limitations of the `_id` field + +While the `_id` field can be used in various queries, it is restricted from use in aggregations, sorting, and scripting. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the `_id` content into another field with `doc_values` enabled. Refer to [IDs query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/) for an example of using the field. From 02af07073f4fa097eee7857a106b35acd515513d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 13:51:40 -0600 Subject: [PATCH 057/166] Update ignored.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 04b0fe42f3..8dcebeb780 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -8,16 +8,16 @@ parent: Metadata fields # Ignored -The `_ignored` field indexes and stores the name of fields within a document that were ignored during the indexing process due to being malformed. This functionality is enabled with the `ignore_malformed` setting is turned on in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/#mapping-example-usage). +The `_ignored` field helps you manage and understand issues related to malformed data in your documents. This field is used to index and store the name of fields within a document that were ignored during the indexing process, when the `ignore_malformed` setting is enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/#mapping-example-usage). The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understadning issues related to malformed data in your documents. -You can query the `_ignored` field using `term`, `terms`, and `exists` queries, and the results will be in the search hits. +You can query the `_ignored` field using `term`, `terms`, and `exists` queries, and the results will be included in the search hits. -The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated. +The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), then malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated. {: .note} -The basic syntax for a request with `_ignored` is as follows: +The following request is an example of how to use the `_ignored` field: ```json GET _search @@ -35,7 +35,7 @@ GET _search #### Example indexing request with `_ignored` field -The following example request adds a new document to the `test-ignored` index, with `ignore_malformed` set to `true` so that no error is thrown during indexing: +The following request is an example of how to add a new document to the `test-ignored` index with `ignore_malformed` set to `true` so that no error is thrown during indexing: ```json PUT test-ignored @@ -110,7 +110,7 @@ GET test-ignored/_search ## Ignoring a specified field -Similarly, you can use a term query to find documents where a specific field, such as created_at, was ignored, as shown in the following example request: +You can use a `term` query to find documents where a specific field was ignored, for example, as in the following request: ```json GET _search From f83fddf49719a3ed80fce532bbc16c6c9e300cc8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:04:03 -0600 Subject: [PATCH 058/166] Update index-metadata.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index b41f2aae85..daf76372f2 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -10,7 +10,7 @@ parent: Metadata fields When querying across multiple indexes, you may need to filter results based on the index a document was indexed into. The `index` field matches documents based on their index. -The following example creates two indexes, `products` and `customers`, and adds a document to each index: +The following request is an example of how to create two indexes, `products` and `customers`, and add a document to each index: ```json PUT products/_doc/1 @@ -25,7 +25,7 @@ PUT customers/_doc/2 ``` {% include copy-curl.html %} -Now, you can query both indexes and filter the results using the `_index` field, as shown in the following example: +You can then query both indexes and filter the results using the `_index` field, as shown in the following request: ```json GET products,customers/_search @@ -71,16 +71,9 @@ In this example: ## Querying on the `_index` field -The `_index` field is a special field in OpenSearch that represents the index a document was indexed into. You can use this field in your queries to filter results based on the index. +The `_index` field is a special field that represents the index a document was indexed into. You can use this field in your queries to filter, aggregate, sort, or retrieve index information for your search results. -Some common use cases for querying on the `_index` field include: - -- Filtering search results to only include documents from specific indexes. -- Performing aggregations to get counts or statistics for each index. -- Sorting search results by the index they belong to. -- Retrieving the index name for each document in the search results. - -The `_index` field is automatically added to every document, so you can use it in your queries like any other field. For instance, you can use the `terms` query to match documents from multiple indexes, as shown in the following example: +Since the `_index` field is automatically added to every document, you can use it in your queries like any other field. For example, you can use the `terms` query to match documents from multiple indexes. The following query example returns all documents from the `products` and `customers` indexes: ```json { @@ -92,5 +85,3 @@ The `_index` field is automatically added to every document, so you can use it i } ``` {% include copy-curl.html %} - -This query returns all documents from the `products` and `customers` indexes. From 9c819eb8ead098c016a040225c0f2ceefd752db7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:08:14 -0600 Subject: [PATCH 059/166] Update index.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index 7823e233d9..24db1bb25d 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -8,7 +8,7 @@ has_toc: false # Metadata fields -OpenSearch has built-in metadata fields that provide information about the documents in an index. These fields can be accessed or used in queries as needed. +OpenSearch provides built-in metadata fields that allow you to access information about the documents in an an index. These fields can be used in your queries as needed. Metadata fields | Description :--- | :--- From 20bdec2af5c0252dc8f8259d0610d8636f650cbb Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:14:17 -0600 Subject: [PATCH 060/166] Update meta.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index ea01b2ba23..b84a6dae69 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -8,16 +8,11 @@ parent: Metadata fields # Meta -The `_meta` field is a mapping property that allows you to associate custom metadata with your index mappings. This metadata is not used by OpenSearch itself, but can be used by your application to store information that is relevant to your use case. The `meta` field can be useful for a variety of scenarios, such as the following: - -- **Versioning**: Store the version of your application or index schema, making it easier to manage changes over time. -- **Ownership**: Identify the owner or author of an index, which can be helpful for maintenance and collaboration. -- **Categorization**: Classify indexes based on the application or domain they belong to, making it easier to organize and manage your data. -- **Auditing**: Track changes to the index metadata over time, providing a history of modifications. +The `_meta` field is a mapping property that allows you to attach custom metadata to your index mappings. This metadata can be used by your application to store information relevant to your use case, such as versioning, ownership, categorization, or auditing. ## Usage -You can define the `_meta` field when creating a new index or updating an existing index's mapping, as shown in the following example: +You can define the `_meta` field when creating a new index or updating an existing index's mapping, for example, as shown in the following request: ```json PUT my-index @@ -44,7 +39,7 @@ PUT my-index In this example, three custom metadata fields are added: application, version, and author. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the version of the application, or the author of the index. -You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, as shown in the following example: +You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, for example, as shown in the following request: ```json PUT my-index/_mapping @@ -60,14 +55,14 @@ PUT my-index/_mapping ## Retrieving meta information -You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, as shown in the following example: +You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, for example, as shown in the following request: ```json GET my-index/_mapping ``` {% include copy-curl.html %} -This returns the full mapping for the index, including the `_meta` field, as shown in the following example: +This returns the full mapping for the index, including the `_meta` field, for example, as shown in the following response: ```json { From e38ec494db9df810a0ba7901ee8c00de40a9149e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:16:08 -0600 Subject: [PATCH 061/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index efec265eb9..792a8c280d 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -8,7 +8,7 @@ parent: Metadata fields # Routing -Documents are routed to specific shards within an index using a hashing algorithm. By default, the document's `_id` field is used as the routing value, but you can also specify a custom routing value for each document. +OpenSearch uses a hashing algorithm to route documents to specific shards within an index. By default, the document's `_id` field is used as the routing value, but you can also specify a custom routing value for each document. ## Default routing From 8f22a9c970b3f9150d85b087944c92a50ae8ad57 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:16:22 -0600 Subject: [PATCH 062/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 792a8c280d..33adce1414 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -2,7 +2,6 @@ layout: default title: Routing nav_order: 35 -has_children: false parent: Metadata fields --- From c1a231582f80ea1219d47157cf0ebc038763b22d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:16:35 -0600 Subject: [PATCH 063/166] Update _field-types/metadata-fields/meta.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index b84a6dae69..ef8867a9ef 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -2,7 +2,6 @@ layout: default title: Meta nav_order: 30 -has_children: false parent: Metadata fields --- From bd8ac77f71ddc7e7371d5984a53cc2ff8530315b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:16:46 -0600 Subject: [PATCH 064/166] Update _field-types/metadata-fields/source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 3ecff61b6d..52bf2d4fb5 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -2,7 +2,6 @@ layout: default title: Source nav_order: 40 -has_children: false parent: Metadata fields --- From cf4222ab3634fc48ae433495fee2544237271c35 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:17:06 -0600 Subject: [PATCH 065/166] Update _field-types/index.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index cc3f47288f..5f85a85828 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -14,7 +14,7 @@ redirect_from: Mappings tell OpenSearch how to store and index your documents and their fields. You can specify the data type for each field (for example, `year` as `date`) to make storage and querying more efficient. -While dynamic mappings automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. +While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. Even if you are unsure about your data structure at first, switching to explicit mappings later is recommended once you understand your data better. This helps avoid potential performance problems and keeps your data consistent. For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. From 243357aa2d7093f8b2156c185f93c5ad247c203c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:18:05 -0600 Subject: [PATCH 066/166] Update _field-types/metadata-fields/field-names.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index f410204779..0a75ee0de1 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -2,7 +2,6 @@ layout: default title: _field_names nav_order: 10 -has_children: false parent: Metadata fields --- From 375488b655fb23bbdc32e797a4d8ff6b5840e398 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:25:41 -0600 Subject: [PATCH 067/166] Update source.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 52bf2d4fb5..8b16f8854a 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -7,11 +7,11 @@ parent: Metadata fields # Source -The `_source` field contains the original JSON document body that was passed at index time. This field is not indexed and is not searchable but is stored so that it can be returned when executing fethc requests such as `get` and `search`. +The `_source` field contains the original JSON document body that was indexed. While this field is not searchable, it is stored so that the full document can be returned when executing fetch requests, such as `get` and `search`. ## Disabling the field -You can disable the `_source` field as shown in the following example: +You can disable the `_source` field by setting the `enabled` parameter to `false`, as shown in the following request: ```json PUT sample-index1 @@ -25,12 +25,12 @@ PUT sample-index1 ``` {% include copy-curl.html %} -Disabling the `_source` field can cause certain featues to become not supported, such as `update`, `update_by_query`, and `reindex` APIs and the ability to debug queries or aggregations by using the the original document at index time. +Disabling the `_source` field can impact the availability of certain features, such as the `update`, `update_by_query`, and `reindex` APIs, as well as the ability to debug queries or aggregations using the original indexed document. {: .warning} ## Including or excluding fields -You can prune contents of the `_source` field after the document has been indexed but before the field is stored. You can use the `includes` and `excludes` parameters to control which field are included in the stored `_source` field, as shown in the following example: +You can selectively control the contents of the `_source` field by using the `includes` and `excludes` parameters. This allows you to prune the stored `_source` field after indexing, but before it is saved. See the following example request: ```json PUT logs From 74b82c3d2f1f237a20510ef3a5c8d4d2215974e6 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:26:03 -0600 Subject: [PATCH 068/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index daf76372f2..fa78946600 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -71,7 +71,7 @@ In this example: ## Querying on the `_index` field -The `_index` field is a special field that represents the index a document was indexed into. You can use this field in your queries to filter, aggregate, sort, or retrieve index information for your search results. +The `_index` field represents the index a document was indexed into. You can use this field in your queries to filter, aggregate, sort, or retrieve index information for your search results. Since the `_index` field is automatically added to every document, you can use it in your queries like any other field. For example, you can use the `terms` query to match documents from multiple indexes. The following query example returns all documents from the `products` and `customers` indexes: From 1262baeeab916ebfe229291e2bd29f26d873a115 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:26:17 -0600 Subject: [PATCH 069/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index fa78946600..aca38cd1ea 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -2,7 +2,6 @@ layout: default title: Index nav_order: 25 -has_children: false parent: Metadata fields --- From 2e2c00bc4fb4a8c31700d3e0d09d019c71a011f9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:26:34 -0600 Subject: [PATCH 070/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 8dcebeb780..5c3377868d 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -12,7 +12,7 @@ The `_ignored` field helps you manage and understand issues related to malformed The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understadning issues related to malformed data in your documents. -You can query the `_ignored` field using `term`, `terms`, and `exists` queries, and the results will be included in the search hits. +You can query the `_ignored` field using the `term`, `terms`, and `exists` queries, and the results will be included in the search hits. The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), then malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated. {: .note} From 1d622dfcdf5fa60618fd05f8e9996ebdebfbe793 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:26:49 -0600 Subject: [PATCH 071/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 5c3377868d..fb8078d0bb 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -10,7 +10,7 @@ parent: Metadata fields The `_ignored` field helps you manage and understand issues related to malformed data in your documents. This field is used to index and store the name of fields within a document that were ignored during the indexing process, when the `ignore_malformed` setting is enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/#mapping-example-usage). -The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understadning issues related to malformed data in your documents. +The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understanding issues related to malformed data in your documents. You can query the `_ignored` field using the `term`, `terms`, and `exists` queries, and the results will be included in the search hits. From 022eb23d81322c5fdbe7492dd4022c719759274f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:27:06 -0600 Subject: [PATCH 072/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index fb8078d0bb..14ef6240d3 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -2,7 +2,6 @@ layout: default title: Ignored nav_order: 15 -has_children: false parent: Metadata fields --- From c23b8050c1e2f6e1f525ddccff80bf761483b493 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:27:45 -0600 Subject: [PATCH 073/166] Update _field-types/metadata-fields/id.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 64da3a017b..f770d1cace 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -2,7 +2,6 @@ layout: default title: ID nav_order: 20 -has_children: false parent: Metadata fields --- From 9f7694e86baae2556b7838c77905efcf631caf41 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:30:25 -0600 Subject: [PATCH 074/166] Update id.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index f770d1cace..b6454a1591 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -75,8 +75,8 @@ This returns returns both documents with `_id` values of `1` and `2`. } ] } - ``` - {% include copy-curl.html %} +``` +{% include copy-curl.html %} If you do not provide an `_id` value, then OpenSearch automatically generates a unique one for the document. {: .note} From 97cb34886fa39d2053d51b03f1b51a17ad2ef0ce Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:35:10 -0600 Subject: [PATCH 075/166] Update _field-types/index.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 5f85a85828..b33f172020 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -16,7 +16,7 @@ Mappings tell OpenSearch how to store and index your documents and their fields. While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. -Even if you are unsure about your data structure at first, switching to explicit mappings later is recommended once you understand your data better. This helps avoid potential performance problems and keeps your data consistent. For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. +For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. ## Dynamic mapping From 4524c1a7bcd1a6fcdaefba1593e0d63966462640 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:35:16 -0600 Subject: [PATCH 076/166] Update _field-types/index.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index b33f172020..c9e526542f 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -68,7 +68,7 @@ This mapping configuration dynamically maps any field with a name starting with ### Dynamic mapping parameters -The `dynamic_templates` support the following parameters for matching conditions and mapping rules. The default vaule is `null`. +The `dynamic_templates` support the following parameters for matching conditions and mapping rules. The default value is `null`. Parameter | Description | ----------|-------------| From 27c40c85dcbe80c6f235e2715dbbd197a8df6468 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:35:51 -0600 Subject: [PATCH 077/166] Update _field-types/index.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index c9e526542f..6fce59f087 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -207,4 +207,4 @@ The response contains the specified fields: ## Mappings use cases -See use case examples, including mapping string fields and ignoring malformed IP addresses, in [Mappings use cases](). +See use case examples, including mapping string fields and ignoring malformed IP addresses, in [Mappings use cases]({{site.url}}{{site.baseurl}}/field-types/mappings-use-cases/). From c33c047d86995cac155e26710fdbb91d8a71b65a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:44:45 -0600 Subject: [PATCH 078/166] Update _field-types/metadata-fields/index.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index 24db1bb25d..fb060f7afb 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -1,4 +1,4 @@ --- +--- layout: default title: Metadata fields nav_order: 90 From 596fa72c2af5c6f65f00b04f3ba043322b4da13f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:44:51 -0600 Subject: [PATCH 079/166] Update _field-types/metadata-fields/id.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index b6454a1591..a61d66ec40 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -38,7 +38,7 @@ GET test-index1/_search ``` {% include copy-curl.html %} -This returns returns both documents with `_id` values of `1` and `2`. +This returns both documents with `_id` values of `1` and `2`. ```json { From 57cbe5eba14a3ce17a18e4db54def70709194daa Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:46:29 -0600 Subject: [PATCH 080/166] Update field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 0a75ee0de1..dfd22a9659 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -13,6 +13,7 @@ However, `_field_names` only indexes field names when both `doc_values` and `nor ## Mapping example +```json { "mappings": { "_field_names": { @@ -38,4 +39,5 @@ However, `_field_names` only indexes field names when both `doc_values` and `nor } } } +``` {% include copy-curl.html %} From 3d9f7ba715be60c9f3346da5277613f3bcef062f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:55:25 -0600 Subject: [PATCH 081/166] Address doc review comments Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- _field-types/metadata-fields/ignored.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index dfd22a9659..d22d9e59db 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -1,6 +1,6 @@ --- layout: default -title: _field_names +title: Field names nav_order: 10 parent: Metadata fields --- diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 14ef6240d3..fc36dfae88 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -1,13 +1,13 @@ --- layout: default title: Ignored -nav_order: 15 +nav_order: 25 parent: Metadata fields --- # Ignored -The `_ignored` field helps you manage and understand issues related to malformed data in your documents. This field is used to index and store the name of fields within a document that were ignored during the indexing process, when the `ignore_malformed` setting is enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/#mapping-example-usage). +The `_ignored` field helps you manage and understand issues related to malformed data in your documents. This field is used to index and store the name of fields within a document that were ignored during the indexing process, when the `ignore_malformed` setting is enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/). The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understanding issues related to malformed data in your documents. From 60ecace82263fdab074a2a5eea5b3e3cc9aed3ff Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 28 Aug 2024 15:58:15 -0600 Subject: [PATCH 082/166] Address doc review comments Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index a61d66ec40..f84a100689 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -9,6 +9,9 @@ parent: Metadata fields Each document in OpenSearch has a unique `_id` field that identifies it. This field is indexed, allowing you to retrieve documents using the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). +If you do not provide an `_id` value, then OpenSearch automatically generates a unique one for the document. +{: .note} + The following example creates an index `test-index1` and adds two documents with different `_id` values: ```json @@ -24,7 +27,7 @@ PUT test-index1/_doc/2?refresh=true ``` {% include copy-curl.html %} -You can then query the documents using the `_id` field: +You can then query the documents using the `_id` field, as shown in the following request: ```json GET test-index1/_search @@ -38,7 +41,7 @@ GET test-index1/_search ``` {% include copy-curl.html %} -This returns both documents with `_id` values of `1` and `2`. +This returns both documents with `_id` values of `1` and `2`, as shown in the following response: ```json { @@ -77,9 +80,6 @@ This returns both documents with `_id` values of `1` and `2`. } ``` {% include copy-curl.html %} - -If you do not provide an `_id` value, then OpenSearch automatically generates a unique one for the document. -{: .note} ## Limitations of the `_id` field From 30313fe5c2bb1ec458eaf272159c3adc59722da9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 08:58:15 -0600 Subject: [PATCH 083/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 6fce59f087..43229d20e3 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -14,7 +14,7 @@ redirect_from: Mappings tell OpenSearch how to store and index your documents and their fields. You can specify the data type for each field (for example, `year` as `date`) to make storage and querying more efficient. -While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. +While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps to maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. From 4f0b8e42989f0ade1bf8774843436bc25427b580 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 08:58:28 -0600 Subject: [PATCH 084/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 43229d20e3..2a4b1f76e7 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -16,7 +16,7 @@ Mappings tell OpenSearch how to store and index your documents and their fields. While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps to maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations. -For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer, instead of both being interpreted as integers by dynamic mapping. +For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer instead of both being interpreted as integers by dynamic mapping. ## Dynamic mapping From fc62bcd476fffb04f79ebb24661f44a1bb12656a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 08:58:48 -0600 Subject: [PATCH 085/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 2a4b1f76e7..6370bea54c 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -26,7 +26,7 @@ When you index a document, OpenSearch adds fields automatically with dynamic map Type | Description :--- | :--- -`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values. +`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if the field has no value. `boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` `float` | A single-precision 32-bit floating point number. `double` | A double-precision 64-bit floating point number. From 3cb42e138342e8925a4293377b1366cb5b922b9c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 08:58:57 -0600 Subject: [PATCH 086/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 6370bea54c..dc16da7543 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -28,7 +28,7 @@ Type | Description :--- | :--- `null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if the field has no value. `boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` -`float` | A single-precision 32-bit floating point number. +`float` | A single-precision, 32-bit floating-point number. `double` | A double-precision 64-bit floating point number. `integer` | A signed 32-bit number. `object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. From 53354fe7bd28fafd16177f5f00f29d138fd2ff48 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 08:59:05 -0600 Subject: [PATCH 087/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index dc16da7543..866f87f943 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -29,7 +29,7 @@ Type | Description `null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if the field has no value. `boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.` `float` | A single-precision, 32-bit floating-point number. -`double` | A double-precision 64-bit floating point number. +`double` | A double-precision, 64-bit floating-point number. `integer` | A signed 32-bit number. `object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. `array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements, not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. From cfc99972e2fb1284a9cfcfa090da521308627b8e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 08:59:33 -0600 Subject: [PATCH 088/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 866f87f943..7abf1ef84d 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -32,7 +32,7 @@ Type | Description `double` | A double-precision, 64-bit floating-point number. `integer` | A signed 32-bit number. `object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. -`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements, not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. +`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements---not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. `text` | A string sequence of characters that represent full-text values. `keyword` | A string sequence of structured characters, such as an email address or ZIP code. date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date. From 31597db564eefb275b837d1d355796df379b1601 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:02:10 -0600 Subject: [PATCH 089/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 7abf1ef84d..094b15a144 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -32,7 +32,7 @@ Type | Description `double` | A double-precision, 64-bit floating-point number. `integer` | A signed 32-bit number. `object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`. -`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements---not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, containing complex objects or other arrays, are also supported for advanced data modeling. +`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements---not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, which may contain complex objects or other arrays, can also be used for advanced data modeling. `text` | A string sequence of characters that represent full-text values. `keyword` | A string sequence of structured characters, such as an email address or ZIP code. date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date. From 95e7ef9302ca125d26228bd21e48eb8110634992 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:02:23 -0600 Subject: [PATCH 090/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 094b15a144..323eaccd50 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -40,7 +40,7 @@ numeric detection string | If disabled, OpenSearch may automatically process num ### Dynamic templates -Dynamic templates are used to define custom mappings for dynamically added fields based on data type, field name, or field path. They allow you to define a flexible schema for your data, which can automatically adapt to changes in the structure or format of the input data. +Dynamic templates are used to define custom mappings for dynamically added fields based on the data type, field name, or field path. They allow you to define a flexible schema for your data that can automatically adapt to changes in the structure or format of the input data. You can use the following syntax to define a dynamic mapping template: From d67b97432e5040a967403a115fc1d0b6e50557f7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:02:38 -0600 Subject: [PATCH 091/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 323eaccd50..14b8e01650 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -73,7 +73,7 @@ The `dynamic_templates` support the following parameters for matching conditions Parameter | Description | ----------|-------------| `match_mapping_type` | Specifies the JSON data type (for example, string, long, double, object, binary, Boolean, date) that triggers the mapping. -`match` | A regular expression to match field names and apply the mapping. +`match` | A regular expression used to match field names and apply the mapping. `unmatch` | A regular expression to exclude field names from the mapping. `match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`. `path_match` | Allows matching nested field paths using a regular expression. From 5ad15d9de0c053a867275432ceb06c4ba18b8637 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:03:01 -0600 Subject: [PATCH 092/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 14b8e01650..237259af3b 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -74,7 +74,7 @@ Parameter | Description | ----------|-------------| `match_mapping_type` | Specifies the JSON data type (for example, string, long, double, object, binary, Boolean, date) that triggers the mapping. `match` | A regular expression used to match field names and apply the mapping. -`unmatch` | A regular expression to exclude field names from the mapping. +`unmatch` | A regular expression used to exclude field names from the mapping. `match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`. `path_match` | Allows matching nested field paths using a regular expression. `path_unmatch` | Excludes nested field paths from the mapping using a regular expression. From 6a30fafae68c78b0fa82f644f10b8097e23732e7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:03:17 -0600 Subject: [PATCH 093/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 237259af3b..d95d600230 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -76,7 +76,7 @@ Parameter | Description | `match` | A regular expression used to match field names and apply the mapping. `unmatch` | A regular expression used to exclude field names from the mapping. `match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`. -`path_match` | Allows matching nested field paths using a regular expression. +`path_match` | Allows you to match nested field paths using a regular expression. `path_unmatch` | Excludes nested field paths from the mapping using a regular expression. `mapping` | The mapping configuration to apply for matching fields. From 28a29234d282a19ae6b375373268283df881a68a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:03:25 -0600 Subject: [PATCH 094/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index d95d600230..f6fc6ef02e 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -78,7 +78,7 @@ Parameter | Description | `match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`. `path_match` | Allows you to match nested field paths using a regular expression. `path_unmatch` | Excludes nested field paths from the mapping using a regular expression. -`mapping` | The mapping configuration to apply for matching fields. +`mapping` | The mapping configuration to apply. ## Explicit mapping From c25d86b904167aa80858657a063b22c13660b507 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:03:40 -0600 Subject: [PATCH 095/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index f6fc6ef02e..cc4ac1aebe 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -82,7 +82,7 @@ Parameter | Description | ## Explicit mapping -If you know exactly what your field data types need to be, then you can specify them in your request body when creating your index, for example, as shown in the following request: +If you know exactly which field data types you need to use, then you can specify them in your request body when creating your index, as shown in the following example request: ```json PUT sample-index1 From 01131a44cf4356def8ae200d9765e5e13da22651 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:03:57 -0600 Subject: [PATCH 096/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index cc4ac1aebe..f19886db1d 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -108,7 +108,7 @@ PUT sample-index1 ``` {% include copy-curl.html %} -To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method, for example, as shown in the following request: +To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method, as shown in the following example request: ```json POST sample-index1/_mapping From a49b1ac3291e08fa2facf14660e88f77be2d5731 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:04:05 -0600 Subject: [PATCH 097/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index f19886db1d..ab40e8d43d 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -127,7 +127,7 @@ You cannot change the mapping of an existing field, you can only modify the fiel ## Mapping parameters -Mapping parameters are used to configure the behavior of fields in an index. See [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) for more information. +Mapping parameters are used to configure the behavior of index fields. See [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) for more information. ## Mapping limit settings From e2d2d0d5bdab55d7e2dd1d2078188f2c93c8d6b5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:04:15 -0600 Subject: [PATCH 098/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index ab40e8d43d..06ddb68b93 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -131,7 +131,7 @@ Mapping parameters are used to configure the behavior of index fields. See [Mapp ## Mapping limit settings -OpenSearch has certain limits or settings related to mappings, such as the settings listed in the following table. Settings can be configured based on your requirements. +OpenSearch has certain mapping limits and settings, such as the settings listed in the following table. Settings can be configured based on your requirements. | Setting | Default value | Allowed value | Type | Description | |-|-|-|-|-| From bfa295227670787e1ecb01b7ea2a4875c1c9a6e1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:04:29 -0600 Subject: [PATCH 099/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 06ddb68b93..abc959ab90 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -136,7 +136,7 @@ OpenSearch has certain mapping limits and settings, such as the settings listed | Setting | Default value | Allowed value | Type | Description | |-|-|-|-|-| | `index.mapping.nested_fields.limit` | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. | -| `index.mapping.nested_objects.limit` | 10000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created within a single document. | +| `index.mapping.nested_objects.limit` | 10,000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created in a single document. | | `index.mapping.total_fields.limit` | 1000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | | `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | | `index.mapping.field_name_length.limit` | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | From 9744ab7943562cf89d1de592a4d3b5bce445419a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:04:37 -0600 Subject: [PATCH 100/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index abc959ab90..a2b0feea04 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -137,7 +137,7 @@ OpenSearch has certain mapping limits and settings, such as the settings listed |-|-|-|-|-| | `index.mapping.nested_fields.limit` | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. | | `index.mapping.nested_objects.limit` | 10,000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created in a single document. | -| `index.mapping.total_fields.limit` | 1000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | +| `index.mapping.total_fields.limit` | 1,000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | | `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | | `index.mapping.field_name_length.limit` | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | | `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | From 4898e094f0c03e1854e9d1350925596ee45696a4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:04:48 -0600 Subject: [PATCH 101/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index a2b0feea04..895a440037 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -139,7 +139,7 @@ OpenSearch has certain mapping limits and settings, such as the settings listed | `index.mapping.nested_objects.limit` | 10,000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created in a single document. | | `index.mapping.total_fields.limit` | 1,000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | | `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | -| `index.mapping.field_name_length.limit` | 50000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | +| `index.mapping.field_name_length.limit` | 50,000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | | `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | --- From c44a46a8c4f016577377a9cd5db73ed265b45893 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:04:56 -0600 Subject: [PATCH 102/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 895a440037..5f353667e0 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -140,7 +140,7 @@ OpenSearch has certain mapping limits and settings, such as the settings listed | `index.mapping.total_fields.limit` | 1,000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. | | `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. | | `index.mapping.field_name_length.limit` | 50,000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. | -| `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be added dynamically to the mapping when they are encountered in a document. | +| `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be dynamically added to a mapping. | --- From e1cef5fad0c76584d76f3e88c6a292ccdc84d046 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:05:33 -0600 Subject: [PATCH 103/166] Update _field-types/index.md Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 5f353667e0..56c5b4978f 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -170,7 +170,7 @@ GET //_mapping/field/ ``` {% include copy-curl.html %} -Both `` and `` can be specified as one value or a comma-separated list. For example, the following request retrieves the mapping for the `year` and `age` fields in `sample-index1`: +Both `` and `` can be specified as either one value or a comma-separated list. For example, the following request retrieves the mapping for the `year` and `age` fields in `sample-index1`: ```json GET sample-index1/_mapping/field/year,age From 7eaefbcdef637cb93029eafdba4c13864a43fc30 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:05:58 -0600 Subject: [PATCH 104/166] Update _field-types/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/index.md b/_field-types/index.md index 56c5b4978f..e9250f409d 100644 --- a/_field-types/index.md +++ b/_field-types/index.md @@ -207,4 +207,4 @@ The response contains the specified fields: ## Mappings use cases -See use case examples, including mapping string fields and ignoring malformed IP addresses, in [Mappings use cases]({{site.url}}{{site.baseurl}}/field-types/mappings-use-cases/). +See [Mappings use cases]({{site.url}}{{site.baseurl}}/field-types/mappings-use-cases/) for use case examples, including examples of mapping string fields and ignoring malformed IP addresses. From 7f268dc340ff23bc818978a2dcef3177744eb5b5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:06:10 -0600 Subject: [PATCH 105/166] Update _field-types/mappings-use-cases.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index a0cd3d8692..8d02c169f7 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -8,7 +8,7 @@ nav_exclude: true # Mappings use cases -Mappings in OpenSearch provide control over how data is indexed and queried, enabling optimized performance and efficient storage for a range of use cases. +Mappings provide control over how data is indexed and queried, enabling optimized performance and efficient storage for a range of use cases. --- From 71e6ed4d6897ad96cd35b2f151b111598ab5b119 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:06:20 -0600 Subject: [PATCH 106/166] Update _field-types/mappings-use-cases.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index 8d02c169f7..51cd54ed40 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -14,7 +14,7 @@ Mappings provide control over how data is indexed and queried, enabling optimize ## Example: Ignoring malformed IP addresses -The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`. +The following example shows you how to create a mapping specifying that OpenSearch should ignore any documents containing malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You can accomplish this by setting the `ignore_malformed` parameter to `true`. ### Create an index with an `ip` mapping From e49c1d38343215b600c0b13cb61e23c4e1eb41a7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:06:28 -0600 Subject: [PATCH 107/166] Update _field-types/mappings-use-cases.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index 51cd54ed40..900f6921e5 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -18,7 +18,7 @@ The following example shows you how to create a mapping specifying that OpenSear ### Create an index with an `ip` mapping -To create an index with an `ip` mapping, use a PUT request, like the following request: +To create an index with an `ip` mapping, use a PUT request: ```json PUT /test-index From a708be79cda67f4d729959bb58c0b1e1310183d9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:06:38 -0600 Subject: [PATCH 108/166] Update _field-types/mappings-use-cases.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index 900f6921e5..ec1407d55e 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -35,7 +35,7 @@ PUT /test-index ``` {% include copy-curl.html %} -You can then add a document with a malformed IP address, like the following request: +Then add a document with a malformed IP address: ```json PUT /test-index/_doc/1 From 5ee0a4db4280eeb77e0bb3e10f4979dffde44333 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:07:02 -0600 Subject: [PATCH 109/166] Update _field-types/mappings-use-cases.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/mappings-use-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/mappings-use-cases.md b/_field-types/mappings-use-cases.md index ec1407d55e..835e030bab 100644 --- a/_field-types/mappings-use-cases.md +++ b/_field-types/mappings-use-cases.md @@ -92,7 +92,7 @@ GET /test-index/_search ## Mapping string fields to `text` and `keyword` types -To create an index named `movies1` with a dynamic template that maps all string fields to both `text` and `keyword` types, you can use the following request: +To create an index named `movies1` with a dynamic template that maps all string fields to both the `text` and `keyword` types, you can use the following request: ```json PUT movies1 From 8f2c11e6e74a123dc2efe9fa7f75f6887d8913d4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:08:40 -0600 Subject: [PATCH 110/166] Update _field-types/metadata-fields/field-names.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index d22d9e59db..301956f561 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -7,7 +7,7 @@ parent: Metadata fields # Field names -The `_field_names` field indexes the names of fields within a document that contain non-null values. This enables the use of the `exists` query, which can identify documents that have or do not have non-null values for a specified field. +The `_field_names` field indexes field names that contain non-null values. This enables the use of the `exists` query, which can identify documents that either have or do not have non-null values for a specified field. However, `_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query still functions but will not rely on the `_field_names` field. From e5bc821ab864c7fd4be4b14de225e762d1f001d8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:10:57 -0600 Subject: [PATCH 111/166] Update _field-types/metadata-fields/id.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index f84a100689..7123435c27 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -7,7 +7,7 @@ parent: Metadata fields # ID -Each document in OpenSearch has a unique `_id` field that identifies it. This field is indexed, allowing you to retrieve documents using the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). +Each document in OpenSearch has a unique `_id` field. This field is indexed, allowing you to retrieve documents using the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). If you do not provide an `_id` value, then OpenSearch automatically generates a unique one for the document. {: .note} From 1c9328d50703a544b41933d3ea2359c80a0d986c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:28:13 -0600 Subject: [PATCH 112/166] Update _field-types/metadata-fields/id.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 7123435c27..2fb3dda9de 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -9,7 +9,7 @@ parent: Metadata fields Each document in OpenSearch has a unique `_id` field. This field is indexed, allowing you to retrieve documents using the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). -If you do not provide an `_id` value, then OpenSearch automatically generates a unique one for the document. +If you do not provide an `_id` value, then OpenSearch automatically generates one for the document. {: .note} The following example creates an index `test-index1` and adds two documents with different `_id` values: From df556d72193352a5a7a39e6988f8455adf6be517 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:48:45 -0600 Subject: [PATCH 113/166] Update _field-types/metadata-fields/field-names.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/field-names.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/field-names.md b/_field-types/metadata-fields/field-names.md index 301956f561..b17e94fbb4 100644 --- a/_field-types/metadata-fields/field-names.md +++ b/_field-types/metadata-fields/field-names.md @@ -9,7 +9,7 @@ parent: Metadata fields The `_field_names` field indexes field names that contain non-null values. This enables the use of the `exists` query, which can identify documents that either have or do not have non-null values for a specified field. -However, `_field_names` only indexes field names when both `doc_values` and `norms` are disabled for those fields. If either `doc_values` or `norms` are enabled, the `exists` query still functions but will not rely on the `_field_names` field. +However, `_field_names` only indexes field names when both `doc_values` and `norms` are disabled. If either `doc_values` or `norms` are enabled, then the `exists` query still functions but will not rely on the `_field_names` field. ## Mapping example From e1876b66ef4ccf0215c74944428a8a799ead0c33 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:49:16 -0600 Subject: [PATCH 114/166] Update _field-types/metadata-fields/id.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 2fb3dda9de..dfad818ceb 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -7,7 +7,7 @@ parent: Metadata fields # ID -Each document in OpenSearch has a unique `_id` field. This field is indexed, allowing you to retrieve documents using the `GET` API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). +Each document in OpenSearch has a unique `_id` field. This field is indexed, allowing you to retrieve documents using the GET API or the [`ids` query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/). If you do not provide an `_id` value, then OpenSearch automatically generates one for the document. {: .note} From 420290252ae22264b8836b67445f8c9f9b810f15 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:49:37 -0600 Subject: [PATCH 115/166] Update _field-types/metadata-fields/id.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index dfad818ceb..b100f9b3c9 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -12,7 +12,7 @@ Each document in OpenSearch has a unique `_id` field. This field is indexed, all If you do not provide an `_id` value, then OpenSearch automatically generates one for the document. {: .note} -The following example creates an index `test-index1` and adds two documents with different `_id` values: +The following example request creates an index named `test-index1` and adds two documents with different `_id` values: ```json PUT test-index1/_doc/1 From b970b51aaea7de36cb30b0fb74e3e454a9e2f925 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:49:54 -0600 Subject: [PATCH 116/166] Update _field-types/metadata-fields/id.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index b100f9b3c9..7b18a5e304 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -27,7 +27,7 @@ PUT test-index1/_doc/2?refresh=true ``` {% include copy-curl.html %} -You can then query the documents using the `_id` field, as shown in the following request: +You can then query the documents using the `_id` field, as shown in the following example request: ```json GET test-index1/_search From 2aefa3cfa185f6484a8d35e528e2b270029c3048 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:50:15 -0600 Subject: [PATCH 117/166] Update _field-types/metadata-fields/id.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 7b18a5e304..8538f341c9 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -41,7 +41,7 @@ GET test-index1/_search ``` {% include copy-curl.html %} -This returns both documents with `_id` values of `1` and `2`, as shown in the following response: +The response returns both documents with `_id` values of `1` and `2`: ```json { From 381f3c90fe040be894daa0eb09d3fa2e8be0af5a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:50:47 -0600 Subject: [PATCH 118/166] Update _field-types/metadata-fields/id.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/id.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/id.md b/_field-types/metadata-fields/id.md index 8538f341c9..f66f4b8e13 100644 --- a/_field-types/metadata-fields/id.md +++ b/_field-types/metadata-fields/id.md @@ -83,4 +83,4 @@ The response returns both documents with `_id` values of `1` and `2`: ## Limitations of the `_id` field -While the `_id` field can be used in various queries, it is restricted from use in aggregations, sorting, and scripting. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the `_id` content into another field with `doc_values` enabled. Refer to [IDs query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/) for an example of using the field. +While the `_id` field can be used in various queries, it is restricted from use in aggregations, sorting, and scripting. If you need to sort or aggregate on the `_id` field, it is recommended to duplicate the `_id` content into another field with `doc_values` enabled. Refer to [IDs query]({{site.url}}{{site.baseurl}}/query-dsl/term/ids/) for an example. From 2b6ada7acb43c44281485e71eae19e0fa93b749c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:52:02 -0600 Subject: [PATCH 119/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index fc36dfae88..2fb27b8a7f 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -7,7 +7,7 @@ parent: Metadata fields # Ignored -The `_ignored` field helps you manage and understand issues related to malformed data in your documents. This field is used to index and store the name of fields within a document that were ignored during the indexing process, when the `ignore_malformed` setting is enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/). +The `_ignored` field helps you manage issues related to malformed data in your documents. This field is used to index and store field names that were ignored during the indexing process as a result of the `ignore_malformed` setting being enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/). The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understanding issues related to malformed data in your documents. From 88d3c1e8babb56f9578fe937f8b4563c87bfe17b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:52:23 -0600 Subject: [PATCH 120/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 2fb27b8a7f..e56d8b369e 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -9,7 +9,7 @@ parent: Metadata fields The `_ignored` field helps you manage issues related to malformed data in your documents. This field is used to index and store field names that were ignored during the indexing process as a result of the `ignore_malformed` setting being enabled in the [index mapping]({{site.url}}{{site.baseurl}}/field-types/). -The `_ignored` field allows you to search and identify documents that contain fields that were ignored, as well as the specific field names that were ignored. The can be useful for troubleshooting and understanding issues related to malformed data in your documents. +The `_ignored` field allows you to search for and identify documents containing fields that were ignored as well as for the specific field names that were ignored. This can be useful for troubleshooting. You can query the `_ignored` field using the `term`, `terms`, and `exists` queries, and the results will be included in the search hits. From 46c06396d1f1fe65e3b4c1391619b294c6d7994f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:52:42 -0600 Subject: [PATCH 121/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index e56d8b369e..0a563ee9b2 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -16,7 +16,7 @@ You can query the `_ignored` field using the `term`, `terms`, and `exists` queri The `_ignored` field is only populated when the `ignore_malformed` setting is enabled in your index mapping. If `ignore_malformed` is set to `false` (the default value), then malformed fields will cause the entire document to be rejected, and the `_ignored` field will not be populated. {: .note} -The following request is an example of how to use the `_ignored` field: +The following example request shows you how to use the `_ignored` field: ```json GET _search From 9087d129025e94033a944b15889fba465966a00b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:52:58 -0600 Subject: [PATCH 122/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 0a563ee9b2..370267283c 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -32,7 +32,7 @@ GET _search --- -#### Example indexing request with `_ignored` field +#### Example indexing request with the `_ignored` field The following request is an example of how to add a new document to the `test-ignored` index with `ignore_malformed` set to `true` so that no error is thrown during indexing: From f1a195d342cc3b91241d81eefec10231b61bfb64 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:53:19 -0600 Subject: [PATCH 123/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 370267283c..2f810172f0 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -34,7 +34,7 @@ GET _search #### Example indexing request with the `_ignored` field -The following request is an example of how to add a new document to the `test-ignored` index with `ignore_malformed` set to `true` so that no error is thrown during indexing: +The following example request adds a new document to the `test-ignored` index with `ignore_malformed` set to `true` so that no error is thrown during indexing: ```json PUT test-ignored From e53ec2af22345be044f1d51135b21311688c667a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:54:06 -0600 Subject: [PATCH 124/166] Update _field-types/metadata-fields/ignored.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/ignored.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/ignored.md b/_field-types/metadata-fields/ignored.md index 2f810172f0..e867cfc754 100644 --- a/_field-types/metadata-fields/ignored.md +++ b/_field-types/metadata-fields/ignored.md @@ -109,7 +109,7 @@ GET test-ignored/_search ## Ignoring a specified field -You can use a `term` query to find documents where a specific field was ignored, for example, as in the following request: +You can use a `term` query to find documents in which a specific field was ignored, as shown in the following example request: ```json GET _search From e90eb171276719a8c2c3634f686e937fef97ba50 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:54:30 -0600 Subject: [PATCH 125/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index aca38cd1ea..6e73fb5410 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -7,7 +7,7 @@ parent: Metadata fields # Index -When querying across multiple indexes, you may need to filter results based on the index a document was indexed into. The `index` field matches documents based on their index. +When querying across multiple indexes, you may need to filter results based on the index into which a document was indexed. The `index` field matches documents based on their index. The following request is an example of how to create two indexes, `products` and `customers`, and add a document to each index: From b700ff7dc21fcc2138ee8a237e9e934cffd30fa7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:54:48 -0600 Subject: [PATCH 126/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 6e73fb5410..4538adbfa7 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -9,7 +9,7 @@ parent: Metadata fields When querying across multiple indexes, you may need to filter results based on the index into which a document was indexed. The `index` field matches documents based on their index. -The following request is an example of how to create two indexes, `products` and `customers`, and add a document to each index: +The following example request creates two indexes, `products` and `customers`, and adds a document to each index: ```json PUT products/_doc/1 From 670328ff49783acf9adb951273e1e590cd24379c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:55:06 -0600 Subject: [PATCH 127/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 4538adbfa7..0421f9920c 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -24,7 +24,7 @@ PUT customers/_doc/2 ``` {% include copy-curl.html %} -You can then query both indexes and filter the results using the `_index` field, as shown in the following request: +You can then query both indexes and filter the results using the `_index` field, as shown in the following example request: ```json GET products,customers/_search From b9ccc47fa88f95de40e232387bc70cde3f075b0a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:55:25 -0600 Subject: [PATCH 128/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 0421f9920c..32d0cdb6bd 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -66,7 +66,7 @@ In this example: - The `query` section uses a `terms` query to match documents from the `products` and `customers` indexes. - The `aggs` section performs a `terms` aggregation on the `_index` field, grouping the results by index. - The `sort` section sorts the results by the `_index` field in ascending order. -- The `script_fields` section adds a new field `index_name` to the search results that contains the value of the `_index` field for each document. +- The `script_fields` section adds a new field called `index_name` to the search results containing the `_index` field value for each document. ## Querying on the `_index` field From 116cecbeb97126c283ddbe592895f5d14664cc5e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:55:44 -0600 Subject: [PATCH 129/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 32d0cdb6bd..3cefd31c18 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -70,7 +70,7 @@ In this example: ## Querying on the `_index` field -The `_index` field represents the index a document was indexed into. You can use this field in your queries to filter, aggregate, sort, or retrieve index information for your search results. +The `_index` field represents the index into which a document was indexed. You can use this field in your queries to filter, aggregate, sort, or retrieve index information for your search results. Since the `_index` field is automatically added to every document, you can use it in your queries like any other field. For example, you can use the `terms` query to match documents from multiple indexes. The following query example returns all documents from the `products` and `customers` indexes: From ce89a8333a000232854e837e69ba0896be25645b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:56:07 -0600 Subject: [PATCH 130/166] Update _field-types/metadata-fields/index-metadata.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index-metadata.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index-metadata.md b/_field-types/metadata-fields/index-metadata.md index 3cefd31c18..657f7d62a5 100644 --- a/_field-types/metadata-fields/index-metadata.md +++ b/_field-types/metadata-fields/index-metadata.md @@ -72,7 +72,7 @@ In this example: The `_index` field represents the index into which a document was indexed. You can use this field in your queries to filter, aggregate, sort, or retrieve index information for your search results. -Since the `_index` field is automatically added to every document, you can use it in your queries like any other field. For example, you can use the `terms` query to match documents from multiple indexes. The following query example returns all documents from the `products` and `customers` indexes: +Because the `_index` field is automatically added to every document, you can use it in your queries like any other field. For example, you can use the `terms` query to match documents from multiple indexes. The following example query returns all documents from the `products` and `customers` indexes: ```json { From 7bc5bfdcd490db1bed7ea84752b876be691da805 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:56:26 -0600 Subject: [PATCH 131/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index fb060f7afb..294699416f 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -8,7 +8,7 @@ has_toc: false # Metadata fields -OpenSearch provides built-in metadata fields that allow you to access information about the documents in an an index. These fields can be used in your queries as needed. +OpenSearch provides built-in metadata fields that allow you to access information about the documents in an index. These fields can be used in your queries as needed. Metadata fields | Description :--- | :--- From 7b051f5b0a6d8c396c972ab6ff844c30d4385d32 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:56:38 -0600 Subject: [PATCH 132/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index 294699416f..f0017d04d3 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -10,7 +10,7 @@ has_toc: false OpenSearch provides built-in metadata fields that allow you to access information about the documents in an index. These fields can be used in your queries as needed. -Metadata fields | Description +Metadata field | Description :--- | :--- `_field_names` | The fields within the document that hold non-empty or non-null values. `_ignored` | The fields in the document that were disregarded during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. From 7213aa9fc67e4996872a127cf15d70ffaf5738d3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:56:59 -0600 Subject: [PATCH 133/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index f0017d04d3..7d8ef92efd 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -12,7 +12,7 @@ OpenSearch provides built-in metadata fields that allow you to access informatio Metadata field | Description :--- | :--- -`_field_names` | The fields within the document that hold non-empty or non-null values. +`_field_names` | The document fields with non-empty or non-null values. `_ignored` | The fields in the document that were disregarded during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. `_id` | The unique identifier assigned to each individual document. `_index` | The specific index within the OpenSearch database where the document is stored and organized. From 581b11802149bf2707dd8d1624a7728a49e54c9e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:57:18 -0600 Subject: [PATCH 134/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index 7d8ef92efd..c2694ae30a 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -13,7 +13,7 @@ OpenSearch provides built-in metadata fields that allow you to access informatio Metadata field | Description :--- | :--- `_field_names` | The document fields with non-empty or non-null values. -`_ignored` | The fields in the document that were disregarded during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. +`_ignored` | The document fields that were ignored during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. `_id` | The unique identifier assigned to each individual document. `_index` | The specific index within the OpenSearch database where the document is stored and organized. `_meta` | Stores custom metadata or additional information specific to the application or use case. From 916a0dc01a1bd576f74e8fb308b369a4404ff22f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:57:34 -0600 Subject: [PATCH 135/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index c2694ae30a..4dfeeedbcc 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -14,7 +14,7 @@ Metadata field | Description :--- | :--- `_field_names` | The document fields with non-empty or non-null values. `_ignored` | The document fields that were ignored during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. -`_id` | The unique identifier assigned to each individual document. +`_id` | The unique identifier assigned to each document. `_index` | The specific index within the OpenSearch database where the document is stored and organized. `_meta` | Stores custom metadata or additional information specific to the application or use case. `_routing` | Allows you to specify a custom value that determines the shard assignment for the document within the OpenSearch cluster. From 9d8a1e75a745af9426f12946562f48b4f466bb29 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:57:54 -0600 Subject: [PATCH 136/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index 4dfeeedbcc..057e7b494d 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -15,7 +15,7 @@ Metadata field | Description `_field_names` | The document fields with non-empty or non-null values. `_ignored` | The document fields that were ignored during the indexing process due to the presence of malformed data, as specified by the `ignore_malformed` setting. `_id` | The unique identifier assigned to each document. -`_index` | The specific index within the OpenSearch database where the document is stored and organized. +`_index` | The index in which the document is stored. `_meta` | Stores custom metadata or additional information specific to the application or use case. `_routing` | Allows you to specify a custom value that determines the shard assignment for the document within the OpenSearch cluster. `_source` | Contains the original JSON representation of the document's data. From 09e64a53ed03ccc0d5dff8a7f720e7017ee0b8ac Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:58:06 -0600 Subject: [PATCH 137/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index 057e7b494d..e77b5d0db8 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -17,5 +17,5 @@ Metadata field | Description `_id` | The unique identifier assigned to each document. `_index` | The index in which the document is stored. `_meta` | Stores custom metadata or additional information specific to the application or use case. -`_routing` | Allows you to specify a custom value that determines the shard assignment for the document within the OpenSearch cluster. +`_routing` | Allows you to specify a custom value that determines the shard assignment for a document in an OpenSearch cluster. `_source` | Contains the original JSON representation of the document's data. From 234d6a0f14a50260d717a1f9b9b3c699900f609a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:58:22 -0600 Subject: [PATCH 138/166] Update _field-types/metadata-fields/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/index.md b/_field-types/metadata-fields/index.md index e77b5d0db8..cdc079e1e5 100644 --- a/_field-types/metadata-fields/index.md +++ b/_field-types/metadata-fields/index.md @@ -18,4 +18,4 @@ Metadata field | Description `_index` | The index in which the document is stored. `_meta` | Stores custom metadata or additional information specific to the application or use case. `_routing` | Allows you to specify a custom value that determines the shard assignment for a document in an OpenSearch cluster. -`_source` | Contains the original JSON representation of the document's data. +`_source` | Contains the original JSON representation of the document data. From 230c83051b306bb4a62ae3545435a2c054dea9f4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:58:39 -0600 Subject: [PATCH 139/166] Update _field-types/metadata-fields/meta.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index ef8867a9ef..8a53db4b82 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -11,7 +11,7 @@ The `_meta` field is a mapping property that allows you to attach custom metadat ## Usage -You can define the `_meta` field when creating a new index or updating an existing index's mapping, for example, as shown in the following request: +You can define the `_meta` field when creating a new index or updating an existing index's mapping, as shown in the following example request: ```json PUT my-index From 5b2452619c07356c8b82cbd1a218052413b01542 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:59:03 -0600 Subject: [PATCH 140/166] Update _field-types/metadata-fields/meta.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 8a53db4b82..5ca35e6b9a 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -36,7 +36,7 @@ PUT my-index ``` {% include copy-curl.html %} -In this example, three custom metadata fields are added: application, version, and author. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the version of the application, or the author of the index. +In this example, three custom metadata fields are added: `application`, `version`, and `author`. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the application version, or the author of the index. You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, for example, as shown in the following request: From f360bafdd17cf2a4ace493b880b25b1656c32fbc Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 09:59:15 -0600 Subject: [PATCH 141/166] Update _field-types/metadata-fields/meta.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 5ca35e6b9a..368d580122 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -38,7 +38,7 @@ PUT my-index In this example, three custom metadata fields are added: `application`, `version`, and `author`. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the application version, or the author of the index. -You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, for example, as shown in the following request: +You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, as shown in the following example request: ```json PUT my-index/_mapping From c8936665d21cf9e64ec1334a70c2f57da582f25e Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:03:54 -0600 Subject: [PATCH 142/166] Update _field-types/metadata-fields/meta.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 368d580122..9492bea71b 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -52,7 +52,7 @@ PUT my-index/_mapping ``` {% include copy-curl.html %} -## Retrieving meta information +## Retrieving `meta` information You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, for example, as shown in the following request: From 580258944f45e9c675c7eb0f4757db9c4c06a120 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:06:57 -0600 Subject: [PATCH 143/166] Update _field-types/metadata-fields/meta.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 9492bea71b..64575927db 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -54,7 +54,7 @@ PUT my-index/_mapping ## Retrieving `meta` information -You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, for example, as shown in the following request: +You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, as shown in the following example request: ```json GET my-index/_mapping From 35522223c84c1c7d7001066b3e33aa4192550cba Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:11:49 -0600 Subject: [PATCH 144/166] Update _field-types/metadata-fields/meta.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 64575927db..cab63236a5 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -38,7 +38,7 @@ PUT my-index In this example, three custom metadata fields are added: `application`, `version`, and `author`. These fields can be used by your application to store any relevant information about the index, such as the application it belongs to, the application version, or the author of the index. -You can update the `_meta` field using the [put mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, as shown in the following example request: +You can update the `_meta` field using the [Put Mapping API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/put-mapping/) operation, as shown in the following example request: ```json PUT my-index/_mapping From ae73ae8952d8b96b352d62066c719e54941db269 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:12:43 -0600 Subject: [PATCH 145/166] Update _field-types/metadata-fields/meta.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index cab63236a5..228d7f0491 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -54,7 +54,7 @@ PUT my-index/_mapping ## Retrieving `meta` information -You can retrieve the `_meta` information for an index using the [get mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, as shown in the following example request: +You can retrieve the `_meta` information for an index using the [Get Mapping API]({{site.url}}{{site.baseurl}}/field-types/#get-a-mapping) operation, as shown in the following example request: ```json GET my-index/_mapping From d2cb1b5c3d27943a1dfa38b954dbba19e63b4bd8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:14:16 -0600 Subject: [PATCH 146/166] Update _field-types/metadata-fields/meta.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/meta.md b/_field-types/metadata-fields/meta.md index 228d7f0491..220d58f106 100644 --- a/_field-types/metadata-fields/meta.md +++ b/_field-types/metadata-fields/meta.md @@ -61,7 +61,7 @@ GET my-index/_mapping ``` {% include copy-curl.html %} -This returns the full mapping for the index, including the `_meta` field, for example, as shown in the following response: +The response returns the full index mapping, including the `_meta` field: ```json { From b0f0b929682c8564fc8303c7cd4b1a1955185cc8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:14:37 -0600 Subject: [PATCH 147/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 33adce1414..b3473e3239 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -7,7 +7,7 @@ parent: Metadata fields # Routing -OpenSearch uses a hashing algorithm to route documents to specific shards within an index. By default, the document's `_id` field is used as the routing value, but you can also specify a custom routing value for each document. +OpenSearch uses a hashing algorithm to route documents to specific shards in an index. By default, the document's `_id` field is used as the routing value, but you can also specify a custom routing value for each document. ## Default routing From cd6b1aac91337081aa9d0c43cebe9ef04ec98042 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:14:51 -0600 Subject: [PATCH 148/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index b3473e3239..eb0ca5a329 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -11,7 +11,7 @@ OpenSearch uses a hashing algorithm to route documents to specific shards in an ## Default routing -The following formula is the default routing formula used in OpenSearch. The `_routing` value is the document's `_id`. +The following is the default OpenSearch routing formula. The `_routing` value is the document's `_id`. ```json shard_num = hash(_routing) % num_primary_shards From 2e8ff8bbbbab7c6fe0750c393a1f1f4521f05277 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:15:05 -0600 Subject: [PATCH 149/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index eb0ca5a329..ea688c0e5d 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -19,7 +19,7 @@ shard_num = hash(_routing) % num_primary_shards ## Custom routing -You can specify a custom routing value when indexing a document, as shown in the following example: +You can specify a custom routing value when indexing a document, as shown in the following example request: ```json PUT sample-index1/_doc/1?routing=JohnDoe1 From a5074df95e89c83abbc1358ec7cc89b8f9cdb08c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:15:26 -0600 Subject: [PATCH 150/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index ea688c0e5d..5228371a1a 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -31,7 +31,7 @@ PUT sample-index1/_doc/1?routing=JohnDoe1 In this example, the document is routed using the value `JohnDoe1` instead of the default `_id`. -You must provide the same routing value when retrieving, deleting, or updating the document, as shown in the following example: +You must provide the same routing value when retrieving, deleting, or updating the document, as shown in the following example request: ```json GET sample-index1/_doc/1?routing=JohnDoe1 From 9654a1b552cb9fdfa9ed94f257d379d92ae820e6 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:15:39 -0600 Subject: [PATCH 151/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 5228371a1a..78b4261e6a 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -40,7 +40,7 @@ GET sample-index1/_doc/1?routing=JohnDoe1 ## Querying by routing -You can query documents based in their routing value using the `_routing` field, as shown in the following example. This query only searches the shard(s) associated with the `JohnDoe1` routing value. +You can query documents based on their routing value by using the `_routing` field, as shown in the following example. This query only searches the shard(s) associated with the `JohnDoe1` routing value: ```json GET sample-index1/_search From 097f3ef0e2125470afb09de4f654d747effb60c3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:22:29 -0600 Subject: [PATCH 152/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 78b4261e6a..9f407b9ffa 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -56,7 +56,7 @@ GET sample-index1/_search ## Required routing -You can make custom routing a required field for all CRUD operations on an index, as shown in the following example. If you try to index a document without providing a routing value, OpenSearch throws an exception. +You can make custom routing a required field for all CRUD operations on an index, as shown in the following example request. If you try to index a document without providing a routing value, OpenSearch will throw an exception. . ```json From 5e263e3890627d1f458aef0863b4ae466d51e5ba Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:22:39 -0600 Subject: [PATCH 153/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 9f407b9ffa..9493a141d7 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -57,7 +57,6 @@ GET sample-index1/_search ## Required routing You can make custom routing a required field for all CRUD operations on an index, as shown in the following example request. If you try to index a document without providing a routing value, OpenSearch will throw an exception. -. ```json PUT sample-index2 From 421e97fed4400d3cccf07a2926688b6bb4fa7121 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:24:03 -0600 Subject: [PATCH 154/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 9493a141d7..83d310f288 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -56,7 +56,7 @@ GET sample-index1/_search ## Required routing -You can make custom routing a required field for all CRUD operations on an index, as shown in the following example request. If you try to index a document without providing a routing value, OpenSearch will throw an exception. +You can make custom routing required for all CRUD operations on an index, as shown in the following example request. If you try to index a document without providing a routing value, OpenSearch will throw an exception. ```json PUT sample-index2 From c834b5a56bb02b85004a88282563c7c913b5b902 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:25:10 -0600 Subject: [PATCH 155/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 83d310f288..1c8d8630a3 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -70,7 +70,7 @@ PUT sample-index2 ``` {% include copy-curl.html %} -## Routing to index partitions +## Routing to specific shards You can configure an index to route documents to a subset of shards, rather than a single shard. This is done using the `index.routing_partition_size` setting, as shown in the following example: From 044fb6c2a05e750b0259e74010f2f0fa374c6274 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:25:29 -0600 Subject: [PATCH 156/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 1c8d8630a3..8b9e4dbf54 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -72,7 +72,7 @@ PUT sample-index2 ## Routing to specific shards -You can configure an index to route documents to a subset of shards, rather than a single shard. This is done using the `index.routing_partition_size` setting, as shown in the following example: +You can configure an index to route documents to a subset of shards rather than a single shard by using the `index.routing_partition_size` setting, as shown in the following example request: ```json PUT sample-index3 From ba7b39b5870687a838b651249b7eecf20ad0ec3a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:25:47 -0600 Subject: [PATCH 157/166] Update _field-types/metadata-fields/routing.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 8b9e4dbf54..25eb38f4a5 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -89,6 +89,6 @@ PUT sample-index3 ``` {% include copy-curl.html %} -In this example, the documents are routed to one of four partitions within the index, based on a combination of the `_routing` and `_id` fields. +In this example, the documents are routed to one of four partitions in the index, based on a combination of the `_routing` and `_id` fields. Partitioned indexes have some limitations, such as not supporting `join` field relationships in the mappings. From dca4c323be66c9d947befaafdb564c77095b4cce Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:29:50 -0600 Subject: [PATCH 158/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 25eb38f4a5..d18cb78ef8 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -89,6 +89,6 @@ PUT sample-index3 ``` {% include copy-curl.html %} -In this example, the documents are routed to one of four partitions in the index, based on a combination of the `_routing` and `_id` fields. +In this example, the documents are routed to one of four routing partitions in the index, based on a combination of the `_routing` and `_id` fields. Partitioned indexes have some limitations, such as not supporting `join` field relationships in the mappings. From 92a949e3aafec5ff75ce3f3a242c1a5f340b86ad Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:44:20 -0600 Subject: [PATCH 159/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index d18cb78ef8..2fb0052419 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -91,4 +91,4 @@ PUT sample-index3 In this example, the documents are routed to one of four routing partitions in the index, based on a combination of the `_routing` and `_id` fields. -Partitioned indexes have some limitations, such as not supporting `join` field relationships in the mappings. +Index partitioning has limitations, such as not supporting `join` field relationships in the mappings. From e3d8d0be51041313957158e3638b8faeeaf100bc Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:44:42 -0600 Subject: [PATCH 160/166] Update _field-types/metadata-fields/source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 8b16f8854a..6cee6fcb0c 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -11,7 +11,7 @@ The `_source` field contains the original JSON document body that was indexed. W ## Disabling the field -You can disable the `_source` field by setting the `enabled` parameter to `false`, as shown in the following request: +You can disable the `_source` field by setting the `enabled` parameter to `false`, as shown in the following example request: ```json PUT sample-index1 From 5573bc9a517bad730d20aff2aa4efc3eb23ad7e5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:45:02 -0600 Subject: [PATCH 161/166] Update _field-types/metadata-fields/source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index 6cee6fcb0c..ffc87f198f 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -30,7 +30,7 @@ Disabling the `_source` field can impact the availability of certain features, s ## Including or excluding fields -You can selectively control the contents of the `_source` field by using the `includes` and `excludes` parameters. This allows you to prune the stored `_source` field after indexing, but before it is saved. See the following example request: +You can selectively control the contents of the `_source` field by using the `includes` and `excludes` parameters. This allows you to prune the stored `_source` field after it is indexed but before it is saved, as shown in the following example request: ```json PUT logs From eac468b19e4c8d0234c37495457c3df9f52e5878 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 10:45:19 -0600 Subject: [PATCH 162/166] Update _field-types/metadata-fields/source.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/source.md b/_field-types/metadata-fields/source.md index ffc87f198f..c9e714f43c 100644 --- a/_field-types/metadata-fields/source.md +++ b/_field-types/metadata-fields/source.md @@ -51,4 +51,4 @@ PUT logs ``` {% include copy-curl.html %} -Although these fields are not stored in the `_source`, you can still search on them, as the data is still indexed. +These fields are not stored in the `_source`, but you can still search them because the data remains indexed. From 1037b6e4b8850161a2af4de6a1b02b3816352a7a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 11:19:34 -0600 Subject: [PATCH 163/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 2fb0052419..d2e4d678f8 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -89,6 +89,6 @@ PUT sample-index3 ``` {% include copy-curl.html %} -In this example, the documents are routed to one of four routing partitions in the index, based on a combination of the `_routing` and `_id` fields. +In this example, the documents are routed to one of four shards in the index, based on a combination of the `_routing` and `_id` fields. Index partitioning has limitations, such as not supporting `join` field relationships in the mappings. From 215a3b71286b3151982429ac94d4426e9ce8edef Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 11:20:09 -0600 Subject: [PATCH 164/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index d2e4d678f8..03a384a456 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -91,4 +91,3 @@ PUT sample-index3 In this example, the documents are routed to one of four shards in the index, based on a combination of the `_routing` and `_id` fields. -Index partitioning has limitations, such as not supporting `join` field relationships in the mappings. From 0cb56be605135bba9de3e9820534ea6b6c2008e4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 11:40:25 -0600 Subject: [PATCH 165/166] Update routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 03a384a456..2d3244e077 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -72,7 +72,9 @@ PUT sample-index2 ## Routing to specific shards -You can configure an index to route documents to a subset of shards rather than a single shard by using the `index.routing_partition_size` setting, as shown in the following example request: +You can configure an index to route custom values to a subset of shards rather than a single shard. This is done by setting `index.routing_partition_size` at the time of index creation. With this setting, the formula for calculating the shard is `shard_num = (hash(_routing) + hash(_id)) % routing_partition_size) % num_primary_shards`. + +The following example request routes documents to one of four shards in the index: ```json PUT sample-index3 @@ -88,6 +90,3 @@ PUT sample-index3 } ``` {% include copy-curl.html %} - -In this example, the documents are routed to one of four shards in the index, based on a combination of the `_routing` and `_id` fields. - From 219b1e3447ae653f232924f1b833305567f2de6f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 29 Aug 2024 11:51:54 -0600 Subject: [PATCH 166/166] Update _field-types/metadata-fields/routing.md Signed-off-by: Melissa Vagi --- _field-types/metadata-fields/routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/metadata-fields/routing.md b/_field-types/metadata-fields/routing.md index 2d3244e077..9064e20c49 100644 --- a/_field-types/metadata-fields/routing.md +++ b/_field-types/metadata-fields/routing.md @@ -72,7 +72,7 @@ PUT sample-index2 ## Routing to specific shards -You can configure an index to route custom values to a subset of shards rather than a single shard. This is done by setting `index.routing_partition_size` at the time of index creation. With this setting, the formula for calculating the shard is `shard_num = (hash(_routing) + hash(_id)) % routing_partition_size) % num_primary_shards`. +You can configure an index to route custom values to a subset of shards rather than a single shard. This is done by setting `index.routing_partition_size` at the time of index creation. The formula for calculating the shard is `shard_num = (hash(_routing) + hash(_id)) % routing_partition_size) % num_primary_shards`. The following example request routes documents to one of four shards in the index: