From 08fa54f2681f54c5275bf1aafc54bf7039db2b25 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Thu, 4 Jul 2024 14:57:30 +0100 Subject: [PATCH 01/10] Adding retrieve specific fields documentation #7507 Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 836 ++++++++++++++++++++ 1 file changed, 836 insertions(+) create mode 100644 _search-plugins/retrieve-specific-fields.md diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md new file mode 100644 index 0000000000..65f492a570 --- /dev/null +++ b/_search-plugins/retrieve-specific-fields.md @@ -0,0 +1,836 @@ +--- +layout: default +title: Retrieve specific fields +nav_order: 250 +--- + +# Retrieve specific fields + +When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned for each hit. It is returned under `_source` field in the response. This can be very large amount of data that is being transferred through network without adding any additional benefit to the user, increasing latency and cost. There are different ways to limit the responses to only the required information. + +## disabling `_source` + +You can include `"_source": false` line in the search request to prevent the `_source` field from being included in the response. See following example: + +``` +GET "/index1/_search?pretty" +{ + "_source": false, + "query": { + "match_all": {} + } +} +``` + +As no fields were selected in the previous search, the retrieved hits will only include `_index`, `_id` and `_score` of the hits. As can be seen in the following example: +``` +{ + "hits" : { + "total" : { + "value" : 2, + "relation" : "eq" + }, + "max_score" : 1.0, + "hits" : [ + { + "_index" : "index1", + "_id" : "41", + "_score" : 1.0 + }, + { + "_index" : "index1", + "_id" : "51", + "_score" : 1.0 + } + ] + } +} +``` + +## Specifying the fields to retrieve + +You can list the fields of interest in the search request using `fields` parameter. Wildcard patterns are also accepted. See following example: +``` +GET "/index1/_search?pretty" +{ + "_source": false, + "fields": ["age", "nam*"], + "query": { + "match_all": {} + } +} +``` +Example response: +``` +{ + "hits" : { + "total" : { + "value" : 2, + "relation" : "eq" + }, + "max_score" : 1.0, + "hits" : [ + { + "_index" : "index1", + "_id" : "41", + "_score" : 1.0, + "fields" : { + "name" : [ + "John Doe" + ], + "age" : [ + 30 + ] + } + }, + { + "_index" : "index1", + "_id" : "51", + "_score" : 1.0, + "fields" : { + "name" : [ + "Jane Smith" + ], + "age" : [ + 25 + ] + } + } + ] + } +} +``` + +You can also use object notation, to apply a custom format for the chosen field, see following example: + +Imagine your document looks like this: + +``` +{ + "_index": "my_index", + "_type": "_doc", + "_id": "1", + "_source": { + "title": "Document 1", + "date": "2023-07-04T12:34:56Z" + } +} +``` +You can query with fields Parameter and Custom Format: +``` +GET /my_index/_search +{ + "query": { + "match_all": {} + }, + "fields": { + "date": { + "format": "yyyy-MM-dd" + } + }, + "_source": false +} +``` + + +Additionally, you can also use [Most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) with `fields`, as it queries both document `_source` and `_mapping` of the index. + +## Docvalue_fields + +`docvalue_fields` is another parameter you can use in OpenSearch to retrieve specific fields from the index, but it works slightly differently compared to the `fields` parameter. `docvalue_fields` retrieves field values from doc values rather than from the `_source` field, which is more efficient for certain types of fields, especially for keyword, date, and numeric fields. Doc values are a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. It is particularly efficient for fields that are not analyzed (like keyword fields) + +To better understand `docvalue_fields` see following example. + + +1. Create index `my_index` with the following mappings: + +``` +PUT my_index +{ + "mappings": { + "properties": { + "title": { "type": "text" }, + "author": { "type": "keyword" }, + "publication_date": { "type": "date" }, + "price": { "type": "double" } + } + } +} +``` + +2. Index the following documents using the newly created index: +``` +POST my_index/_doc/1 +{ + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2021-01-01", + "price": 29.99 +} + +POST my_index/_doc/2 +{ + "title": "Advanced OpenSearch", + "author": "Jane Smith", + "publication_date": "2022-01-01", + "price": 39.99 +} +``` +3. Retrieve only the `author` and `publication_date` fields using `docvalue_fields`: + +``` +POST my_index/_search +{ + "_source": false, + "docvalue_fields": ["author", "publication_date"], + "query": { + "match_all": {} + } +} +``` + +Expected response: +``` +{ + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1.0, + "fields": { + "author": ["John Doe"], + "publication_date": ["2021-01-01T00:00:00.000Z"] + } + }, + { + "_index": "my_index", + "_id": "2", + "_score": 1.0, + "fields": { + "author": ["Jane Smith"], + "publication_date": ["2022-01-01T00:00:00.000Z"] + } + } + ] + } +} +``` + +### Docvalue_fields with nested objects + +In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, see following example. + +1. Define the Index and Mappings. +``` +PUT my_index +{ + "mappings": { + "properties": { + "title": { "type": "text" }, + "author": { "type": "keyword" }, + "comments": { + "type": "nested", + "properties": { + "username": { "type": "keyword" }, + "content": { "type": "text" }, + "created_at": { "type": "date" } + } + } + } + } +} +``` +2. Index your data. +``` +POST my_index/_doc/1 +{ + "title": "OpenSearch Basics", + "author": "John Doe", + "comments": [ + { + "username": "alice", + "content": "Great article!", + "created_at": "2023-01-01T12:00:00Z" + }, + { + "username": "bob", + "content": "Very informative.", + "created_at": "2023-01-02T12:00:00Z" + } + ] +} +``` +3. Perform a Search with `inner_hits` and `docvalue_fields` +``` +POST my_index/_search +{ + "query": { + "nested": { + "path": "comments", + "query": { + "match_all": {} + }, + "inner_hits": { + "docvalue_fields": ["username", "created_at"] + } + } + } +} +``` + +Expected response: +``` +{ + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1.0, + "_source": { + "title": "OpenSearch Basics", + "author": "John Doe", + "comments": [ + { + "username": "alice", + "content": "Great article!", + "created_at": "2023-01-01T12:00:00Z" + }, + { + "username": "bob", + "content": "Very informative.", + "created_at": "2023-01-02T12:00:00Z" + } + ] + }, + "inner_hits": { + "comments": { + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_nested": { + "field": "comments", + "offset": 0 + }, + "docvalue_fields": { + "username": ["alice"], + "created_at": ["2023-01-01T12:00:00Z"] + } + }, + { + "_index": "my_index", + "_id": "1", + "_nested": { + "field": "comments", + "offset": 1 + }, + "docvalue_fields": { + "username": ["bob"], + "created_at": ["2023-01-02T12:00:00Z"] + } + } + ] + } + } + } + } + ] + } +} +``` + +## Stored fields + +`stored_fields` is another feature in OpenSearch that allows you to explicitly store and retrieve specific fields from documents, separate from the `_source` field. By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. + +Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. See following example. + +1. Create index and mappings +``` +PUT my_index +{ + "mappings": { + "properties": { + "title": { + "type": "text", + "store": true // Store the title field separately + }, + "author": { + "type": "keyword", + "store": true // Store the author field separately + }, + "publication_date": { + "type": "date" + }, + "price": { + "type": "double" + } + } + } +} +``` + +2. Index your data +``` +POST my_index/_doc/1 +{ + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2022-01-01", + "price": 29.99 +} + +POST my_index/_doc/2 +{ + "title": "Advanced OpenSearch", + "author": "Jane Smith", + "publication_date": "2023-01-01", + "price": 39.99 +} +``` +3. Perform a Search with `stored_fields` +``` +POST my_index/_search +{ + "_source": false, + "stored_fields": ["title", "author"], + "query": { + "match_all": {} + } +} +``` +Expected response: +``` +{ + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1.0, + "fields": { + "title": ["OpenSearch Basics"], + "author": ["John Doe"] + } + }, + { + "_index": "my_index", + "_id": "2", + "_score": 1.0, + "fields": { + "title": ["Advanced OpenSearch"], + "author": ["Jane Smith"] + } + } + ] + } +} +``` + +Stored_fields can be disabled completely in search request using `"stored_fields": "_none_"`. +{: .note} + +### Stored fields with nested objects + +In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, see following example. + +1. Create index and mappings +``` +PUT my_index +{ + "mappings": { + "properties": { + "title": { "type": "text" }, + "author": { "type": "keyword" }, + "comments": { + "type": "nested", + "properties": { + "username": { "type": "keyword", "store": true }, + "content": { "type": "text", "store": true }, + "created_at": { "type": "date", "store": true } + } + } + } + } +} +``` + +2. Index your data +``` +POST my_index/_doc/1 +{ + "title": "OpenSearch Basics", + "author": "John Doe", + "comments": [ + { + "username": "alice", + "content": "Great article!", + "created_at": "2023-01-01T12:00:00Z" + }, + { + "username": "bob", + "content": "Very informative.", + "created_at": "2023-01-02T12:00:00Z" + } + ] +} +``` + +3. Perform a Search with `inner_hits` and `stored_fields` +``` +POST my_index/_search +{ + "_source": false, + "query": { + "nested": { + "path": "comments", + "query": { + "match_all": {} + }, + "inner_hits": { + "stored_fields": ["comments.username", "comments.content", "comments.created_at"] + } + } + } +} +``` + +Expected response: +``` +{ + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1.0, + "inner_hits": { + "comments": { + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_nested": { + "field": "comments", + "offset": 0 + }, + "fields": { + "comments.username": ["alice"], + "comments.content": ["Great article!"], + "comments.created_at": ["2023-01-01T12:00:00.000Z"] + } + }, + { + "_index": "my_index", + "_id": "1", + "_nested": { + "field": "comments", + "offset": 1 + }, + "fields": { + "comments.username": ["bob"], + "comments.content": ["Very informative."], + "comments.created_at": ["2023-01-02T12:00:00.000Z"] + } + } + ] + } + } + } + } + ] + } +} +``` + +## Source filtering + +Source filtering in OpenSearch is a way to control which parts of the `_source` field are included in the search response. This can help reduce the amount of data transferred over the network and improve performance by including only the necessary fields in the response. + +You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. See following example: + +1. Index your data +``` +PUT my_index/_doc/1 +{ + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2021-01-01", + "price": 29.99 +} +``` + +2. Perform a search using source filtering +``` +POST my_index/_search +{ + "_source": ["title", "author"], + "query": { + "match_all": {} + } +} +``` +Expected response: +``` +{ + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1.0, + "_source": { + "title": "OpenSearch Basics", + "author": "John Doe" + } + } + ] + } +} +``` + +### Excluding fields with source filtering + +You can choose to exclude fields using `"excludes"` parameter in search request, see following example: + +``` +POST my_index/_search +{ + "_source": { + "excludes": ["price"] + }, + "query": { + "match_all": {} + } +} +``` + +Expected response: +``` +{ + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1.0, + "_source": { + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2021-01-01" + } + } + ] + } +} +``` + +### Including and excluding fields in the same search + +There might be cases where both `include` and `exclude` parameters are necessary. See following examples which demonstrates the usefulness of this option. + +If you have an index `products` with the following document: +``` +{ + "product_id": "123", + "name": "Smartphone", + "category": "Electronics", + "price": 699.99, + "description": "A powerful smartphone with a sleek design.", + "reviews": [ + { + "user": "john_doe", + "rating": 5, + "comment": "Great phone!", + "date": "2023-01-01" + }, + { + "user": "jane_doe", + "rating": 4, + "comment": "Good value for money.", + "date": "2023-02-15" + } + ], + "supplier": { + "name": "TechCorp", + "contact_email": "support@techcorp.com", + "address": { + "street": "123 Tech St", + "city": "Techville", + "zipcode": "12345" + } + }, + "inventory": { + "stock": 50, + "warehouse_location": "A1" + } +} +``` + +Assuming you want to perform a search, but in the response you only want to include the `name`, `price`, `reviews`, and `supplier` fields and exclude any `contact_email` fields from the `supplier` object, and `comment` fields from the `reviews` object. Following is the search you can run: + +``` +GET /products/_search +{ + "_source": { + "includes": ["name", "price", "reviews.*", "supplier.*"], + "excludes": ["reviews.comment", "supplier.contact_email"] + }, + "query": { + "match": { + "category": "Electronics" + } + } +} +``` + +Expected response: +``` +{ + "hits": { + "hits": [ + { + "_source": { + "name": "Smartphone", + "price": 699.99, + "reviews": [ + { + "user": "john_doe", + "rating": 5, + "date": "2023-01-01" + }, + { + "user": "jane_doe", + "rating": 4, + "date": "2023-02-15" + } + ], + "supplier": { + "name": "TechCorp", + "address": { + "street": "123 Tech St", + "city": "Techville", + "zipcode": "12345" + } + } + } + } + ] + } +} +``` + +## Scripted fields + +The `script_fields` parameter in OpenSearch allows you to include custom fields in your search results, where the values of these fields are computed using scripts. This can be useful for calculating values on the fly based on the data in the document. + +Following example demonstrates the power of `script_fields`. + +Let's say you have an index of products, and each product document contains the fields `price` and `discount_percentage`. You want to include a custom field in the search results that shows the discounted price of each product. + + +1. Index the data: +``` +PUT /products/_doc/123 +{ + "product_id": "123", + "name": "Smartphone", + "price": 699.99, + "discount_percentage": 10, + "category": "Electronics", + "description": "A powerful smartphone with a sleek design." +} +``` +2. Search using scripted field +You can now use the `script_fields` parameter to include a custom field called `discounted_price` in the search results. This field will be calculated based on the `price` and `discount_percentage` fields using a script. See following example: +``` +GET /products/_search +{ + "_source": ["product_id", "name", "price", "discount_percentage"], + "query": { + "match": { + "category": "Electronics" + } + }, + "script_fields": { + "discounted_price": { + "script": { + "lang": "painless", + "source": "doc['price'].value * (1 - doc['discount_percentage'].value / 100)" + } + } + } +} +``` +Example response: +``` +{ + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1.0, + "hits": [ + { + "_index": "products", + "_id": "123", + "_score": 1.0, + "_source": { + "product_id": "123", + "name": "Smartphone", + "price": 699.99, + "discount_percentage": 10 + }, + "fields": { + "discounted_price": [629.991] + } + } + ] + } +} +``` \ No newline at end of file From 12434f5b367c6d395e563d21e3062afd862c7b80 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Fri, 5 Jul 2024 11:03:15 +0100 Subject: [PATCH 02/10] Adding retrieve specific fields documentation #7507 Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 420 +++++++++++--------- 1 file changed, 225 insertions(+), 195 deletions(-) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md index 65f492a570..f893e6557b 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/retrieve-specific-fields.md @@ -6,9 +6,18 @@ nav_order: 250 # Retrieve specific fields -When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned for each hit. It is returned under `_source` field in the response. This can be very large amount of data that is being transferred through network without adding any additional benefit to the user, increasing latency and cost. There are different ways to limit the responses to only the required information. +When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned in response for each hit under the `_source` field. This can be large amounts of data that is being transferred through network without adding any additional benefit to the user, increasing latency and cost. There are different ways to limit the responses to only the required information. -## disabling `_source` +--- + +#### Table of contents +1. TOC +{:toc} + + +--- + +## Disabling _source You can include `"_source": false` line in the search request to prevent the `_source` field from being included in the response. See following example: @@ -23,6 +32,7 @@ GET "/index1/_search?pretty" ``` As no fields were selected in the previous search, the retrieved hits will only include `_index`, `_id` and `_score` of the hits. As can be seen in the following example: + ``` { "hits" : { @@ -50,6 +60,7 @@ As no fields were selected in the previous search, the retrieved hits will only ## Specifying the fields to retrieve You can list the fields of interest in the search request using `fields` parameter. Wildcard patterns are also accepted. See following example: + ``` GET "/index1/_search?pretty" { @@ -60,7 +71,9 @@ GET "/index1/_search?pretty" } } ``` + Example response: + ``` { "hits" : { @@ -101,9 +114,11 @@ Example response: } ``` -You can also use object notation, to apply a custom format for the chosen field, see following example: +### Extracting fields with custom format + +You can also use object notation, to apply a custom format to the chosen field, see following example. -Imagine your document looks like this: +If you have the following document: ``` { @@ -116,7 +131,9 @@ Imagine your document looks like this: } } ``` -You can query with fields Parameter and Custom Format: + +You can query with `fields` parameter and custom format: + ``` GET /my_index/_search { @@ -132,64 +149,64 @@ GET /my_index/_search } ``` +Additionally, you can also use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) with `fields` parameter, as it queries both document `_source` and `_mappings` of the index. -Additionally, you can also use [Most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) with `fields`, as it queries both document `_source` and `_mapping` of the index. - -## Docvalue_fields +## Searching with docvalue_fields -`docvalue_fields` is another parameter you can use in OpenSearch to retrieve specific fields from the index, but it works slightly differently compared to the `fields` parameter. `docvalue_fields` retrieves field values from doc values rather than from the `_source` field, which is more efficient for certain types of fields, especially for keyword, date, and numeric fields. Doc values are a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. It is particularly efficient for fields that are not analyzed (like keyword fields) +`docvalue_fields` is another parameter you can use in OpenSearch to retrieve specific fields from the index, but it works slightly differently compared to the `fields` parameter. The `docvalue_fields` parameter retrieves details from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values are a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. To better understand `docvalue_fields` see following example. 1. Create index `my_index` with the following mappings: -``` -PUT my_index -{ - "mappings": { - "properties": { - "title": { "type": "text" }, - "author": { "type": "keyword" }, - "publication_date": { "type": "date" }, - "price": { "type": "double" } + ``` + PUT my_index + { + "mappings": { + "properties": { + "title": { "type": "text" }, + "author": { "type": "keyword" }, + "publication_date": { "type": "date" }, + "price": { "type": "double" } + } + } } - } -} -``` + ``` 2. Index the following documents using the newly created index: -``` -POST my_index/_doc/1 -{ - "title": "OpenSearch Basics", - "author": "John Doe", - "publication_date": "2021-01-01", - "price": 29.99 -} - -POST my_index/_doc/2 -{ - "title": "Advanced OpenSearch", - "author": "Jane Smith", - "publication_date": "2022-01-01", - "price": 39.99 -} -``` + ``` + POST my_index/_doc/1 + { + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2021-01-01", + "price": 29.99 + } + + POST my_index/_doc/2 + { + "title": "Advanced OpenSearch", + "author": "Jane Smith", + "publication_date": "2022-01-01", + "price": 39.99 + } + ``` 3. Retrieve only the `author` and `publication_date` fields using `docvalue_fields`: -``` -POST my_index/_search -{ - "_source": false, - "docvalue_fields": ["author", "publication_date"], - "query": { - "match_all": {} - } -} -``` + ``` + POST my_index/_search + { + "_source": false, + "docvalue_fields": ["author", "publication_date"], + "query": { + "match_all": {} + } + } + ``` Expected response: + ``` { "hits": { @@ -222,69 +239,72 @@ Expected response: } ``` -### Docvalue_fields with nested objects +### Using docvalue_fields with nested objects In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, see following example. 1. Define the Index and Mappings. -``` -PUT my_index -{ - "mappings": { - "properties": { - "title": { "type": "text" }, - "author": { "type": "keyword" }, - "comments": { - "type": "nested", + ``` + PUT my_index + { + "mappings": { "properties": { - "username": { "type": "keyword" }, - "content": { "type": "text" }, - "created_at": { "type": "date" } + "title": { "type": "text" }, + "author": { "type": "keyword" }, + "comments": { + "type": "nested", + "properties": { + "username": { "type": "keyword" }, + "content": { "type": "text" }, + "created_at": { "type": "date" } + } + } } } } - } -} -``` + ``` + 2. Index your data. -``` -POST my_index/_doc/1 -{ - "title": "OpenSearch Basics", - "author": "John Doe", - "comments": [ - { - "username": "alice", - "content": "Great article!", - "created_at": "2023-01-01T12:00:00Z" - }, + ``` + POST my_index/_doc/1 { - "username": "bob", - "content": "Very informative.", - "created_at": "2023-01-02T12:00:00Z" + "title": "OpenSearch Basics", + "author": "John Doe", + "comments": [ + { + "username": "alice", + "content": "Great article!", + "created_at": "2023-01-01T12:00:00Z" + }, + { + "username": "bob", + "content": "Very informative.", + "created_at": "2023-01-02T12:00:00Z" + } + ] } - ] -} -``` + ``` + 3. Perform a Search with `inner_hits` and `docvalue_fields` -``` -POST my_index/_search -{ - "query": { - "nested": { - "path": "comments", + ``` + POST my_index/_search + { "query": { - "match_all": {} - }, - "inner_hits": { - "docvalue_fields": ["username", "created_at"] + "nested": { + "path": "comments", + "query": { + "match_all": {} + }, + "inner_hits": { + "docvalue_fields": ["username", "created_at"] + } + } } } - } -} -``` + ``` + + Expected response: -Expected response: ``` { "hits": { @@ -357,67 +377,70 @@ Expected response: } ``` -## Stored fields +## Searching with stored fields `stored_fields` is another feature in OpenSearch that allows you to explicitly store and retrieve specific fields from documents, separate from the `_source` field. By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. See following example. 1. Create index and mappings -``` -PUT my_index -{ - "mappings": { - "properties": { - "title": { - "type": "text", - "store": true // Store the title field separately - }, - "author": { - "type": "keyword", - "store": true // Store the author field separately - }, - "publication_date": { - "type": "date" - }, - "price": { - "type": "double" + ``` + PUT my_index + { + "mappings": { + "properties": { + "title": { + "type": "text", + "store": true // Store the title field separately + }, + "author": { + "type": "keyword", + "store": true // Store the author field separately + }, + "publication_date": { + "type": "date" + }, + "price": { + "type": "double" + } + } } } - } -} -``` + ``` 2. Index your data -``` -POST my_index/_doc/1 -{ - "title": "OpenSearch Basics", - "author": "John Doe", - "publication_date": "2022-01-01", - "price": 29.99 -} + ``` + POST my_index/_doc/1 + { + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2022-01-01", + "price": 29.99 + } + + POST my_index/_doc/2 + { + "title": "Advanced OpenSearch", + "author": "Jane Smith", + "publication_date": "2023-01-01", + "price": 39.99 + } + ``` -POST my_index/_doc/2 -{ - "title": "Advanced OpenSearch", - "author": "Jane Smith", - "publication_date": "2023-01-01", - "price": 39.99 -} -``` 3. Perform a Search with `stored_fields` -``` -POST my_index/_search -{ - "_source": false, - "stored_fields": ["title", "author"], - "query": { - "match_all": {} - } -} -``` + ``` + POST my_index/_search + { + "_source": false, + "stored_fields": ["title", "author"], + "query": { + "match_all": {} + } + } + ``` + Expected response: + ``` { "hits": { @@ -453,72 +476,73 @@ Expected response: Stored_fields can be disabled completely in search request using `"stored_fields": "_none_"`. {: .note} -### Stored fields with nested objects +### Searching stored fields with nested objects In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, see following example. 1. Create index and mappings -``` -PUT my_index -{ - "mappings": { - "properties": { - "title": { "type": "text" }, - "author": { "type": "keyword" }, - "comments": { - "type": "nested", + ``` + PUT my_index + { + "mappings": { "properties": { - "username": { "type": "keyword", "store": true }, - "content": { "type": "text", "store": true }, - "created_at": { "type": "date", "store": true } + "title": { "type": "text" }, + "author": { "type": "keyword" }, + "comments": { + "type": "nested", + "properties": { + "username": { "type": "keyword", "store": true }, + "content": { "type": "text", "store": true }, + "created_at": { "type": "date", "store": true } + } + } } } } - } -} -``` + ``` 2. Index your data -``` -POST my_index/_doc/1 -{ - "title": "OpenSearch Basics", - "author": "John Doe", - "comments": [ + ``` + POST my_index/_doc/1 { - "username": "alice", - "content": "Great article!", - "created_at": "2023-01-01T12:00:00Z" - }, - { - "username": "bob", - "content": "Very informative.", - "created_at": "2023-01-02T12:00:00Z" + "title": "OpenSearch Basics", + "author": "John Doe", + "comments": [ + { + "username": "alice", + "content": "Great article!", + "created_at": "2023-01-01T12:00:00Z" + }, + { + "username": "bob", + "content": "Very informative.", + "created_at": "2023-01-02T12:00:00Z" + } + ] } - ] -} -``` + ``` 3. Perform a Search with `inner_hits` and `stored_fields` -``` -POST my_index/_search -{ - "_source": false, - "query": { - "nested": { - "path": "comments", + ``` + POST my_index/_search + { + "_source": false, "query": { - "match_all": {} - }, - "inner_hits": { - "stored_fields": ["comments.username", "comments.content", "comments.created_at"] + "nested": { + "path": "comments", + "query": { + "match_all": {} + }, + "inner_hits": { + "stored_fields": ["comments.username", "comments.content", "comments.created_at"] + } + } } } - } -} -``` + ``` Expected response: + ``` { "hits": { @@ -577,7 +601,7 @@ Expected response: } ``` -## Source filtering +## Using source filtering Source filtering in OpenSearch is a way to control which parts of the `_source` field are included in the search response. This can help reduce the amount of data transferred over the network and improve performance by including only the necessary fields in the response. @@ -604,7 +628,9 @@ POST my_index/_search } } ``` + Expected response: + ``` { "hits": { @@ -645,6 +671,7 @@ POST my_index/_search ``` Expected response: + ``` { "hits": { @@ -729,6 +756,7 @@ GET /products/_search ``` Expected response: + ``` { "hits": { @@ -764,7 +792,7 @@ Expected response: } ``` -## Scripted fields +## Using scripted fields The `script_fields` parameter in OpenSearch allows you to include custom fields in your search results, where the values of these fields are computed using scripts. This can be useful for calculating values on the fly based on the data in the document. @@ -773,7 +801,7 @@ Following example demonstrates the power of `script_fields`. Let's say you have an index of products, and each product document contains the fields `price` and `discount_percentage`. You want to include a custom field in the search results that shows the discounted price of each product. -1. Index the data: +1. Index the data. ``` PUT /products/_doc/123 { @@ -785,7 +813,8 @@ PUT /products/_doc/123 "description": "A powerful smartphone with a sleek design." } ``` -2. Search using scripted field + +2. Search using scripted field. You can now use the `script_fields` parameter to include a custom field called `discounted_price` in the search results. This field will be calculated based on the `price` and `discount_percentage` fields using a script. See following example: ``` GET /products/_search @@ -800,12 +829,13 @@ GET /products/_search "discounted_price": { "script": { "lang": "painless", - "source": "doc['price'].value * (1 - doc['discount_percentage'].value / 100)" + "source": "doc[\"price\"].value * (1 - doc[\"discount_percentage\"].value / 100)" } } } } ``` + Example response: ``` { From 0f9c975efd4a79a596908f7c4e4d73d4bd543c97 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Tue, 9 Jul 2024 21:08:18 +0100 Subject: [PATCH 03/10] Update retrieve-specific-fields.md Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md index f893e6557b..8487162296 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/retrieve-specific-fields.md @@ -17,7 +17,7 @@ When you run a basic search in OpenSearch, by default, the original JSON objects --- -## Disabling _source +## Disabling `_source` You can include `"_source": false` line in the search request to prevent the `_source` field from being included in the response. See following example: @@ -863,4 +863,4 @@ Example response: ] } } -``` \ No newline at end of file +``` From 12214423ea93eacb7382d3a5c11f4763526d2e3c Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Tue, 9 Jul 2024 21:38:49 +0100 Subject: [PATCH 04/10] updating retrieve specific field typos Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 88 ++++++++++----------- 1 file changed, 44 insertions(+), 44 deletions(-) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md index f893e6557b..3ef64226b2 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/retrieve-specific-fields.md @@ -243,7 +243,7 @@ Expected response: In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, see following example. -1. Define the Index and Mappings. +1. Define the Index and Mappings: ``` PUT my_index { @@ -264,7 +264,7 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot } ``` -2. Index your data. +2. Index your data: ``` POST my_index/_doc/1 { @@ -285,7 +285,7 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot } ``` -3. Perform a Search with `inner_hits` and `docvalue_fields` +3. Perform a search with `inner_hits` and `docvalue_fields`: ``` POST my_index/_search { @@ -303,7 +303,7 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot } ``` - Expected response: +Expected response: ``` { @@ -383,7 +383,7 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. See following example. -1. Create index and mappings +1. Create index and mappings: ``` PUT my_index { @@ -427,7 +427,7 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for } ``` -3. Perform a Search with `stored_fields` +3. Perform a search with `stored_fields` ``` POST my_index/_search { @@ -476,11 +476,11 @@ Expected response: Stored_fields can be disabled completely in search request using `"stored_fields": "_none_"`. {: .note} -### Searching stored fields with nested objects +### Searching `stored_fields` with nested objects In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, see following example. -1. Create index and mappings +1. Create index and mappings: ``` PUT my_index { @@ -501,7 +501,7 @@ In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you c } ``` -2. Index your data +2. Index your data: ``` POST my_index/_doc/1 { @@ -522,7 +522,7 @@ In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you c } ``` -3. Perform a Search with `inner_hits` and `stored_fields` +3. Perform a search with `inner_hits` and `stored_fields`: ``` POST my_index/_search { @@ -607,27 +607,27 @@ Source filtering in OpenSearch is a way to control which parts of the `_source` You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. See following example: -1. Index your data -``` -PUT my_index/_doc/1 -{ - "title": "OpenSearch Basics", - "author": "John Doe", - "publication_date": "2021-01-01", - "price": 29.99 -} -``` +1. Index your data: + ``` + PUT my_index/_doc/1 + { + "title": "OpenSearch Basics", + "author": "John Doe", + "publication_date": "2021-01-01", + "price": 29.99 + } + ``` -2. Perform a search using source filtering -``` -POST my_index/_search -{ - "_source": ["title", "author"], - "query": { - "match_all": {} - } -} -``` +2. Perform a search using source filtering: + ``` + POST my_index/_search + { + "_source": ["title", "author"], + "query": { + "match_all": {} + } + } + ``` Expected response: @@ -738,7 +738,7 @@ If you have an index `products` with the following document: } ``` -Assuming you want to perform a search, but in the response you only want to include the `name`, `price`, `reviews`, and `supplier` fields and exclude any `contact_email` fields from the `supplier` object, and `comment` fields from the `reviews` object. Following is the search you can run: +If you want to perform a search, but in the response you only want to include the `name`, `price`, `reviews`, and `supplier` fields and exclude any `contact_email` fields from the `supplier` object, and `comment` fields from the `reviews` object. Following is the search you can run: ``` GET /products/_search @@ -801,20 +801,20 @@ Following example demonstrates the power of `script_fields`. Let's say you have an index of products, and each product document contains the fields `price` and `discount_percentage`. You want to include a custom field in the search results that shows the discounted price of each product. -1. Index the data. -``` -PUT /products/_doc/123 -{ - "product_id": "123", - "name": "Smartphone", - "price": 699.99, - "discount_percentage": 10, - "category": "Electronics", - "description": "A powerful smartphone with a sleek design." -} -``` +1. Index the data: + ``` + PUT /products/_doc/123 + { + "product_id": "123", + "name": "Smartphone", + "price": 699.99, + "discount_percentage": 10, + "category": "Electronics", + "description": "A powerful smartphone with a sleek design." + } + ``` -2. Search using scripted field. +2. Search using `script_fields`: You can now use the `script_fields` parameter to include a custom field called `discounted_price` in the search results. This field will be calculated based on the `price` and `discount_percentage` fields using a script. See following example: ``` GET /products/_search From 406b619d1988c346449ca75364b2552b66f8cf6a Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Tue, 9 Jul 2024 21:45:37 +0100 Subject: [PATCH 05/10] updating retrieve specific field typos Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md index 130d8c13d7..5ce1ae32da 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/retrieve-specific-fields.md @@ -151,7 +151,7 @@ GET /my_index/_search Additionally, you can also use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) with `fields` parameter, as it queries both document `_source` and `_mappings` of the index. -## Searching with docvalue_fields +## Searching with `docvalue_fields` `docvalue_fields` is another parameter you can use in OpenSearch to retrieve specific fields from the index, but it works slightly differently compared to the `fields` parameter. The `docvalue_fields` parameter retrieves details from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values are a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. @@ -239,7 +239,7 @@ Expected response: } ``` -### Using docvalue_fields with nested objects +### Using `docvalue_fields` with nested objects In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, see following example. @@ -377,7 +377,7 @@ Expected response: } ``` -## Searching with stored fields +## Searching with `stored_fields` `stored_fields` is another feature in OpenSearch that allows you to explicitly store and retrieve specific fields from documents, separate from the `_source` field. By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. @@ -408,7 +408,7 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for } ``` -2. Index your data +2. Index your data: ``` POST my_index/_doc/1 { @@ -427,7 +427,7 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for } ``` -3. Perform a search with `stored_fields` +3. Perform a search with `stored_fields`: ``` POST my_index/_search { From d49ab0f471c8f520fbff1441fe232c0301ad9e69 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Fri, 12 Jul 2024 11:29:37 +0100 Subject: [PATCH 06/10] adding details per comments Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md index 5ce1ae32da..218c99cea0 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/retrieve-specific-fields.md @@ -57,6 +57,17 @@ As no fields were selected in the previous search, the retrieved hits will only } ``` +The `_source` can also be disabled in mappings for the index using the following configuration: +``` +"mappings": { + "_source": { + "enabled": false + } +} +``` + +In such case, using methods like [searching with `docvalue_files`]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with `stored fields`]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. + ## Specifying the fields to retrieve You can list the fields of interest in the search request using `fields` parameter. Wildcard patterns are also accepted. See following example: @@ -794,7 +805,7 @@ Expected response: ## Using scripted fields -The `script_fields` parameter in OpenSearch allows you to include custom fields in your search results, where the values of these fields are computed using scripts. This can be useful for calculating values on the fly based on the data in the document. +The `script_fields` parameter in OpenSearch allows you to include custom fields in your search results, where the values of these fields are computed using scripts. This can be useful for calculating values on the fly based on the data in the document. You can also retrieve `derived fields` using similar approach, see [Retrieving fields]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/derived/#retrieving-fields) for further details. Following example demonstrates the power of `script_fields`. From 331a629ea645432ef5b1a25f18309e62a186f8d2 Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Thu, 15 Aug 2024 16:06:53 +0100 Subject: [PATCH 07/10] Apply suggestions from code review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: AntonEliatra --- _search-plugins/retrieve-specific-fields.md | 59 ++++++++++----------- 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/retrieve-specific-fields.md index 218c99cea0..46d5015110 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/retrieve-specific-fields.md @@ -6,7 +6,7 @@ nav_order: 250 # Retrieve specific fields -When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned in response for each hit under the `_source` field. This can be large amounts of data that is being transferred through network without adding any additional benefit to the user, increasing latency and cost. There are different ways to limit the responses to only the required information. +When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned in the response for each hit in the `_source` object. This can lead to large amounts of data transferred through the network, increasing latency and cost. There are different ways to limit the responses to only the required information. --- @@ -19,10 +19,10 @@ When you run a basic search in OpenSearch, by default, the original JSON objects ## Disabling `_source` -You can include `"_source": false` line in the search request to prevent the `_source` field from being included in the response. See following example: +You can set `_source` to `false` in a search request to exclude the `_source` field from the response: ``` -GET "/index1/_search?pretty" +GET /index1/_search { "_source": false, "query": { @@ -31,7 +31,7 @@ GET "/index1/_search?pretty" } ``` -As no fields were selected in the previous search, the retrieved hits will only include `_index`, `_id` and `_score` of the hits. As can be seen in the following example: +Because no fields were selected in the preceding search, the retrieved hits will only include the `_index`, `_id` and `_score` of the hits: ``` { @@ -66,11 +66,11 @@ The `_source` can also be disabled in mappings for the index using the following } ``` -In such case, using methods like [searching with `docvalue_files`]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with `stored fields`]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. +If source is disabled in the index mappings, [searching with docvalue fields]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with stored fields]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. ## Specifying the fields to retrieve -You can list the fields of interest in the search request using `fields` parameter. Wildcard patterns are also accepted. See following example: +You can list the fields you want to be returned in the `fields` parameter. Wildcard patterns are also accepted: ``` GET "/index1/_search?pretty" @@ -83,7 +83,7 @@ GET "/index1/_search?pretty" } ``` -Example response: +The response contains the `name` and `age` fields: ``` { @@ -127,7 +127,7 @@ Example response: ### Extracting fields with custom format -You can also use object notation, to apply a custom format to the chosen field, see following example. +You can also use object notation to apply a custom format to the chosen field. If you have the following document: @@ -143,7 +143,7 @@ If you have the following document: } ``` -You can query with `fields` parameter and custom format: +You can query using the `fields` parameter and custom format: ``` GET /my_index/_search @@ -160,16 +160,16 @@ GET /my_index/_search } ``` -Additionally, you can also use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) with `fields` parameter, as it queries both document `_source` and `_mappings` of the index. +Additionally, you can use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) in the `fields` parameter because it queries both document `_source` and `_mappings` of the index. ## Searching with `docvalue_fields` -`docvalue_fields` is another parameter you can use in OpenSearch to retrieve specific fields from the index, but it works slightly differently compared to the `fields` parameter. The `docvalue_fields` parameter retrieves details from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values are a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. +To retrieve specific fields from the index, you can also use the `docvalue_fields` parameter. This parameter works slightly differently compared to the `fields` parameter. It retrieves details from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values have a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. -To better understand `docvalue_fields` see following example. +To better understand `docvalue_fields` see the following example. -1. Create index `my_index` with the following mappings: +1. Create an index with the following mappings: ``` PUT my_index @@ -185,7 +185,7 @@ To better understand `docvalue_fields` see following example. } ``` -2. Index the following documents using the newly created index: +2. Index the following documents into the newly created index: ``` POST my_index/_doc/1 { @@ -216,7 +216,7 @@ To better understand `docvalue_fields` see following example. } ``` -Expected response: +The response contains the `author` and `publication_date` fields: ``` { @@ -252,9 +252,9 @@ Expected response: ### Using `docvalue_fields` with nested objects -In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, see following example. +In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, as shown in the following example. -1. Define the Index and Mappings: +1. Define the index mappings: ``` PUT my_index { @@ -390,11 +390,11 @@ Expected response: ## Searching with `stored_fields` -`stored_fields` is another feature in OpenSearch that allows you to explicitly store and retrieve specific fields from documents, separate from the `_source` field. By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. +By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. You can explicitly store and retrieve specific fields from documents separately from the `_source` field using `stored_fields`. -Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. See following example. +Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. The following example shows using `stored_fields`. -1. Create index and mappings: +1. Create an index with the following mappings: ``` PUT my_index { @@ -484,7 +484,7 @@ Expected response: } ``` -Stored_fields can be disabled completely in search request using `"stored_fields": "_none_"`. +Stored_fields can be disabled completely in search request by setting `stored_fields` to `_none_`. {: .note} ### Searching `stored_fields` with nested objects @@ -614,9 +614,9 @@ Expected response: ## Using source filtering -Source filtering in OpenSearch is a way to control which parts of the `_source` field are included in the search response. This can help reduce the amount of data transferred over the network and improve performance by including only the necessary fields in the response. +Source filtering in OpenSearch is a way to control which parts of the `_source` field are included in the search response. Including only the necessary fields in the response can help reduce the amount of data transferred over the network and improve performance. -You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. See following example: +You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. The following example demonstrates including specific fields. 1. Index your data: ``` @@ -709,9 +709,9 @@ Expected response: ### Including and excluding fields in the same search -There might be cases where both `include` and `exclude` parameters are necessary. See following examples which demonstrates the usefulness of this option. +There might be cases where both `include` and `exclude` parameters are necessary. The following examples demonstrate the usefulness of this option. -If you have an index `products` with the following document: +Consider a`products` index containing the following document: ``` { "product_id": "123", @@ -749,7 +749,7 @@ If you have an index `products` with the following document: } ``` -If you want to perform a search, but in the response you only want to include the `name`, `price`, `reviews`, and `supplier` fields and exclude any `contact_email` fields from the `supplier` object, and `comment` fields from the `reviews` object. Following is the search you can run: +You want to perform a search on this index but include only the `name`, `price`, `reviews`, and `supplier` fields in the response. Additionally, you want to exclude any `contact_email` fields from the `supplier` object and `comment` fields from the `reviews` object. To achieve this, run the following search: ``` GET /products/_search @@ -805,9 +805,9 @@ Expected response: ## Using scripted fields -The `script_fields` parameter in OpenSearch allows you to include custom fields in your search results, where the values of these fields are computed using scripts. This can be useful for calculating values on the fly based on the data in the document. You can also retrieve `derived fields` using similar approach, see [Retrieving fields]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/derived/#retrieving-fields) for further details. +The `script_fields` parameter allows you to include custom fields whose values are computed using scripts in your search results. This can be useful for calculating values dynamically based on the data in the document. You can also retrieve `derived fields` using a similar approach. For more information, see [Retrieving fields]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/derived/#retrieving-fields). -Following example demonstrates the power of `script_fields`. +The following example demonstrates using `script_fields`. Let's say you have an index of products, and each product document contains the fields `price` and `discount_percentage`. You want to include a custom field in the search results that shows the discounted price of each product. @@ -825,8 +825,7 @@ Let's say you have an index of products, and each product document contains the } ``` -2. Search using `script_fields`: -You can now use the `script_fields` parameter to include a custom field called `discounted_price` in the search results. This field will be calculated based on the `price` and `discount_percentage` fields using a script. See following example: +2. Use the `script_fields` parameter to include a custom field called `discounted_price` in the search results. This field will be calculated based on the `price` and `discount_percentage` fields using a script: ``` GET /products/_search { From ee5609d71cd6ec83abd532deb5a82fbd905b5508 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Thu, 15 Aug 2024 17:04:22 +0100 Subject: [PATCH 08/10] updating as per review comments Signed-off-by: Anton Rubin --- .../retrieve-specific-fields.md | 152 +++++++++++------- 1 file changed, 92 insertions(+), 60 deletions(-) rename _search-plugins/{ => searching-data}/retrieve-specific-fields.md (92%) diff --git a/_search-plugins/retrieve-specific-fields.md b/_search-plugins/searching-data/retrieve-specific-fields.md similarity index 92% rename from _search-plugins/retrieve-specific-fields.md rename to _search-plugins/searching-data/retrieve-specific-fields.md index 46d5015110..fe861b00ce 100644 --- a/_search-plugins/retrieve-specific-fields.md +++ b/_search-plugins/searching-data/retrieve-specific-fields.md @@ -1,27 +1,20 @@ --- layout: default +parent: Searching data title: Retrieve specific fields -nav_order: 250 +nav_order: 60 --- # Retrieve specific fields When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned in the response for each hit in the `_source` object. This can lead to large amounts of data transferred through the network, increasing latency and cost. There are different ways to limit the responses to only the required information. ---- - -#### Table of contents -1. TOC -{:toc} - - ---- - -## Disabling `_source` - + +## Disabling _source + You can set `_source` to `false` in a search request to exclude the `_source` field from the response: -``` +```json GET /index1/_search { "_source": false, @@ -30,10 +23,11 @@ GET /index1/_search } } ``` +{% include copy-curl.html %} Because no fields were selected in the preceding search, the retrieved hits will only include the `_index`, `_id` and `_score` of the hits: -``` +```json { "hits" : { "total" : { @@ -58,7 +52,8 @@ Because no fields were selected in the preceding search, the retrieved hits will ``` The `_source` can also be disabled in mappings for the index using the following configuration: -``` + +```json "mappings": { "_source": { "enabled": false @@ -66,13 +61,13 @@ The `_source` can also be disabled in mappings for the index using the following } ``` -If source is disabled in the index mappings, [searching with docvalue fields]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with stored fields]({{site.url}}{{site.baseurl}}/search-plugins/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. +If source is disabled in the index mappings, [searching with docvalue fields]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with stored fields]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. ## Specifying the fields to retrieve You can list the fields you want to be returned in the `fields` parameter. Wildcard patterns are also accepted: -``` +```json GET "/index1/_search?pretty" { "_source": false, @@ -82,10 +77,11 @@ GET "/index1/_search?pretty" } } ``` +{% include copy-curl.html %} The response contains the `name` and `age` fields: -``` +```json { "hits" : { "total" : { @@ -131,7 +127,7 @@ You can also use object notation to apply a custom format to the chosen field. If you have the following document: -``` +```json { "_index": "my_index", "_type": "_doc", @@ -145,7 +141,7 @@ If you have the following document: You can query using the `fields` parameter and custom format: -``` +```json GET /my_index/_search { "query": { @@ -159,11 +155,12 @@ GET /my_index/_search "_source": false } ``` +{% include copy-curl.html %} Additionally, you can use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) in the `fields` parameter because it queries both document `_source` and `_mappings` of the index. - -## Searching with `docvalue_fields` - + +## Searching with docvalue_fields + To retrieve specific fields from the index, you can also use the `docvalue_fields` parameter. This parameter works slightly differently compared to the `fields` parameter. It retrieves details from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values have a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. To better understand `docvalue_fields` see the following example. @@ -171,7 +168,7 @@ To better understand `docvalue_fields` see the following example. 1. Create an index with the following mappings: - ``` + ```json PUT my_index { "mappings": { @@ -184,9 +181,11 @@ To better understand `docvalue_fields` see the following example. } } ``` + {% include copy-curl.html %} 2. Index the following documents into the newly created index: - ``` + + ```json POST my_index/_doc/1 { "title": "OpenSearch Basics", @@ -203,9 +202,11 @@ To better understand `docvalue_fields` see the following example. "price": 39.99 } ``` + {% include copy-curl.html %} + 3. Retrieve only the `author` and `publication_date` fields using `docvalue_fields`: - ``` + ```json POST my_index/_search { "_source": false, @@ -215,10 +216,11 @@ To better understand `docvalue_fields` see the following example. } } ``` + {% include copy-curl.html %} The response contains the `author` and `publication_date` fields: -``` +```json { "hits": { "total": { @@ -249,13 +251,14 @@ The response contains the `author` and `publication_date` fields: } } ``` - -### Using `docvalue_fields` with nested objects - + +### Using docvalue_fields with nested objects + In OpenSearch, if you want to retrieve doc values for nested objects, you cannot directly use the `docvalue_fields` parameter because it will return an empty array. Instead, you should use the `inner_hits` parameter with its own `docvalue_fields` property, as shown in the following example. 1. Define the index mappings: - ``` + + ```json PUT my_index { "mappings": { @@ -274,9 +277,11 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot } } ``` + {% include copy-curl.html %} 2. Index your data: - ``` + + ```json POST my_index/_doc/1 { "title": "OpenSearch Basics", @@ -295,9 +300,11 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot ] } ``` + {% include copy-curl.html %} 3. Perform a search with `inner_hits` and `docvalue_fields`: - ``` + + ```json POST my_index/_search { "query": { @@ -313,10 +320,11 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot } } ``` + {% include copy-curl.html %} Expected response: -``` +```json { "hits": { "total": { @@ -387,15 +395,16 @@ Expected response: } } ``` - -## Searching with `stored_fields` - + +## Searching with stored_fields + By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. You can explicitly store and retrieve specific fields from documents separately from the `_source` field using `stored_fields`. Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. The following example shows using `stored_fields`. 1. Create an index with the following mappings: - ``` + + ```json PUT my_index { "mappings": { @@ -418,9 +427,11 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for } } ``` + {% include copy-curl.html %} 2. Index your data: - ``` + + ```json POST my_index/_doc/1 { "title": "OpenSearch Basics", @@ -437,9 +448,11 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for "price": 39.99 } ``` + {% include copy-curl.html %} 3. Perform a search with `stored_fields`: - ``` + + ```json POST my_index/_search { "_source": false, @@ -449,10 +462,11 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for } } ``` + {% include copy-curl.html %} Expected response: -``` +```json { "hits": { "total": { @@ -486,13 +500,14 @@ Expected response: Stored_fields can be disabled completely in search request by setting `stored_fields` to `_none_`. {: .note} - -### Searching `stored_fields` with nested objects - + +### Searching stored_fields with nested objects + In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, see following example. 1. Create index and mappings: - ``` + + ```json PUT my_index { "mappings": { @@ -511,9 +526,11 @@ In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you c } } ``` + {% include copy-curl.html %} 2. Index your data: - ``` + + ```json POST my_index/_doc/1 { "title": "OpenSearch Basics", @@ -532,9 +549,11 @@ In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you c ] } ``` + {% include copy-curl.html %} 3. Perform a search with `inner_hits` and `stored_fields`: - ``` + + ```json POST my_index/_search { "_source": false, @@ -551,10 +570,11 @@ In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you c } } ``` + {% include copy-curl.html %} Expected response: -``` +```json { "hits": { "total": { @@ -619,7 +639,8 @@ Source filtering in OpenSearch is a way to control which parts of the `_source` You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. The following example demonstrates including specific fields. 1. Index your data: - ``` + + ```json PUT my_index/_doc/1 { "title": "OpenSearch Basics", @@ -628,9 +649,11 @@ You can include or exclude specific fields from the `_source` field in the searc "price": 29.99 } ``` + {% include copy-curl.html %} 2. Perform a search using source filtering: - ``` + + ```json POST my_index/_search { "_source": ["title", "author"], @@ -639,10 +662,11 @@ You can include or exclude specific fields from the `_source` field in the searc } } ``` + {% include copy-curl.html %} Expected response: -``` +```json { "hits": { "total": { @@ -669,7 +693,7 @@ Expected response: You can choose to exclude fields using `"excludes"` parameter in search request, see following example: -``` +```json POST my_index/_search { "_source": { @@ -680,10 +704,11 @@ POST my_index/_search } } ``` +{% include copy-curl.html %} Expected response: -``` +```json { "hits": { "total": { @@ -712,7 +737,8 @@ Expected response: There might be cases where both `include` and `exclude` parameters are necessary. The following examples demonstrate the usefulness of this option. Consider a`products` index containing the following document: -``` + +```json { "product_id": "123", "name": "Smartphone", @@ -751,7 +777,7 @@ Consider a`products` index containing the following document: You want to perform a search on this index but include only the `name`, `price`, `reviews`, and `supplier` fields in the response. Additionally, you want to exclude any `contact_email` fields from the `supplier` object and `comment` fields from the `reviews` object. To achieve this, run the following search: -``` +```json GET /products/_search { "_source": { @@ -765,10 +791,11 @@ GET /products/_search } } ``` +{% include copy-curl.html %} Expected response: -``` +```json { "hits": { "hits": [ @@ -813,7 +840,8 @@ Let's say you have an index of products, and each product document contains the 1. Index the data: - ``` + + ```json PUT /products/_doc/123 { "product_id": "123", @@ -824,9 +852,11 @@ Let's say you have an index of products, and each product document contains the "description": "A powerful smartphone with a sleek design." } ``` + {% include copy-curl.html %} 2. Use the `script_fields` parameter to include a custom field called `discounted_price` in the search results. This field will be calculated based on the `price` and `discount_percentage` fields using a script: -``` + +```json GET /products/_search { "_source": ["product_id", "name", "price", "discount_percentage"], @@ -845,9 +875,11 @@ GET /products/_search } } ``` +{% include copy-curl.html %} -Example response: -``` +You should get back the following response: + +```json { "hits": { "total": { From 3ba945cf7e7ed7a7f361214ba825b5d274b80dba Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Mon, 19 Aug 2024 17:43:02 +0100 Subject: [PATCH 09/10] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: AntonEliatra --- .../retrieve-specific-fields.md | 52 +++++++++---------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/_search-plugins/searching-data/retrieve-specific-fields.md b/_search-plugins/searching-data/retrieve-specific-fields.md index fe861b00ce..a5e9eb61a0 100644 --- a/_search-plugins/searching-data/retrieve-specific-fields.md +++ b/_search-plugins/searching-data/retrieve-specific-fields.md @@ -7,7 +7,7 @@ nav_order: 60 # Retrieve specific fields -When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned in the response for each hit in the `_source` object. This can lead to large amounts of data transferred through the network, increasing latency and cost. There are different ways to limit the responses to only the required information. +When you run a basic search in OpenSearch, by default, the original JSON objects that were used during indexing are also returned in the response for each hit in the `_source` object. This can lead to large amounts of data being transferred through the network, increasing latency and costs. There are several ways to limit the responses to only the required information. ## Disabling _source @@ -51,7 +51,7 @@ Because no fields were selected in the preceding search, the retrieved hits will } ``` -The `_source` can also be disabled in mappings for the index using the following configuration: +The `_source` can also be disabled in index mappings by using the following configuration: ```json "mappings": { @@ -65,7 +65,7 @@ If source is disabled in the index mappings, [searching with docvalue fields]({{ ## Specifying the fields to retrieve -You can list the fields you want to be returned in the `fields` parameter. Wildcard patterns are also accepted: +You can list the fields you want to retrieve in the `fields` parameter. Wildcard patterns are also accepted: ```json GET "/index1/_search?pretty" @@ -121,7 +121,7 @@ The response contains the `name` and `age` fields: } ``` -### Extracting fields with custom format +### Extracting fields with a custom format You can also use object notation to apply a custom format to the chosen field. @@ -139,7 +139,7 @@ If you have the following document: } ``` -You can query using the `fields` parameter and custom format: +Then you can query using the `fields` parameter and a custom format: ```json GET /my_index/_search @@ -157,13 +157,13 @@ GET /my_index/_search ``` {% include copy-curl.html %} -Additionally, you can use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) in the `fields` parameter because it queries both document `_source` and `_mappings` of the index. +Additionally, you can use [most fields]({{site.url}}{{site.baseurl}}/query-dsl/full-text/multi-match/#most-fields) and [field aliases]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/alias/) in the `fields` parameter because it queries both the document `_source` and `_mappings` of the index. ## Searching with docvalue_fields -To retrieve specific fields from the index, you can also use the `docvalue_fields` parameter. This parameter works slightly differently compared to the `fields` parameter. It retrieves details from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values have a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. +To retrieve specific fields from the index, you can also use the `docvalue_fields` parameter. This parameter works slightly differently as compared to the `fields` parameter. It retrieves information from doc values rather than from the `_source` field, which is more efficient for fields that are not analyzed, like keyword, date, and numeric fields. Doc values have a columnar storage format optimized for efficient sorting and aggregations. It stores the values on disk in a way that is easy to read. When you use `docvalue_fields`, OpenSearch reads the values directly from this optimized storage format. It is useful for retrieving values of fields that are primarily used for sorting, aggregations, and for use in scripts. -To better understand `docvalue_fields` see the following example. +The following example demonstrates how to use the `docvalue_fields` parameter. 1. Create an index with the following mappings: @@ -322,7 +322,7 @@ In OpenSearch, if you want to retrieve doc values for nested objects, you cannot ``` {% include copy-curl.html %} -Expected response: +The following is the expected response: ```json { @@ -398,9 +398,9 @@ Expected response: ## Searching with stored_fields -By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, sometimes you might want to store certain fields separately for more efficient retrieval. You can explicitly store and retrieve specific fields from documents separately from the `_source` field using `stored_fields`. +By default, OpenSearch stores the entire document in the `_source` field and uses it to return document contents in search results. However, you might also want to store certain fields separately for more efficient retrieval. You can explicitly store and retrieve specific document fields separately from the `_source` field by using `stored_fields`. -Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. The following example shows using `stored_fields`. +Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for fields you want to store separately. It can be useful if you frequently need to retrieve only a small subset of fields and want to avoid retrieving the entire `_source` field. The following example demonstrates how to use the `stored_fields` parameter. 1. Create an index with the following mappings: @@ -464,7 +464,7 @@ Unlike `_source`, `stored_fields` must be explicitly defined in the mappings for ``` {% include copy-curl.html %} -Expected response: +The following is the expected response: ```json { @@ -498,12 +498,12 @@ Expected response: } ``` -Stored_fields can be disabled completely in search request by setting `stored_fields` to `_none_`. +The `stored_fields` parameter can be disabled completely by setting `stored_fields` to `_none_`. {: .note} ### Searching stored_fields with nested objects -In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, see following example. +In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, as shown in the following example. 1. Create index and mappings: @@ -572,7 +572,7 @@ In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you c ``` {% include copy-curl.html %} -Expected response: +The following is the expected response: ```json { @@ -634,9 +634,9 @@ Expected response: ## Using source filtering -Source filtering in OpenSearch is a way to control which parts of the `_source` field are included in the search response. Including only the necessary fields in the response can help reduce the amount of data transferred over the network and improve performance. +Source filtering is a way to control which parts of the `_source` field are included in the search response. Including only the necessary fields in the response can help reduce the amount of data transferred over the network and improve performance. -You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. The following example demonstrates including specific fields. +You can include or exclude specific fields from the `_source` field in the search response using complete field names or simple wildcard patterns. The following example demonstrates how to include specific fields. 1. Index your data: @@ -664,7 +664,7 @@ You can include or exclude specific fields from the `_source` field in the searc ``` {% include copy-curl.html %} -Expected response: +The following is the expected response: ```json { @@ -691,7 +691,7 @@ Expected response: ### Excluding fields with source filtering -You can choose to exclude fields using `"excludes"` parameter in search request, see following example: +You can choose to exclude fields by using the `"excludes"` parameter in a search request, as shown in the following example: ```json POST my_index/_search @@ -706,7 +706,7 @@ POST my_index/_search ``` {% include copy-curl.html %} -Expected response: +The following is the expected response: ```json { @@ -734,9 +734,9 @@ Expected response: ### Including and excluding fields in the same search -There might be cases where both `include` and `exclude` parameters are necessary. The following examples demonstrate the usefulness of this option. +In some cases, both the `include` and `exclude` parameters may be necessary. The following examples demonstrate how to include and exclude fields in the same search. -Consider a`products` index containing the following document: +Consider a `products` index containing the following document: ```json { @@ -793,7 +793,7 @@ GET /products/_search ``` {% include copy-curl.html %} -Expected response: +The following is the expected response: ```json { @@ -832,9 +832,9 @@ Expected response: ## Using scripted fields -The `script_fields` parameter allows you to include custom fields whose values are computed using scripts in your search results. This can be useful for calculating values dynamically based on the data in the document. You can also retrieve `derived fields` using a similar approach. For more information, see [Retrieving fields]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/derived/#retrieving-fields). +The `script_fields` parameter allows you to include custom fields whose values are computed using scripts in your search results. This can be useful for calculating values dynamically based on the document data. You can also retrieve `derived fields` by using a similar approach. For more information, see [Retrieving fields]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/derived/#retrieving-fields). -The following example demonstrates using `script_fields`. +The following example demonstrates how to use the `script_fields` parameter. Let's say you have an index of products, and each product document contains the fields `price` and `discount_percentage`. You want to include a custom field in the search results that shows the discounted price of each product. @@ -877,7 +877,7 @@ GET /products/_search ``` {% include copy-curl.html %} -You should get back the following response: +You should receive the following response: ```json { From af3f162fac9dd4aa8ba381085bb997db56bf190d Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Mon, 19 Aug 2024 17:57:36 +0100 Subject: [PATCH 10/10] updating as per PR comments Signed-off-by: Anton Rubin --- _search-plugins/searching-data/index.md | 1 + .../searching-data/retrieve-specific-fields.md | 10 ++++------ 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/_search-plugins/searching-data/index.md b/_search-plugins/searching-data/index.md index 6781b5a56b..dab57d6460 100644 --- a/_search-plugins/searching-data/index.md +++ b/_search-plugins/searching-data/index.md @@ -18,3 +18,4 @@ Feature | Description [Paginate results]({{site.url}}{{site.baseurl}}/opensearch/search/paginate/) | Rather than a single, long list, separate search results into pages. [Sort results]({{site.url}}{{site.baseurl}}/opensearch/search/sort/) | Allow sorting of results by different criteria. [Highlight query matches]({{site.url}}{{site.baseurl}}/opensearch/search/highlight/) | Highlight the search term in the results. +[Retrieve specific fields]({{site.url}}{{site.baseurl}}search-plugins/searching-data/retrieve-specific-fields/) | Retrieve only the specific fields diff --git a/_search-plugins/searching-data/retrieve-specific-fields.md b/_search-plugins/searching-data/retrieve-specific-fields.md index a5e9eb61a0..ce860470dc 100644 --- a/_search-plugins/searching-data/retrieve-specific-fields.md +++ b/_search-plugins/searching-data/retrieve-specific-fields.md @@ -61,7 +61,7 @@ The `_source` can also be disabled in index mappings by using the following conf } ``` -If source is disabled in the index mappings, [searching with docvalue fields]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with stored fields]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. +If `_source` is disabled in the index mappings, [searching with docvalue fields]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/retrieve-specific-fields/#searching-with-docvalue_fields) and [searching with stored fields]({{site.url}}{{site.baseurl}}/search-plugins/searching-data/retrieve-specific-fields/#searching-with-stored_fields) become extremely useful. ## Specifying the fields to retrieve @@ -505,7 +505,7 @@ The `stored_fields` parameter can be disabled completely by setting `stored_fiel In OpenSearch, if you want to retrieve `stored_fields` for nested objects, you cannot directly use the `stored_fields` parameter because no data will be returned. Instead, you should use the `inner_hits` parameter with its own `stored_fields` property, as shown in the following example. -1. Create index and mappings: +1. Create an index with the following mappings: ```json PUT my_index @@ -775,7 +775,7 @@ Consider a `products` index containing the following document: } ``` -You want to perform a search on this index but include only the `name`, `price`, `reviews`, and `supplier` fields in the response. Additionally, you want to exclude any `contact_email` fields from the `supplier` object and `comment` fields from the `reviews` object. To achieve this, run the following search: +To perform a search on this index while including only the `name`, `price`, `reviews`, and `supplier` fields in the response, and excluding the `contact_email` field from the `supplier` object and the `comment` field from the `reviews` object, execute the following search: ```json GET /products/_search @@ -834,9 +834,7 @@ The following is the expected response: The `script_fields` parameter allows you to include custom fields whose values are computed using scripts in your search results. This can be useful for calculating values dynamically based on the document data. You can also retrieve `derived fields` by using a similar approach. For more information, see [Retrieving fields]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/derived/#retrieving-fields). -The following example demonstrates how to use the `script_fields` parameter. - -Let's say you have an index of products, and each product document contains the fields `price` and `discount_percentage`. You want to include a custom field in the search results that shows the discounted price of each product. +If you have an index of products, where each product document contains the `price` and `discount_percentage` fields. You can use `script_fields` parameter to include a custom field in the search results that displays the discounted price of each product. The following example demonstrates how to use the `script_fields` parameter: 1. Index the data: