From 32b1d17855e7401dadf723292ba95ef56c78e054 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Mon, 19 Jun 2023 14:24:06 -0400 Subject: [PATCH] Add date nanoseconds field type (#4348) * Add date nanoseconds field type Signed-off-by: Fanit Kolchina * Fix links Signed-off-by: Fanit Kolchina * Apply suggestions from code review Co-authored-by: Melissa Vagi Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Fanit Kolchina Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi --- .../supported-field-types/date-nanos.md | 290 ++++++++++++++++++ _field-types/supported-field-types/date.md | 3 +- _field-types/supported-field-types/dates.md | 17 + .../supported-field-types/geographic.md | 2 +- _field-types/supported-field-types/index.md | 2 +- .../supported-field-types/object-fields.md | 2 +- _field-types/supported-field-types/rank.md | 2 +- _field-types/supported-field-types/string.md | 2 +- 8 files changed, 314 insertions(+), 6 deletions(-) create mode 100644 _field-types/supported-field-types/date-nanos.md create mode 100644 _field-types/supported-field-types/dates.md diff --git a/_field-types/supported-field-types/date-nanos.md b/_field-types/supported-field-types/date-nanos.md new file mode 100644 index 0000000000..12399a69d4 --- /dev/null +++ b/_field-types/supported-field-types/date-nanos.md @@ -0,0 +1,290 @@ +--- +layout: default +title: Date nanoseconds +nav_order: 35 +has_children: false +parent: Date field types +grand_parent: Supported field types +--- + +# Date nanoseconds field type + +The `date_nanos` field type is similar to the [`date`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date/) field type in that it holds a date. However, `date` stores the date in millisecond resolution, while `date_nanos` stores the date in nanosecond resolution. Dates are stored as `long` values that correspond to nanoseconds since the epoch. Therefore, the range of supported dates is approximately 1970--2262. + +Queries on `date_nanos` fields are converted to range queries on the field value's `long` representation. Then the stored fields and aggregation results are converted to a string using the format set on the field. + +The `date_nanos` field supports all [formats]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date#formats) and [parameters]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date#parameters) that `date` supports. You can use multiple formats separated by `||`. +{: .note} + +For `date_nanos` fields, you can use the `strict_date_optional_time_nanos` format to preserve nanosecond resolution. If you don't specify the format when mapping a field as `date_nanos`, the default format is `strict_date_optional_time||epoch_millis` that lets you pass values in either `strict_date_optional_time` or `epoch_millis` format. The `strict_date_optional_time` format supports dates in nanosecond resolution, but the `epoch_millis` format supports dates in millisecond resolution only. + +## Example + +Create a mapping with the `date` field of type `date_nanos` that has the `strict_date_optional_time_nanos` format: + +```json +PUT testindex/_mapping +{ + "properties": { + "date": { + "type": "date_nanos", + "format" : "strict_date_optional_time_nanos" + } + } +} +``` +{% include copy-curl.html %} + +Index two documents into the index: + +```json +PUT testindex/_doc/1 +{ "date": "2022-06-15T10:12:52.382719622Z" } +``` +{% include copy-curl.html %} + +```json +PUT testindex/_doc/2 +{ "date": "2022-06-15T10:12:52.382719624Z" } +``` +{% include copy-curl.html %} + +You can use a range query to search for a date range: + +```json +GET testindex/_search +{ + "query": { + "range": { + "date": { + "gte": "2022-06-15T10:12:52.382719621Z", + "lte": "2022-06-15T10:12:52.382719623Z" + } + } + } +} +``` +{% include copy-curl.html %} + +The response contains the document whose date is in the specified range: + +```json +{ + "took": 43, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "testindex", + "_id": "1", + "_score": 1, + "_source": { + "date": "2022-06-15T10:12:52.382719622Z" + } + } + ] + } +} +``` + +When querying documents with `date_nanos` fields, you can use `fields` or `docvalue_fields`: + +```json +GET testindex/_search +{ + "fields": ["date"] +} +``` +{% include copy-curl.html %} + +```json +GET testindex/_search +{ + "docvalue_fields" : [ + { + "field" : "date" + } + ] +} +``` +{% include copy-curl.html %} + +The response to either of the preceding queries contains both indexed documents: + +```json +{ + "took": 4, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "testindex", + "_id": "1", + "_score": 1, + "_source": { + "date": "2022-06-15T10:12:52.382719622Z" + }, + "fields": { + "date": [ + "2022-06-15T10:12:52.382719622Z" + ] + } + }, + { + "_index": "testindex", + "_id": "2", + "_score": 1, + "_source": { + "date": "2022-06-15T10:12:52.382719624Z" + }, + "fields": { + "date": [ + "2022-06-15T10:12:52.382719624Z" + ] + } + } + ] + } +} +``` + +You can sort on a `date_nanos` field as follows: + +```json +GET testindex/_search +{ + "sort": { + "date": "asc" + } +} +``` +{% include copy-curl.html %} + +The response contains the sorted documents: + +```json +{ + "took": 5, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": null, + "hits": [ + { + "_index": "testindex", + "_id": "1", + "_score": null, + "_source": { + "date": "2022-06-15T10:12:52.382719622Z" + }, + "sort": [ + 1655287972382719700 + ] + }, + { + "_index": "testindex", + "_id": "2", + "_score": null, + "_source": { + "date": "2022-06-15T10:12:52.382719624Z" + }, + "sort": [ + 1655287972382719700 + ] + } + ] + } +} +``` + +You can also use a Painless script to access the nanoseconds part of the field: + +```json +GET testindex/_search +{ + "script_fields" : { + "my_field" : { + "script" : { + "lang" : "painless", + "source" : "doc['date'].value.nano" + } + } + } +} +``` +{% include copy-curl.html %} + +The response contains only the nanosecond parts of the fields: + +```json +{ + "took": 4, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "testindex", + "_id": "1", + "_score": 1, + "fields": { + "my_field": [ + 382719622 + ] + } + }, + { + "_index": "testindex", + "_id": "2", + "_score": 1, + "fields": { + "my_field": [ + 382719624 + ] + } + } + ] + } +} +``` \ No newline at end of file diff --git a/_field-types/supported-field-types/date.md b/_field-types/supported-field-types/date.md index ea09311718..da551a1dd1 100644 --- a/_field-types/supported-field-types/date.md +++ b/_field-types/supported-field-types/date.md @@ -3,7 +3,8 @@ layout: default title: Date nav_order: 25 has_children: false -parent: Supported field types +parent: Date field types +grand_parent: Supported field types redirect_from: - /opensearch/supported-field-types/date/ - /field-types/date/ diff --git a/_field-types/supported-field-types/dates.md b/_field-types/supported-field-types/dates.md new file mode 100644 index 0000000000..7c6e47cb60 --- /dev/null +++ b/_field-types/supported-field-types/dates.md @@ -0,0 +1,17 @@ +--- +layout: default +title: Date field types +nav_order: 25 +has_children: true +has_toc: false +parent: Supported field types +--- + +# Date field types + +Date field types contain a date value that can be formatted using different date formats. The following table lists all date field types that OpenSearch supports. + +Field data type | Description +:--- | :--- +[`date`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date/) | A date stored in millisecond resolution. +[`date_nanos`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/date-nanos/) | A date stored in nanosecond resolution. diff --git a/_field-types/supported-field-types/geographic.md b/_field-types/supported-field-types/geographic.md index 07d0382082..cbe3982a4d 100644 --- a/_field-types/supported-field-types/geographic.md +++ b/_field-types/supported-field-types/geographic.md @@ -12,7 +12,7 @@ redirect_from: # Geographic field types -The following table lists all geographic field types that OpenSearch supports. +Geographic fields contain values that represent points or shapes on a map. The following table lists all geographic field types that OpenSearch supports. Field data type | Description :--- | :--- diff --git a/_field-types/supported-field-types/index.md b/_field-types/supported-field-types/index.md index 38b45860ba..3cb8bff8cd 100644 --- a/_field-types/supported-field-types/index.md +++ b/_field-types/supported-field-types/index.md @@ -19,7 +19,7 @@ Field data type | Description [`binary`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary/) | A binary value in Base64 encoding. [Numeric]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/numeric/) | `byte`, `double`, `float`, `half_float`, `integer`, `long`, `unsigned_long`, `scaled_float`, `short`. [`boolean`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/boolean/) | A Boolean value. -[`date`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date/) | A date value as a formatted string, a long value, or an integer. +[Date]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/dates/) | `date`, `date_nanos`. [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) | An IP address in IPv4 or IPv6 format. [Range]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/range/) | `integer_range`, `long_range`,`double_range`, `float_range`, `date_range`,`ip_range`. [Object]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object/) | `object`, `nested`, `join`. diff --git a/_field-types/supported-field-types/object-fields.md b/_field-types/supported-field-types/object-fields.md index 64869fc34d..429c5b94c7 100644 --- a/_field-types/supported-field-types/object-fields.md +++ b/_field-types/supported-field-types/object-fields.md @@ -12,7 +12,7 @@ redirect_from: # Object field types -The following table lists all object field types that OpenSearch supports. +Object field types contain values that are objects or relations. The following table lists all object field types that OpenSearch supports. Field data type | Description :--- | :--- diff --git a/_field-types/supported-field-types/rank.md b/_field-types/supported-field-types/rank.md index c46467f8a5..a4ec0fac4c 100644 --- a/_field-types/supported-field-types/rank.md +++ b/_field-types/supported-field-types/rank.md @@ -23,7 +23,7 @@ Rank feature and rank features fields can be queried with [rank feature queries] ## Rank feature -A rank feature field type uses a positive [float]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/numeric/) value to boost or decrease the relevance score of a document in a `rank_feature` query. By default, this value boosts the relevance score. To decrease the relevance score, set the optional `positive_score_impact` parameter to false. +A rank feature field type uses a positive [float]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/numeric/) value to boost or decrease the relevance score of a document in a `rank_feature` query. By default, this value boosts the relevance score. To decrease the relevance score, set the optional `positive_score_impact` parameter to false. ### Example diff --git a/_field-types/supported-field-types/string.md b/_field-types/supported-field-types/string.md index 21cee52dad..f24dea2325 100644 --- a/_field-types/supported-field-types/string.md +++ b/_field-types/supported-field-types/string.md @@ -12,7 +12,7 @@ redirect_from: # String field types -The following table lists all string field types that OpenSearch supports. +String field types contain text values or values derived from text. The following table lists all string field types that OpenSearch supports. Field data type | Description :--- | :---