Add date nanoseconds field type (opensearch-project#4348)

* Add date nanoseconds field type Signed-off-by: Fanit Kolchina <[email protected]> * Fix links Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]>
harshavamsi · Oct 31, 2023 · 32b1d17 · 32b1d17
1 parent 96823d2
commit 32b1d17
Show file tree

Hide file tree

Showing 8 changed files with 314 additions and 6 deletions.
diff --git a/_field-types/supported-field-types/date-nanos.md b/_field-types/supported-field-types/date-nanos.md
@@ -0,0 +1,290 @@
+---
+layout: default
+title: Date nanoseconds
+nav_order: 35
+has_children: false
+parent: Date field types
+grand_parent: Supported field types
+---
+
+# Date nanoseconds field type
+
+The `date_nanos` field type is similar to the [`date`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date/) field type in that it holds a date. However, `date` stores the date in millisecond resolution, while `date_nanos` stores the date in nanosecond resolution. Dates are stored as `long` values that correspond to nanoseconds since the epoch. Therefore, the range of supported dates is approximately 1970--2262.
+
+Queries on `date_nanos` fields are converted to range queries on the field value's `long` representation. Then the stored fields and aggregation results are converted to a string using the format set on the field. 
+
+The `date_nanos` field supports all [formats]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date#formats) and [parameters]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date#parameters) that `date` supports. You can use multiple formats separated by `||`.
+{: .note}
+
+For `date_nanos` fields, you can use the `strict_date_optional_time_nanos` format to preserve nanosecond resolution. If you don't specify the format when mapping a field as `date_nanos`, the default format is `strict_date_optional_time||epoch_millis` that lets you pass values in either `strict_date_optional_time` or `epoch_millis` format. The `strict_date_optional_time` format supports dates in nanosecond resolution, but the `epoch_millis` format supports dates in millisecond resolution only.
+
+## Example
+
+Create a mapping with the `date` field of type `date_nanos` that has the `strict_date_optional_time_nanos` format:
+
+```json
+PUT testindex/_mapping
+{
+  "properties": {
+      "date": {
+        "type": "date_nanos",
+        "format" : "strict_date_optional_time_nanos"
+      }
+    }
+}
+```
+{% include copy-curl.html %}
+
+Index two documents into the index:
+
+```json
+PUT testindex/_doc/1
+{ "date": "2022-06-15T10:12:52.382719622Z" }
+```
+{% include copy-curl.html %}
+
+```json
+PUT testindex/_doc/2
+{ "date": "2022-06-15T10:12:52.382719624Z" }
+```
+{% include copy-curl.html %}
+
+You can use a range query to search for a date range:
+
+```json
+GET testindex/_search
+{
+  "query": {
+    "range": {
+      "date": {
+        "gte": "2022-06-15T10:12:52.382719621Z",
+        "lte": "2022-06-15T10:12:52.382719623Z"
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The response contains the document whose date is in the specified range:
+
+```json
+{
+  "took": 43,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 1,
+    "hits": [
+      {
+        "_index": "testindex",
+        "_id": "1",
+        "_score": 1,
+        "_source": {
+          "date": "2022-06-15T10:12:52.382719622Z"
+        }
+      }
+    ]
+  }
+}
+```
+
+When querying documents with `date_nanos` fields, you can use `fields` or `docvalue_fields`:
+
+```json
+GET testindex/_search
+{
+  "fields": ["date"]
+}
+```
+{% include copy-curl.html %}
+
+```json
+GET testindex/_search
+{
+  "docvalue_fields" : [
+    {
+      "field" : "date"
+    }
+  ]
+}
+```
+{% include copy-curl.html %}
+
+The response to either of the preceding queries contains both indexed documents:
+
+```json
+{
+  "took": 4,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 2,
+      "relation": "eq"
+    },
+    "max_score": 1,
+    "hits": [
+      {
+        "_index": "testindex",
+        "_id": "1",
+        "_score": 1,
+        "_source": {
+          "date": "2022-06-15T10:12:52.382719622Z"
+        },
+        "fields": {
+          "date": [
+            "2022-06-15T10:12:52.382719622Z"
+          ]
+        }
+      },
+      {
+        "_index": "testindex",
+        "_id": "2",
+        "_score": 1,
+        "_source": {
+          "date": "2022-06-15T10:12:52.382719624Z"
+        },
+        "fields": {
+          "date": [
+            "2022-06-15T10:12:52.382719624Z"
+          ]
+        }
+      }
+    ]
+  }
+}
+```
+
+You can sort on a `date_nanos` field as follows:
+
+```json
+GET testindex/_search
+{
+  "sort": { 
+    "date": "asc"
+  } 
+}
+```
+{% include copy-curl.html %}
+
+The response contains the sorted documents:
+
+```json
+{
+  "took": 5,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 2,
+      "relation": "eq"
+    },
+    "max_score": null,
+    "hits": [
+      {
+        "_index": "testindex",
+        "_id": "1",
+        "_score": null,
+        "_source": {
+          "date": "2022-06-15T10:12:52.382719622Z"
+        },
+        "sort": [
+          1655287972382719700
+        ]
+      },
+      {
+        "_index": "testindex",
+        "_id": "2",
+        "_score": null,
+        "_source": {
+          "date": "2022-06-15T10:12:52.382719624Z"
+        },
+        "sort": [
+          1655287972382719700
+        ]
+      }
+    ]
+  }
+}
+```
+
+You can also use a Painless script to access the nanoseconds part of the field:
+
+```json
+GET testindex/_search
+{
+  "script_fields" : {
+    "my_field" : {
+      "script" : {
+        "lang" : "painless",
+        "source" : "doc['date'].value.nano" 
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The response contains only the nanosecond parts of the fields:
+
+```json
+{
+  "took": 4,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 2,
+      "relation": "eq"
+    },
+    "max_score": 1,
+    "hits": [
+      {
+        "_index": "testindex",
+        "_id": "1",
+        "_score": 1,
+        "fields": {
+          "my_field": [
+            382719622
+          ]
+        }
+      },
+      {
+        "_index": "testindex",
+        "_id": "2",
+        "_score": 1,
+        "fields": {
+          "my_field": [
+            382719624
+          ]
+        }
+      }
+    ]
+  }
+}
+```
diff --git a/_field-types/supported-field-types/date.md b/_field-types/supported-field-types/date.md
@@ -3,7 +3,8 @@ layout: default
 title: Date
 nav_order: 25
 has_children: false
-parent: Supported field types
+parent: Date field types
+grand_parent: Supported field types
 redirect_from:
   - /opensearch/supported-field-types/date/
   - /field-types/date/

diff --git a/_field-types/supported-field-types/dates.md b/_field-types/supported-field-types/dates.md
@@ -0,0 +1,17 @@
+---
+layout: default
+title: Date field types
+nav_order: 25
+has_children: true
+has_toc: false
+parent: Supported field types
+---
+
+# Date field types
+
+Date field types contain a date value that can be formatted using different date formats. The following table lists all date field types that OpenSearch supports. 
+
+Field data type | Description
+:--- | :---  
+[`date`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date/) | A date stored in millisecond resolution.
+[`date_nanos`]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/date-nanos/) | A date stored in nanosecond resolution.
diff --git a/_field-types/supported-field-types/geographic.md b/_field-types/supported-field-types/geographic.md
@@ -12,7 +12,7 @@ redirect_from:
 
 # Geographic field types
 
-The following table lists all geographic field types that OpenSearch supports.
+Geographic fields contain values that represent points or shapes on a map. The following table lists all geographic field types that OpenSearch supports.
 
 Field data type | Description
 :--- | :---  

diff --git a/_field-types/supported-field-types/index.md b/_field-types/supported-field-types/index.md
@@ -19,7 +19,7 @@ Field data type | Description
 [`binary`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/binary/) |  A binary value in Base64 encoding. 
 [Numeric]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/numeric/) | `byte`, `double`, `float`, `half_float`, `integer`, `long`, `unsigned_long`, `scaled_float`, `short`.
 [`boolean`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/boolean/) | A Boolean value. 
-[`date`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/date/) | A date value as a formatted string, a long value, or an integer. 
+[Date]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/dates/) | `date`, `date_nanos`.
 [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) | An IP address in IPv4 or IPv6 format. 
 [Range]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/range/) | `integer_range`, `long_range`,`double_range`, `float_range`, `date_range`,`ip_range`. 
 [Object]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object/) | `object`, `nested`, `join`.

diff --git a/_field-types/supported-field-types/object-fields.md b/_field-types/supported-field-types/object-fields.md
@@ -12,7 +12,7 @@ redirect_from:
 
 # Object field types
 
-The following table lists all object field types that OpenSearch supports.
+Object field types contain values that are objects or relations. The following table lists all object field types that OpenSearch supports.
 
 Field data type | Description
 :--- | :---  

diff --git a/_field-types/supported-field-types/rank.md b/_field-types/supported-field-types/rank.md
@@ -23,7 +23,7 @@ Rank feature and rank features fields can be queried with [rank feature queries]
 
 ## Rank feature
 
-A rank feature field type uses a positive [float]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/numeric/)  value to boost or decrease the relevance score of a document in a `rank_feature` query. By default, this value boosts the relevance score. To decrease the relevance score, set the optional `positive_score_impact` parameter to false.
+A rank feature field type uses a positive [float]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/numeric/) value to boost or decrease the relevance score of a document in a `rank_feature` query. By default, this value boosts the relevance score. To decrease the relevance score, set the optional `positive_score_impact` parameter to false.
 
 ### Example
 

diff --git a/_field-types/supported-field-types/string.md b/_field-types/supported-field-types/string.md
@@ -12,7 +12,7 @@ redirect_from:
 
 # String field types
 
-The following table lists all string field types that OpenSearch supports.
+String field types contain text values or values derived from text. The following table lists all string field types that OpenSearch supports.
 
 Field data type | Description
 :--- | :---