Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metadata fields for mappings (content gap initiative) #6933

Merged
merged 176 commits into from
Aug 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
176 commits
Select commit Hold shift + click to select a range
a7d4567
Add mapping as part of content gap initiative
vagimeli Apr 9, 2024
7b9aa58
Update mapping as part of content gap initiative
vagimeli Apr 9, 2024
3ad5c25
Add content to address feedback
vagimeli Apr 10, 2024
653d2f4
Merge branch 'main' into mapping-content-gap
vagimeli Apr 10, 2024
48d3a6e
Add delete mapping section
vagimeli Apr 15, 2024
8722b15
Update index.md
vagimeli Apr 15, 2024
b2dfc7a
Add metadata fields index page
vagimeli Apr 15, 2024
cf4a80c
Add setting descriptions
vagimeli Apr 17, 2024
061cc80
Delete remove mappings type section
vagimeli Apr 17, 2024
a047b4c
Add individual metadata field docs
vagimeli Apr 17, 2024
0b4bf5e
Added documentation for field names and ignored
vagimeli Apr 17, 2024
bd3b85c
Add id field doc
vagimeli Apr 17, 2024
d6cadd1
Add field docs
vagimeli Apr 17, 2024
4b6f1d1
Add new docs
vagimeli Apr 18, 2024
57be501
Merge branch 'main' into mapping-content-gap
vagimeli Apr 29, 2024
a9f6f4f
Add default and allowed values
vagimeli Apr 29, 2024
5b7ab8b
Add default and allowed values
vagimeli Apr 29, 2024
2b4db59
Update field-names.md
vagimeli Apr 29, 2024
edd60c6
Update index-metadata.md
vagimeli Apr 29, 2024
a593836
Update index-metadata.md
vagimeli Apr 29, 2024
0e41a24
Update meta.md
vagimeli Apr 29, 2024
6c0c5c8
Update meta.md
vagimeli Apr 29, 2024
5e11c4a
Update index-metadata.md
vagimeli Apr 29, 2024
3dcfd50
Update meta.md
vagimeli Apr 29, 2024
9fcc2eb
Update index-metadata.md
vagimeli Apr 29, 2024
3d30dfb
Update routing.md
vagimeli Apr 29, 2024
56991f3
Update routing.md
vagimeli Apr 29, 2024
a7f39c0
Update source.md
vagimeli Apr 29, 2024
c286a28
Merge branch 'main' into mapping-content-gap
vagimeli May 8, 2024
078c49f
Update index.md
vagimeli May 8, 2024
02cc9dd
Merge branch 'main' into mapping-content-gap
vagimeli May 14, 2024
aa6ac4d
Merge branch 'main' into mapping-content-gap
vagimeli May 14, 2024
415ef50
Update _field-types/index.md
vagimeli May 15, 2024
11e1a1c
Update _field-types/metadata-fields/id.md
vagimeli May 15, 2024
ffd96dc
Add dynamic templates section and examples
vagimeli Jun 3, 2024
5c64fd6
Add dynamic templates code snippet
vagimeli Jun 3, 2024
8e7a288
Update _field-types/metadata-fields/source.md
vagimeli Jun 26, 2024
459f3d0
Update _field-types/index.md
vagimeli Jun 26, 2024
0c451a5
Update _field-types/index.md
vagimeli Jun 26, 2024
8eefb2b
Update _field-types/index.md
vagimeli Jun 26, 2024
709d4b3
Update _field-types/index.md
vagimeli Jun 26, 2024
ced8826
Update _field-types/index.md
vagimeli Jun 26, 2024
23943c8
Update _field-types/index.md
vagimeli Jun 26, 2024
91d0f18
Update _field-types/metadata-fields/field-names.md
vagimeli Jun 26, 2024
8485d22
Update field-names.md
vagimeli Jun 26, 2024
a5ccbd9
Update _field-types/metadata-fields/id.md
vagimeli Jun 26, 2024
d5a6655
Merge branch 'main' into mapping-content-gap
vagimeli Jun 26, 2024
f478040
Update _field-types/index.md
vagimeli Jul 11, 2024
f3337a6
Update _field-types/index.md
vagimeli Jul 11, 2024
e21c9af
Writing in progress
vagimeli Jul 16, 2024
f2516c9
Update _field-types/metadata-fields/id.md
vagimeli Jul 17, 2024
365b145
Update _field-types/metadata-fields/field-names.md
vagimeli Jul 17, 2024
2292abc
Update _field-types/metadata-fields/field-names.md
vagimeli Jul 17, 2024
a114b6e
Update _field-types/metadata-fields/field-names.md
vagimeli Jul 17, 2024
47a1bf9
Update _field-types/metadata-fields/index.md
vagimeli Jul 17, 2024
00fc9c1
Update ignored.md
vagimeli Jul 17, 2024
ae99ef0
Merge branch 'main' into mapping-content-gap
vagimeli Jul 17, 2024
61a9f6b
Update index.md
vagimeli Aug 28, 2024
40d9b43
Update index.md
vagimeli Aug 28, 2024
284b73e
Merge branch 'main' into mapping-content-gap
vagimeli Aug 28, 2024
c7d9585
Update _field-types/mappings-use-cases.md
vagimeli Aug 28, 2024
dc2792e
Update mappings-use-cases.md
vagimeli Aug 28, 2024
83bfc25
Update field-names.md
vagimeli Aug 28, 2024
4389093
Update id.md
vagimeli Aug 28, 2024
02af070
Update ignored.md
vagimeli Aug 28, 2024
f83fddf
Update index-metadata.md
vagimeli Aug 28, 2024
9c819eb
Update index.md
vagimeli Aug 28, 2024
20bdec2
Update meta.md
vagimeli Aug 28, 2024
e38ec49
Update _field-types/metadata-fields/routing.md
vagimeli Aug 28, 2024
8f22a9c
Update _field-types/metadata-fields/routing.md
vagimeli Aug 28, 2024
c1a2315
Update _field-types/metadata-fields/meta.md
vagimeli Aug 28, 2024
bd8ac77
Update _field-types/metadata-fields/source.md
vagimeli Aug 28, 2024
cf4222a
Update _field-types/index.md
vagimeli Aug 28, 2024
243357a
Update _field-types/metadata-fields/field-names.md
vagimeli Aug 28, 2024
375488b
Update source.md
vagimeli Aug 28, 2024
74b82c3
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 28, 2024
1262bae
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 28, 2024
2e2c00b
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 28, 2024
1d622df
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 28, 2024
022eb23
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 28, 2024
c23b805
Update _field-types/metadata-fields/id.md
vagimeli Aug 28, 2024
9f7694e
Update id.md
vagimeli Aug 28, 2024
97cb348
Update _field-types/index.md
vagimeli Aug 28, 2024
4524c1a
Update _field-types/index.md
vagimeli Aug 28, 2024
27c40c8
Update _field-types/index.md
vagimeli Aug 28, 2024
ff7c843
Merge branch 'main' into mapping-content-gap
vagimeli Aug 28, 2024
c33c047
Update _field-types/metadata-fields/index.md
vagimeli Aug 28, 2024
596fa72
Update _field-types/metadata-fields/id.md
vagimeli Aug 28, 2024
57cbe5e
Update field-names.md
vagimeli Aug 28, 2024
3d9f7ba
Address doc review comments
vagimeli Aug 28, 2024
60ecace
Address doc review comments
vagimeli Aug 28, 2024
30313fe
Update _field-types/index.md
vagimeli Aug 29, 2024
4f0b8e4
Update _field-types/index.md
vagimeli Aug 29, 2024
fc62bcd
Update _field-types/index.md
vagimeli Aug 29, 2024
3cb42e1
Update _field-types/index.md
vagimeli Aug 29, 2024
53354fe
Update _field-types/index.md
vagimeli Aug 29, 2024
cfc9997
Update _field-types/index.md
vagimeli Aug 29, 2024
31597db
Update _field-types/index.md
vagimeli Aug 29, 2024
95e7ef9
Update _field-types/index.md
vagimeli Aug 29, 2024
d67b974
Update _field-types/index.md
vagimeli Aug 29, 2024
5ad15d9
Update _field-types/index.md
vagimeli Aug 29, 2024
6a30faf
Update _field-types/index.md
vagimeli Aug 29, 2024
28a2923
Update _field-types/index.md
vagimeli Aug 29, 2024
c25d86b
Update _field-types/index.md
vagimeli Aug 29, 2024
01131a4
Update _field-types/index.md
vagimeli Aug 29, 2024
a49b1ac
Update _field-types/index.md
vagimeli Aug 29, 2024
e2d2d0d
Update _field-types/index.md
vagimeli Aug 29, 2024
bfa2952
Update _field-types/index.md
vagimeli Aug 29, 2024
9744ab7
Update _field-types/index.md
vagimeli Aug 29, 2024
4898e09
Update _field-types/index.md
vagimeli Aug 29, 2024
c44a46a
Update _field-types/index.md
vagimeli Aug 29, 2024
e1cef5f
Update _field-types/index.md
vagimeli Aug 29, 2024
7eaefbc
Update _field-types/index.md
vagimeli Aug 29, 2024
7f268dc
Update _field-types/mappings-use-cases.md
vagimeli Aug 29, 2024
71e6ed4
Update _field-types/mappings-use-cases.md
vagimeli Aug 29, 2024
e49c1d3
Update _field-types/mappings-use-cases.md
vagimeli Aug 29, 2024
a708be7
Update _field-types/mappings-use-cases.md
vagimeli Aug 29, 2024
5ee0a4d
Update _field-types/mappings-use-cases.md
vagimeli Aug 29, 2024
8f2c11e
Update _field-types/metadata-fields/field-names.md
vagimeli Aug 29, 2024
e5bc821
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
1c9328d
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
df556d7
Update _field-types/metadata-fields/field-names.md
vagimeli Aug 29, 2024
e1876b6
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
4202902
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
b970b51
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
2aefa3c
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
381f3c9
Update _field-types/metadata-fields/id.md
vagimeli Aug 29, 2024
2b6ada7
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 29, 2024
88d3c1e
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 29, 2024
46c0639
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 29, 2024
9087d12
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 29, 2024
f1a195d
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 29, 2024
e53ec2a
Update _field-types/metadata-fields/ignored.md
vagimeli Aug 29, 2024
e90eb17
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 29, 2024
b700ff7
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 29, 2024
670328f
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 29, 2024
b9ccc47
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 29, 2024
116cecb
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 29, 2024
ce89a83
Update _field-types/metadata-fields/index-metadata.md
vagimeli Aug 29, 2024
7bc5bfd
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
7b051f5
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
7213aa9
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
581b118
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
916a0dc
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
9d8a1e7
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
09e64a5
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
234d6a0
Update _field-types/metadata-fields/index.md
vagimeli Aug 29, 2024
230c830
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
5b24526
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
f360baf
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
c893666
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
5802589
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
3552222
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
ae73ae8
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
d2cb1b5
Update _field-types/metadata-fields/meta.md
vagimeli Aug 29, 2024
b0f0b92
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
cd6b1aa
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
2e8ff8b
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
a5074df
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
9654a1b
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
097f3ef
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
5e263e3
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
421e97f
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
c834b5a
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
044fb6c
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
ba7b39b
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
dca4c32
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
92a949e
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
e3d8d0b
Update _field-types/metadata-fields/source.md
vagimeli Aug 29, 2024
5573bc9
Update _field-types/metadata-fields/source.md
vagimeli Aug 29, 2024
eac468b
Update _field-types/metadata-fields/source.md
vagimeli Aug 29, 2024
1037b6e
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
215a3b7
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
0cb56be
Update routing.md
vagimeli Aug 29, 2024
219b1e3
Update _field-types/metadata-fields/routing.md
vagimeli Aug 29, 2024
2dc0e2a
Merge branch 'main' into mapping-content-gap
vagimeli Aug 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 85 additions & 97 deletions _field-types/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,43 +12,77 @@

# Mappings and field types

You can define how documents and their fields are stored and indexed by creating a _mapping_. The mapping specifies the list of fields for a document. Every field in the document has a _field type_, which defines the type of data the field contains. For example, you may want to specify that the `year` field should be of type `date`. To learn more, see [Supported field types]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/).
Mappings tell OpenSearch how to store and index your documents and their fields. You can specify the data type for each field (for example, `year` as `date`) to make storage and querying more efficient.

If you're just starting to build out your cluster and data, you may not know exactly how your data should be stored. In those cases, you can use dynamic mappings, which tell OpenSearch to dynamically add data and its fields. However, if you know exactly what types your data falls under and want to enforce that standard, then you can use explicit mappings.
While [dynamic mappings](#dynamic-mapping) automatically add new data and fields, using explicit mappings is recommended. Explicit mappings let you define the exact structure and data types upfront. This helps to maintain data consistency and optimize performance, especially for large datasets or high-volume indexing operations.

For example, if you want to indicate that `year` should be of type `text` instead of an `integer`, and `age` should be an `integer`, you can do so with explicit mappings. By using dynamic mapping, OpenSearch might interpret both `year` and `age` as integers.
For example, with explicit mappings, you can ensure that `year` is treated as text and `age` as an integer instead of both being interpreted as integers by dynamic mapping.

This section provides an example for how to create an index mapping and how to add a document to it that will get ip_range validated.

#### Table of contents
1. TOC
{:toc}


---
## Dynamic mapping

When you index a document, OpenSearch adds fields automatically with dynamic mapping. You can also explicitly add fields to an index mapping.

#### Dynamic mapping types
### Dynamic mapping types

Type | Description
:--- | :---
null | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if that field has no values.
boolean | OpenSearch accepts `true` and `false` as boolean values. An empty string is equal to `false.`
float | A single-precision 32-bit floating point number.
double | A double-precision 64-bit floating point number.
integer | A signed 32-bit number.
object | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
array | Arrays in OpenSearch can only store values of one type, such as an array of just integers or strings. Empty arrays are treated as though they are fields with no values.
text | A string sequence of characters that represent full-text values.
keyword | A string sequence of structured characters, such as an email address or ZIP code.
`null` | A `null` field can't be indexed or searched. When a field is set to null, OpenSearch behaves as if the field has no value.
`boolean` | OpenSearch accepts `true` and `false` as Boolean values. An empty string is equal to `false.`
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`float` | A single-precision, 32-bit floating-point number.
`double` | A double-precision, 64-bit floating-point number.
`integer` | A signed 32-bit number.
`object` | Objects are standard JSON objects, which can have fields and mappings of their own. For example, a `movies` object can have additional properties such as `title`, `year`, and `director`.
`array` | OpenSearch does not have a specific array data type. Arrays are represented as a set of values of the same data type (for example, integers or strings) associated with a field. When indexing, you can pass multiple values for a field, and OpenSearch will treat it as an array. Empty arrays are valid and recognized as array fields with zero elements---not as fields with no values. OpenSearch supports querying and filtering arrays, including checking for values, range queries, and array operations like concatenation and intersection. Nested arrays, which may contain complex objects or other arrays, can also be used for advanced data modeling.
`text` | A string sequence of characters that represent full-text values.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above, last sentence: "Nested arrays, which may contain complex objects or other arrays, can also be used for advanced data modeling"?

`keyword` | A string sequence of structured characters, such as an email address or ZIP code.
date detection string | Enabled by default, if new string fields match a date's format, then the string is processed as a `date` field. For example, `date: "2012/03/11"` is processed as a date.
numeric detection string | If disabled, OpenSearch may automatically process numeric values as strings when they should be processed as numbers. When enabled, OpenSearch can process strings into `long`, `integer`, `short`, `byte`, `double`, `float`, `half_float`, `scaled_float`, and `unsigned_long`. Default is disabled.

### Dynamic templates

Dynamic templates are used to define custom mappings for dynamically added fields based on the data type, field name, or field path. They allow you to define a flexible schema for your data that can automatically adapt to changes in the structure or format of the input data.

You can use the following syntax to define a dynamic mapping template:

```json
PUT index
{
"mappings": {
"dynamic_templates": [
{
"fields": {
"mapping": {
"type": "short"
},
"match_mapping_type": "string",
"path_match": "status*"
}
}
]
}
}
```
{% include copy-curl.html %}

This mapping configuration dynamically maps any field with a name starting with `status` (for example, `status_code`) to the `short` data type if the initial value provided during indexing is a string.

### Dynamic mapping parameters

The `dynamic_templates` support the following parameters for matching conditions and mapping rules. The default value is `null`.

Parameter | Description |
----------|-------------|
`match_mapping_type` | Specifies the JSON data type (for example, string, long, double, object, binary, Boolean, date) that triggers the mapping.
`match` | A regular expression used to match field names and apply the mapping.
`unmatch` | A regular expression used to exclude field names from the mapping.
`match_pattern` | Determines the pattern matching behavior, either `regex` or `simple`. Default is `simple`.
`path_match` | Allows you to match nested field paths using a regular expression.
`path_unmatch` | Excludes nested field paths from the mapping using a regular expression.
`mapping` | The mapping configuration to apply.

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
## Explicit mapping

If you know exactly what your field data types need to be, you can specify them in your request body when creating your index.
If you know exactly which field data types you need to use, then you can specify them in your request body when creating your index, as shown in the following example request:

```json
PUT sample-index1
Expand All @@ -62,17 +96,19 @@
}
}
```
{% include copy-curl.html %}

### Response
#### Response
```json
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "sample-index1"
}
```
{% include copy-curl.html %}

To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method:
To add mappings to an existing index or data stream, you can send a request to the `_mapping` endpoint using the `PUT` or `POST` HTTP method, as shown in the following example request:

```json
POST sample-index1/_mapping
Expand All @@ -84,84 +120,29 @@
}
}
```
{% include copy-curl.html %}

You cannot change the mapping of an existing field, you can only modify the field's mapping parameters.
{: .note}

---
## Mapping example usage
## Mapping parameters

The following example shows how to create a mapping to specify that OpenSearch should ignore any documents with malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You accomplish this by setting the `ignore_malformed` parameter to `true`.
Mapping parameters are used to configure the behavior of index fields. See [Mappings and field types]({{site.url}}{{site.baseurl}}/field-types/) for more information.

### Create an index with an `ip` mapping
## Mapping limit settings

To create an index, use a PUT request:
OpenSearch has certain mapping limits and settings, such as the settings listed in the following table. Settings can be configured based on your requirements.

```json
PUT /test-index
{
"mappings" : {
"properties" : {
"ip_address" : {
"type" : "ip",
"ignore_malformed": true
}
}
}
}
```

You can add a document that has a malformed IP address to your index:

```json
PUT /test-index/_doc/1
{
"ip_address" : "malformed ip address"
}
```

This indexed IP address does not throw an error because `ignore_malformed` is set to true.

You can query the index using the following request:

```json
GET /test-index/_search
```
| Setting | Default value | Allowed value | Type | Description |
|-|-|-|-|-|
| `index.mapping.nested_fields.limit` | 50 | [0,) | Dynamic | Limits the maximum number of nested fields that can be defined in an index mapping. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `index.mapping.nested_objects.limit` | 10,000 | [0,) | Dynamic | Limits the maximum number of nested objects that can be created in a single document. |
| `index.mapping.total_fields.limit` | 1,000 | [0,) | Dynamic | Limits the maximum number of fields that can be defined in an index mapping. |
| `index.mapping.depth.limit` | 20 | [1,100] | Dynamic | Limits the maximum depth of nested objects and nested fields that can be defined in an index mapping. |
| `index.mapping.field_name_length.limit` | 50,000 | [1,50000] | Dynamic | Limits the maximum length of field names that can be defined in an index mapping. |
| `index.mapper.dynamic` | true | {true,false} | Dynamic | Determines whether new fields should be dynamically added to a mapping. |

Check failure on line 143 in _field-types/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _field-types/index.md#L143

[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'true,false'.
Raw output
{"message": "[OpenSearch.SpacingPunctuation] There should be no space before and one space after the punctuation mark in 'true,false'.", "location": {"path": "_field-types/index.md", "range": {"start": {"line": 143, "column": 36}}}, "severity": "ERROR"}

The response shows that the `ip_address` field is ignored in the indexed document:

```json
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "test-index",
"_id": "1",
"_score": 1,
"_ignored": [
"ip_address"
],
"_source": {
"ip_address": "malformed ip address"
}
}
]
}
}
```
---

## Get a mapping

Expand All @@ -170,29 +151,31 @@
```json
GET <index>/_mapping
```
{% include copy-curl.html %}

In the above request, `<index>` may be an index name or a comma-separated list of index names.
In the previous request, `<index>` may be an index name or a comma-separated list of index names.

To get all mappings for all indexes, use the following request:

```json
GET _mapping
```
{% include copy-curl.html %}

To get a mapping for a specific field, provide the index name and the field name:

```json
GET _mapping/field/<fields>
GET /<index>/_mapping/field/<fields>
```
{% include copy-curl.html %}

Both `<index>` and `<fields>` can be specified as one value or a comma-separated list.

For example, the following request retrieves the mapping for the `year` and `age` fields in `sample-index1`:
Both `<index>` and `<fields>` can be specified as either one value or a comma-separated list. For example, the following request retrieves the mapping for the `year` and `age` fields in `sample-index1`:

```json
GET sample-index1/_mapping/field/year,age
```
{% include copy-curl.html %}

The response contains the specified fields:

Expand Down Expand Up @@ -220,3 +203,8 @@
}
}
```
{% include copy-curl.html %}

## Mappings use cases

See [Mappings use cases]({{site.url}}{{site.baseurl}}/field-types/mappings-use-cases/) for use case examples, including examples of mapping string fields and ignoring malformed IP addresses.
122 changes: 122 additions & 0 deletions _field-types/mappings-use-cases.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
---
layout: default
title: Mappings use cases
parent: Mappings and fields types
nav_order: 5
nav_exclude: true
---

# Mappings use cases

Mappings provide control over how data is indexed and queried, enabling optimized performance and efficient storage for a range of use cases.

---

## Example: Ignoring malformed IP addresses

The following example shows you how to create a mapping specifying that OpenSearch should ignore any documents containing malformed IP addresses that do not conform to the [`ip`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/ip/) data type. You can accomplish this by setting the `ignore_malformed` parameter to `true`.

### Create an index with an `ip` mapping

To create an index with an `ip` mapping, use a PUT request:

```json
PUT /test-index
{
"mappings" : {
"properties" : {
"ip_address" : {
"type" : "ip",
"ignore_malformed": true
}
}
}
}
```
{% include copy-curl.html %}

Then add a document with a malformed IP address:

```json
PUT /test-index/_doc/1
{
"ip_address" : "malformed ip address"
}
```
{% include copy-curl.html %}

When you query the index, the `ip_address` field will be ignored. You can query the index using the following request:

```json
GET /test-index/_search
```
{% include copy-curl.html %}

#### Response

```json
{
"took": 14,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "test-index",
"_id": "1",
"_score": 1,
"_ignored": [
"ip_address"
],
"_source": {
"ip_address": "malformed ip address"
}
}
]
}
}
```
{% include copy-curl.html %}

---

## Mapping string fields to `text` and `keyword` types

To create an index named `movies1` with a dynamic template that maps all string fields to both the `text` and `keyword` types, you can use the following request:

```json
PUT movies1
{
"mappings": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
]
}
}
```
{% include copy-curl.html %}

This dynamic template ensures that any string fields in your documents will be indexed as both a full-text `text` type and a `keyword` type.
Loading
Loading