Add flat object field type #3714
Conversation
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
_field-types/flat-object.md
Outdated
|
|
||
| In OpenSearch, you don't have to specify a mapping before indexing documents. If you don't specify a mapping, OpenSearch uses [dynamic mapping]({{site.url}}{{site.baseurl}}/field-types/mappings#dynamic-mapping) to map every field and its subfields in the document automatically. When you ingest documents such as logs, you may not know every field's subfield name and type in advance. In this case, dynamically mapping all new subfields can quickly lead to a "mapping explosion", where the growing number of fields may degrade the performance of your cluster. | ||
|
|
||
| Flat object solves this problem by not indexing the field itself or its subfields. Instead, for any JSON object, the values of all subfields and their paths are stored in two string fields. Subfields within the JSON are accessible using standard dot path notation but are not indexed for fast lookup. |
There was a problem hiding this comment.
"Flat object solves this problem by treating the entire JSON object as string and does not indexing its subfields". The parent field is indexed, but the subfields are not indexable fields.
_field-types/flat-object.md
Outdated
|
|
||
| Flat object solves this problem by not indexing the field itself or its subfields. Instead, for any JSON object, the values of all subfields and their paths are stored in two string fields. Subfields within the JSON are accessible using standard dot path notation but are not indexed for fast lookup. | ||
|
|
||
| The maximum field path length in the dot notation is 2<sup>24</sup> − 1. |
There was a problem hiding this comment.
Should be "The maximum field value length "
_field-types/flat-object.md
Outdated
|
|
||
| - Efficient reads: Fetching performance is similar to that of a keyword field. | ||
| - Memory efficiency: Storing the entire complex JSON object in one field without indexing all its subfields reduces the number of fields in an index. | ||
| - Space efficiency: OpenSearch does not create an inverted index for flat objects, thereby saving space. |
There was a problem hiding this comment.
OpenSearch does not create an inverted index for subfields in flat objects
_field-types/flat-object.md
Outdated
| Flat objects support: | ||
|
|
||
| - Exact match queries. | ||
| - Textual sorting. |
There was a problem hiding this comment.
Not supported for sorting, aggregation and filtering
There was a problem hiding this comment.
@mingshl So both textual and numerical sorting are not supported?
_field-types/flat-object.md
Outdated
|
|
||
| Mapping a field as flat object applies when a field and its subfields are mostly read and not used as a search criteria because the subfields are not indexed. Flat objects are useful for objects with a large number of fields or when you don't know the keys in advance. | ||
|
|
||
| Flat objects support: |
There was a problem hiding this comment.
supports querying with dot path and without dot path
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
|
LGTM |
_field-types/flat-object.md
Outdated
|
|
||
| The following example illustrates mapping a field as a flat object, indexing documents with flat object fields, and searching for leaf values of the flat object in those documents. | ||
|
|
||
| Only root fields of a document can be defined as a flat objects. You cannot define an object that is part of another JSON object as a flat object because when a flat object it is flattened to a string, the nested architecture of the leaves is lost. |
There was a problem hiding this comment.
Only the root field of a document can be defined as a flat object. Better to keep the singular form.
_field-types/flat-object.md
Outdated
|
|
||
| # Flat object field type | ||
|
|
||
| In OpenSearch, you don't have to specify a mapping before indexing documents. If you don't specify a mapping, OpenSearch uses [dynamic mapping]({{site.url}}{{site.baseurl}}/field-types/mappings#dynamic-mapping) to map every field and its subfields in the document automatically. When you ingest documents such as logs, you may not know every field's subfield name and type in advance. In this case, dynamically mapping all new subfields can quickly lead to a "mapping explosion", where the growing number of fields may degrade the performance of your cluster. |
There was a problem hiding this comment.
not a blocker for this documentation, but it would be good for us to have more documentation on our site to define "mapping explosion." It is a specific issue where too many distinct fields cause issues on the cluster manager. Does that fit into the current doc site somewhere?
There was a problem hiding this comment.
@macohen I'm open to having a separate page if we have more information to put on it. If it's only this one definition, then I think the last sentence in this paragraph conveys the same information.
_field-types/flat-object.md
Outdated
|
|
||
| This functionality is planned for a future release. | ||
|
|
||
| ## Using flat objects |
There was a problem hiding this comment.
similar to Mingshi's comment, should this be "Using flat object?"
| - Flat objects do not support open parameters. | ||
| - Painless scripting and wildcard queries are not supported for retrieving values of subfields. | ||
|
|
||
| This functionality is planned for a future release. |
There was a problem hiding this comment.
do we ever put GH issue links here to get feedback for these features? i understand if documentation isn't the right place, but it could be a really nice way to solicit feedback if someone is this deep into reading the docs.
There was a problem hiding this comment.
@macohen We do for experimental features. For non-experimental features, we have the feedback panel where people can leave comments on the page. However, in this case, I think a link to a GitHub issue to solicit feedback is appropriate. I can add it if you give me the link.
Signed-off-by: Fanit Kolchina <[email protected]>
There was a problem hiding this comment.
@kolchfa-aws Some comments that should be quick to resolve.
_field-types/flat-object.md
Outdated
| The maximum field value length in the dot notation is 2<sup>24</sup> − 1. | ||
| {: .note} | ||
|
|
||
| Flat objects provide the following advantages: |
There was a problem hiding this comment.
In the preceding paragraphs, "flat object" is used in singular form. Is there a distinction between the usage above and the usage in lines 19-33? For example, in line 14, are we referring to the field type? If so, I suggest writing "The flat object field type..."
There was a problem hiding this comment.
Changed. Thanks!
| } | ||
| ``` | ||
|
|
||
| Alternatively, if you know the subfield name in which to search, provide the field's path in dot notation: |
There was a problem hiding this comment.
We use the verbiage "dot path notation" in preceding sections. Should we be consistent here?
There was a problem hiding this comment.
This sentence already has the word "path" so I think we don't need to specify it again. Let's ask @natebower
There was a problem hiding this comment.
I'm fine with it as is as long as both "dot path notation" and "dot notation" are commonly understood as referring to the same thing.
Co-authored-by: Melissa Vagi <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
_field-types/flat-object.md
Outdated
| - Space efficiency: OpenSearch does not create an inverted index for subfields in flat objects, thereby saving space. | ||
| - Compatibility for migration: You can migrate your data from systems that support similar flat types to OpenSearch. | ||
|
|
||
| Mapping a field as a flat object applies when a field and its subfields are mostly read and not used as a search criteria because the subfields are not indexed. Flat objects are useful for objects with a large number of fields or when you don't know the keys in advance. |
There was a problem hiding this comment.
Either "as search criteria" or "as a search criterion"
|
|
||
| This functionality is planned for a future release. | ||
|
|
||
| ## Using flat object |
There was a problem hiding this comment.
Wasn't this going to be "Using flat objects"?
|
|
||
| ## Using flat object | ||
|
|
||
| The following example illustrates mapping a field as a flat object, indexing documents with flat object fields, and searching for leaf values of the flat object in those documents. |
There was a problem hiding this comment.
Confirm that the last instance of "object" is intentionally singular.
| } | ||
| ``` | ||
|
|
||
| Alternatively, if you know the subfield name in which to search, provide the field's path in dot notation: |
There was a problem hiding this comment.
I'm fine with it as is as long as both "dot path notation" and "dot notation" are commonly understood as referring to the same thing.
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
hdhalter
added
6 - Done but waiting to merge
|
|
||
| Flat objects support exact match queries with and without dot path notation. For a complete list of supported query types, see [Supported queries](#supported-queries). | ||
|
|
||
| Searching for a specific value of a nested field in a document may be inefficient because it may require a full scan of the index, which can be an expensive operation. |
There was a problem hiding this comment.
expensive in what sense? Literally, it costs a business money? Or does it use a lot of resources?
There was a problem hiding this comment.
This is standard computer science speak. In terms of resources :)
|
|
||
| The following example illustrates mapping a field as a flat object, indexing documents with flat object fields, and searching for leaf values of the flat object in those documents. | ||
|
|
||
| Only the root field of a document can be defined as a flat object. You cannot define an object that is part of another JSON object as a flat object because when a flat object is flattened to a string, the nested architecture of the leaves is lost. |
There was a problem hiding this comment.
| Only the root field of a document can be defined as a flat object. You cannot define an object that is part of another JSON object as a flat object because when a flat object is flattened to a string, the nested architecture of the leaves is lost. | |
| Only the root field of a document can be defined as a flat object. You cannot define an object as a flat object when it is part of another JSON object because when a flat object is flattened to a string, the nested architecture of the leaves is lost. |
I'm not familiar with "leaves" in this usage. Just wanted to confirm that's right. I see "leaf value" referred to in other places. Could this be "leaf values" for the plural?
There was a problem hiding this comment.
Leaves or leaf values is the same. Refers to leaves of a tree.
|
|
||
| ```json | ||
| POST /test-index/_search | ||
| GET /test-index/_search |
There was a problem hiding this comment.
Why changing the request command from POST to GET? the HTTP POST command is used for queries that have a request body, such as search queries.
The request body can be large: search queries can be complex. When using the HTTP GET command, the URL has a limited length, and if the request body is too large, it may not fit in the URL. By using the HTTP POST command, the request body can be sent separately in the request payload, and there is no limit to its size.
There was a problem hiding this comment.
@mingshl GET is the standard for searching in OpenSearch. it was surprising to me, too, that you can send a payload like you do with POST in a GET. Take a look at https://opensearch.org/docs/latest/aggregations/metric-agg/ and other query docs. the HTTP spec says we're putting meaning where it shouldn't be, but it doesn't disallow it: "A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request." https://datatracker.ietf.org/doc/html/rfc7231#section-4.3.1
kolchfa-aws
added
the
release-notes
* Add flat object field type Signed-off-by: Fanit Kolchina <[email protected]> * Adds more examples and notes Signed-off-by: Fanit Kolchina <[email protected]> * Implemented tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Update page order Signed-off-by: Fanit Kolchina <[email protected]> * Implemented tech review feedback Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Melissa Vagi <[email protected]> * More doc review comments Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> * Implemented the last editorial comment Signed-off-by: Fanit Kolchina <[email protected]> * Changed POST to GET Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
* Add flat object field type Signed-off-by: Fanit Kolchina <[email protected]> * Adds more examples and notes Signed-off-by: Fanit Kolchina <[email protected]> * Implemented tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Update page order Signed-off-by: Fanit Kolchina <[email protected]> * Implemented tech review feedback Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Melissa Vagi <[email protected]> * More doc review comments Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> * Implemented the last editorial comment Signed-off-by: Fanit Kolchina <[email protected]> * Changed POST to GET Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
Fixes #2657
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.