Reflect latest changes in synthetic source documentation #109501

lkts · 2024-06-07T21:49:57Z

This PR changes synthetic source documentation to reflect recent improvements.

github-actions · 2024-06-07T21:50:07Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-06-07T21:50:20Z

Pinging @elastic/es-docs (Team:Docs)

elasticsearchmachine · 2024-06-07T21:50:20Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

lkts · 2024-06-07T21:50:58Z

docs/reference/mapping/fields/synthetic-source.asciidoc

@@ -61,10 +60,8 @@ types:
 ** <<numeric-synthetic-source,`long`>>
 ** <<range-synthetic-source,`range` types>>
 ** <<numeric-synthetic-source,`scaled_float`>>
-** <<search-as-you-type-synthetic-source,`search_as_you_type`>>
 ** <<numeric-synthetic-source,`short`>>
 ** <<text-synthetic-source,`text`>>


text is controversial - if there is no keyword subfield it will store text in a field which is kind of the same as non-native approach

We should have a category "It's complicated" :P

imo native means the field type handling synthetic source without relying on ignored source. It says nothing about how it implements synthetic source.

lkts · 2024-06-07T21:52:59Z

I wonder if we should expand "modifications" section on this page and mention f.e. precision loss in point fields.

lkts · 2024-06-07T21:57:56Z

Another question is if we should add section for synthetic source for all field types. It is not too useful in my opinion since it will just say "Synthetic source is supported by default.".

kkrik-es · 2024-06-08T05:26:25Z

docs/reference/mapping/fields/synthetic-source.asciidoc


-There are a couple of restrictions to be aware of:
+<<synthetic-source-fields-native-list, Majority of field types>> natively support synthetic `_source`. Source for the field is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning no additional space is used.


Nit: s/Majority/ The majority/ ?

Maybe "Source for the field is usually reconstructed using doc values. However, sometimes stored fields are used, which can result in slight storage overhead."

kkrik-es · 2024-06-08T05:29:06Z

Wanna mention something about objects (disabled, dynamic, ignore_*) and nested objects?

Also, shall we add the highlight note as mentioned in the mtg?

kkrik-es · 2024-06-08T05:32:39Z

I wonder if we should expand "modifications" section on this page and mention f.e. precision loss in point fields.

++

kkrik-es · 2024-06-08T05:33:30Z

Another question is if we should add section for synthetic source for all field types. It is not too useful in my opinion since it will just say "Synthetic source is supported by default.".

Yeah maybe use italics or smth to stress the by default part.

martijnvg · 2024-06-10T07:07:45Z

docs/reference/mapping/fields/synthetic-source.asciidoc

-<<synthetic-source-modifications,modifications>> compared to the original JSON.
-* Synthetic `_source` can be used with indices that contain only these field
-types:
+If a field type does not have native synthetic `_source` support, original data of this field is stored as is in the same way as the `_source` field in non-synthetic mode. Similarly, malformed values of fields that use <<ignore-malformed,`ignore_malformed`>> need to be stored as is, requiring additional storage space.


Maybe add that the part of the _source is then stored in a special source field called _ignore_source?

I think we either explain somewhere the difference between native and non-native synthetic source...so that we can use the terminology later or we don't use such terminology and explain it differently like for instance:

"If a field type supports document reconstruction by means of doc values...otherwise, if doc values are not available the field value is stored as is..."

I am not in favor of one or the other...just saying that if we use "native synthetic source" we should explain it beforehand...or don't use it at all.

martijnvg · 2024-06-10T07:09:03Z

docs/reference/mapping/fields/source-field.asciidoc

@@ -7,8 +7,7 @@ searchable), but it is stored so that it can be returned when executing
 _fetch_ requests, like <<docs-get,get>> or <<search-search,search>>.

 If disk usage is important to you then have a look at
-<<synthetic-source,synthetic `_source`>> which shrinks disk usage at the cost of
-only supporting a subset of mappings and slower fetches or (not recommended)
+<<synthetic-source,synthetic `_source`>> which shrinks disk usage at the cost of slower fetches or (not recommended)


Maybe rename at the cost of slower fetches or (not recommended) into at the cost of slower access to _source, in get and search APIs

lkts · 2024-06-10T14:51:30Z

@kkrik-es Yes, i'll add a highlight separately. We are still accumulating highlights :)

shainaraskas

This info is super helpful! 🎉 Added some minor grammar/flow recommendations.

shainaraskas · 2024-06-11T17:03:03Z

docs/reference/data-streams/tsds.asciidoc

@@ -54,7 +54,7 @@ documents, the document `_id` is a hash of the document's dimensions and
 `@timestamp`. A TSDS doesn't support custom document `_id` values.

 * A TSDS uses <<synthetic-source,synthetic `_source`>>, and as a result is
-subject to a number of <<synthetic-source-restrictions,restrictions>>.
+subject to <<synthetic-source-modifications,modifications>> applied to `_source` field.


Suggested change

subject to <<synthetic-source-modifications,modifications>> applied to `_source` field.

subject to <<synthetic-source-modifications,modifications>> applied to the `_source` field.

missing the haunts me all my life, thank you

shainaraskas · 2024-06-11T17:49:11Z

docs/reference/mapping/fields/source-field.asciidoc

@@ -7,8 +7,7 @@ searchable), but it is stored so that it can be returned when executing
 _fetch_ requests, like <<docs-get,get>> or <<search-search,search>>.

 If disk usage is important to you then have a look at


Suggested change

If disk usage is important to you then have a look at

If disk usage is important to you, then consider the following options:

shainaraskas · 2024-06-11T17:56:02Z

docs/reference/mapping/fields/source-field.asciidoc

+<<synthetic-source,synthetic `_source`>> which shrinks disk usage at the cost of slower fetches or (not recommended)
 <<disable-source-field,disabling the `_source` field>> which also shrinks disk
 usage but disables many features.


consider defining these briefly up here. if you don't want to define them here, I'd remove the downsides and keep the links only.

Suggested change

<<synthetic-source,synthetic `_source`>> which shrinks disk usage at the cost of slower fetches or (not recommended)

<<disable-source-field,disabling the `_source` field>> which also shrinks disk

usage but disables many features.

- Using <<synthetic-source,synthetic `_source`>>, which reconstructs source content at the time of retrieval instead of storing it on disk. This shrinks disk usage, at the cost of slower access to `_source` in <<docs-get,Get>> and <<search-search,Search>> queries.

- <<disable-source-field,Disabling the `_source` field completely>>. This shrinks disk

usage but disables features that rely on `_source`.

shainaraskas · 2024-06-11T17:58:23Z

docs/reference/mapping/fields/synthetic-source.asciidoc


-There are a couple of restrictions to be aware of:
+<<synthetic-source-fields-native-list, Majority of field types>> natively support synthetic `_source`. Source for the field is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning no additional space is used.


Suggested change

<<synthetic-source-fields-native-list, Majority of field types>> natively support synthetic `_source`. Source for the field is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning no additional space is used.

<<synthetic-source-fields-native-list,Most field types>> natively support synthetic `_source`. The source for natively supported fields is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning that no additional space is used.

lkts · 2024-06-27T18:52:32Z

@elasticmachine update branch

elasticsearchmachine · 2024-06-27T21:49:49Z

Hi @lkts, I've created a changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

kkrik-es · 2024-06-28T07:18:30Z

docs/reference/mapping/fields/synthetic-source.asciidoc

+
+[[synthetic-source-fields-native-list]]
+===== Field types that support synthetic source with no storage overhead
+Following field types support synthetic source using data from <<doc-values,`doc_values`>> and <<stored-fields, stored fields>> and require no additional storage space to construct `_source` field. Additional storage is required to store ignored field values produced as a result of using <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> parameters.


s/Following/The following/
s/construct _source field/construct the _source field/

kkrik-es · 2024-06-28T07:20:56Z

docs/reference/mapping/fields/synthetic-source.asciidoc

+
+
+[[synthetic-source-fields-native-list]]
+===== Field types that support synthetic source with no storage overhead


I'm on the fence on whether to report the fields that have additional indexing overhead when ignore_malformed or ignore_above are set. Wdyt?

I think it's fair, i imagine normally you expect malformed values to be a small portion of your data.

Well, the copying parser always copies the current value preemptively, iiuc? This should increase overhead for all indexed docs, not just the ones with malformed values..

I see what you mean. I don't know if it should be in this section but we can mention it somewhere. Let me follow up.

Let's wait with mentioning the copy parser overhead until we have a better understanding how much it actually is? I suspect it is minimal.

kkrik-es

We're getting there.

Shall we call out that object params are supported too? Esp disabled is an interesting one, as it increases storage but works as expected.

tylerperk

Looks good, just some tiny changes. Thanks

docs/reference/mapping/fields/synthetic-source.asciidoc

Co-authored-by: Tyler Perkins <[email protected]>

shainaraskas

approved with a couple of small additional suggestions!

docs/reference/mapping/fields/synthetic-source.asciidoc

shainaraskas · 2024-07-03T18:02:04Z

docs/reference/mapping/fields/synthetic-source.asciidoc

@@ -178,4 +153,38 @@ that ordering.

 [[synthetic-source-modifications-ranges]]
 ====== Representation of ranges
-Range field vales (e.g. `long_range`) are always represented as inclusive on both sides with bounds adjusted accordingly.  See <<range-synthetic-source-inclusive, examples>>.
+Range field vales (e.g. `long_range`) are always represented as inclusive on both sides with bounds adjusted accordingly. See <<range-synthetic-source-inclusive, examples>>.


should this say values?

docs/reference/mapping/fields/synthetic-source.asciidoc

Co-authored-by: shainaraskas <[email protected]>

elasticsearchmachine · 2024-07-04T09:16:44Z

Hi @lkts, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

This reverts commit 32e80d3.

martijnvg

LGTM

)

…110489)

Reflect latest changes in synthetic source documentation

ad8d7cc

lkts added >docs General docs changes :StorageEngine/Mapping The storage related side of mappings labels Jun 7, 2024

lkts requested review from martijnvg, salvatore-campagna and kkrik-es June 7, 2024 21:49

elasticsearchmachine added v8.15.0 Team:Docs Meta label for docs team Team:StorageEngine labels Jun 7, 2024

lkts commented Jun 7, 2024

View reviewed changes

kkrik-es reviewed Jun 8, 2024

View reviewed changes

martijnvg reviewed Jun 10, 2024

View reviewed changes

shainaraskas reviewed Jun 11, 2024

View reviewed changes

elasticmachine and others added 2 commits June 28, 2024 04:52

Merge branch 'main' into rework_synthetic_source_docs

d426e2f

iter

dd16e84

lkts requested review from shainaraskas, martijnvg and kkrik-es June 27, 2024 21:32

lkts added the release highlight label Jun 27, 2024

kkrik-es reviewed Jun 28, 2024

View reviewed changes

iter

ccd889c

tylerperk requested changes Jun 29, 2024

View reviewed changes

docs/reference/mapping/fields/synthetic-source.asciidoc Outdated Show resolved Hide resolved

kkrik-es approved these changes Jun 29, 2024

View reviewed changes

Update docs/reference/mapping/fields/synthetic-source.asciidoc

873bb35

Co-authored-by: Tyler Perkins <[email protected]>

lkts requested a review from tylerperk July 2, 2024 22:01

shainaraskas approved these changes Jul 3, 2024

View reviewed changes

lkts and others added 5 commits July 3, 2024 11:33

Update docs/reference/mapping/fields/synthetic-source.asciidoc

42af247

Co-authored-by: shainaraskas <[email protected]>

Update docs/reference/mapping/fields/synthetic-source.asciidoc

64822ca

Co-authored-by: shainaraskas <[email protected]>

Update docs/reference/mapping/fields/synthetic-source.asciidoc

4a234ae

Co-authored-by: shainaraskas <[email protected]>

Update docs/reference/mapping/fields/synthetic-source.asciidoc

18732e4

Co-authored-by: shainaraskas <[email protected]>

iter

a475445

elasticsearchmachine added v8.16.0 and removed v8.15.0 labels Jul 4, 2024

Update docs/changelog/109501.yaml

32e80d3

lkts added v8.15.0 and removed v8.16.0 labels Jul 4, 2024

Revert "Update docs/changelog/109501.yaml"

6e95674

This reverts commit 32e80d3.

salvatore-campagna approved these changes Jul 4, 2024

View reviewed changes

martijnvg approved these changes Jul 4, 2024

View reviewed changes

lkts merged commit 276ae12 into elastic:main Jul 4, 2024
5 checks passed

lkts deleted the rework_synthetic_source_docs branch July 4, 2024 16:48

lkts added the auto-backport Automatically create backport pull requests when merged label Jul 4, 2024

lkts added a commit to lkts/elasticsearch that referenced this pull request Jul 4, 2024

Reflect latest changes in synthetic source documentation (elastic#109501

3939271

)

lkts mentioned this pull request Jul 4, 2024

[8.15] Reflect latest changes in synthetic source documentation (#109501) #110489

Merged

lkts added a commit that referenced this pull request Jul 4, 2024

Reflect latest changes in synthetic source documentation (#109501) (#…

e666865

…110489)

lkts mentioned this pull request Aug 13, 2024

Fix references to logsdb index mode in release highlights lkts/elasticsearch#1

Closed


		There are a couple of restrictions to be aware of:
		<<synthetic-source-fields-native-list, Majority of field types>> natively support synthetic `_source`. Source for the field is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning no additional space is used.

	subject to <<synthetic-source-modifications,modifications>> applied to `_source` field.
	subject to <<synthetic-source-modifications,modifications>> applied to the `_source` field.

		@@ -7,8 +7,7 @@ searchable), but it is stored so that it can be returned when executing
		_fetch_ requests, like <<docs-get,get>> or <<search-search,search>>.

		If disk usage is important to you then have a look at

	If disk usage is important to you then have a look at
	If disk usage is important to you, then consider the following options:

	<<synthetic-source-fields-native-list, Majority of field types>> natively support synthetic `_source`. Source for the field is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning no additional space is used.
	<<synthetic-source-fields-native-list,Most field types>> natively support synthetic `_source`. The source for natively supported fields is constructed using existing data, most commonly <<doc-values,`doc_values`>>, meaning that no additional space is used.



		[[synthetic-source-fields-native-list]]
		===== Field types that support synthetic source with no storage overhead

Reflect latest changes in synthetic source documentation #109501

Reflect latest changes in synthetic source documentation #109501

Conversation

lkts commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

elasticsearchmachine commented Jun 7, 2024

elasticsearchmachine commented Jun 7, 2024

lkts Jun 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lkts commented Jun 7, 2024

lkts commented Jun 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkrik-es commented Jun 8, 2024

kkrik-es commented Jun 8, 2024

kkrik-es commented Jun 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lkts commented Jun 10, 2024

shainaraskas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shainaraskas Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lkts commented Jun 27, 2024

elasticsearchmachine commented Jun 27, 2024

kkrik-es Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkrik-es Jun 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkrik-es left a comment

Choose a reason for hiding this comment

tylerperk left a comment

Choose a reason for hiding this comment

shainaraskas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Jul 4, 2024

martijnvg left a comment

Choose a reason for hiding this comment

lkts Jun 7, 2024 •

edited

Loading

shainaraskas Jun 11, 2024 •

edited

Loading

kkrik-es Jun 28, 2024 •

edited

Loading

kkrik-es Jun 29, 2024 •

edited

Loading