consolidate json and auto indexers, remove v4 nested column serializer #14456
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imply-cheddar
approved these changes
Jun 22, 2023
ektravel
reviewed
Jul 12, 2023
| @@ -23,12 +23,14 @@ sidebar_label: Nested columns | |||
| ~ under the License. | |||
| --> | |||
|
|
|||
| Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns. | |||
| Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types, as well as ARRAY of STRING, LONG, and DOUBLE values. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns. | |||
There was a problem hiding this comment.
Suggested change
| Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types, as well as ARRAY of STRING, LONG, and DOUBLE values. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns. | |
| Apache Druid supports directly storing nested data structures in `COMPLEX<json>` columns. `COMPLEX<json>` columns store a copy of the structured data in JSON format and specialized internal columns and indexes for nested literal values—STRING, LONG, and DOUBLE types, as well as ARRAY of STRING, LONG, and DOUBLE values. An optimized [virtual column](./virtual-columns.md#nested-field-virtual-column) allows Druid to read and filter these values at speeds consistent with standard Druid LONG, DOUBLE, and STRING columns. |
Escaping the <> is required for Docusaurus 2. Otherwise, it treats it as a JavaScript/HTML tag.
There was a problem hiding this comment.
even inside of backticks? that seems frustrating
There was a problem hiding this comment.
hmm, looking at #14411 and https://github.com/apache/druid/pull/14417/files#diff-1509d7fece1f79b5fc731c62d0fd4f114d48e1c606b3d23a7779e408df4590de i don't see changes to this page, and everything looks ok on the staged site https://druid.staged.apache.org/docs/latest/querying/nested-columns
There was a problem hiding this comment.
@317brian I thought escaping <> was requited for Docusaurus 2?
There was a problem hiding this comment.
@clintropolis I was wrong and we do not need to escape <> inside backticks.
ektravel
reviewed
Jul 12, 2023
docs/querying/nested-columns.md
Outdated
|
|
||
| Druid [SQL JSON functions](./sql-json-functions.md) allow you to extract, transform, and create `COMPLEX<json>` values in SQL queries, using the specialized virtual columns where appropriate. You can use the [JSON nested columns functions](math-expr.md#json-functions) in [native queries](./querying.md) using [expression virtual columns](./virtual-columns.md#expression-virtual-column), and in native ingestion with a [`transformSpec`](../ingestion/ingestion-spec.md#transformspec). | ||
|
|
||
| You can use the JSON functions in INSERT and REPLACE statements in SQL-based ingestion, or in a `transformSpec` in native ingestion as an alternative to using a [`flattenSpec`](../ingestion/data-formats.md#flattenspec) object to "flatten" nested data for ingestion. | ||
|
|
||
| Columns ingested as `COMPLEX<json>` are automatically optimized to store the most appropriate physical column based on the data processed. For example, if only LONG values are processed, Druid will store a LONG column, ARRAY columns if the data consists of arrays, or `COMPLEX<json>` in the general case if the data is actually nested. This is the same functionality that powers ['type aware' schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery). |
There was a problem hiding this comment.
Suggested change
| Columns ingested as `COMPLEX<json>` are automatically optimized to store the most appropriate physical column based on the data processed. For example, if only LONG values are processed, Druid will store a LONG column, ARRAY columns if the data consists of arrays, or `COMPLEX<json>` in the general case if the data is actually nested. This is the same functionality that powers ['type aware' schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery). | |
| Columns ingested as `COMPLEX<json>` are automatically optimized to store the most appropriate physical column based on the data processed. For example, if only LONG values are processed, Druid stores a LONG column, ARRAY columns if the data consists of arrays, or `COMPLEX<json>` in the general case if the data is actually nested. This is the same functionality that powers ['type aware' schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery). |
ektravel
approved these changes
Jul 27, 2023
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR consolidates 'json' and 'auto' column indexers, bring all the improvements of 'auto' to 'json'. 'auto' is the next generation (v5) of 'json' (v4) nested column format. However, 'json' was kept separate and on the previous version of the format for backwards compatibility during rolling upgrades from 25 to 26. Basically this allows us to dump a bunch of code, leaving only the v4 reader so that we can continue to read the older columns.
I will open a separate PR to handle future backwards compatibility concerns in order to degrade gracefully when new column format features are added to make what we had to do here less important.
Release note
'json' is now equivalent to using 'auto' in native ingestion dimension specs, upgrading 'json' to get all of the features and functionality of 'auto', such as type specializations including ARRAY typed columns, better support for nested arrays and smarter index utilization. However, this means that 'json' columns created with Druid 28 are not backwards compatible with Druid versions older than 26, so it is recommended to upgrade to Druid 26 prior to upgrading to Druid 28 if upgrading from an older version and using 'json' columns. Additionally, to downgrade to a version older than Druid 26, any new segments created in Druid 28 should be re-ingested using Druid 26 or 27 prior to further downgrading.
This PR has: