Bump org.apache.druid:druid-processing from 30.0.1 to 31.0.0

[dependabot]

Bumps org.apache.druid:druid-processing from 30.0.1 to 31.0.0.

Release notes

Sourced from org.apache.druid:druid-processing's releases.

Druid 31.0.0

Apache Druid 31.0.0 contains over 589 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 64 contributors.

See the complete set of changes for additional details, including bug fixes.

Review the upgrade notes and incompatible changes before you upgrade to Druid 31.0.0. If you are upgrading across multiple versions, see the Upgrade notes page, which lists upgrade notes for the most recent Druid versions.

# Important features, changes, and deprecations

This section contains important information about new and existing features.

# Compaction features

Druid now supports the following features:

• Compaction scheduler with greater flexibility and control over when and what to compact.
• MSQ task engine-based auto-compaction for more performant compaction jobs.

For more information, see Compaction supervisors.

#16291 #16768

Additionally, compaction tasks that take advantage of concurrent append and replace is now generally available as part of concurrent append and replace becoming GA.

# Window functions are GA

Window functions are now generally available in Druid's native engine and in the MSQ task engine.

• You no longer need to use the query context enableWindowing to use window functions. #17087

# Concurrent append and replace GA

Concurrent append and replace is now GA. The feature safely replaces the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this feature is appending new data (such as with streaming ingestion) to an interval while compaction of that interval is already in progress.

# Delta Lake improvements

The community extension for Delta Lake has been improved to support complex types and snapshot versions.

# Iceberg improvements

The community extension for Iceberg has been improved. For more information, see Iceberg improvements

# Projections (experimental)

Druid 31.0.0 includes experimental support for new feature called projections. Projections are grouped pre-aggregates of a segment that are automatically used at query time to optimize execution for any queries which 'fit' the shape of the projection by reducing both computation and i/o cost by reducing the number of rows which need to be processed. Projections are contained within segments of a datasource and do increase the segment size. But they can share data, such as value dictionaries of dictionary encoded columns, with the columns of the base segment.

Projections currently only support JSON-based ingestion, but they can be used by queries that use the MSQ task engine or the new Dart engine. Future development will allow projections to be created as part of SQL-based ingestion.

We have a lot of plans to continue to improve this feature in the coming releases, but are excited to get it out there so users can begin experimentation since projections can dramatically improve query performance.

... (truncated)

Commits

• de96f9f [maven-release-plugin] prepare release druid-31.0.0-rc2
• 3ba7bad Backport "Fixes an issue with AppendableMemory that can cause MSQ jobs to fai...
• 26d5e95 [Backport]docs: backport msq autocompact docs #16681 (#17374)
• 9dac0f7 Fix pip installation after ubuntu upgrade (#17358)
• fd487d7 Revert "Use canonical hostname instead of ip by default (#16386)" (#17347)
• 2b98fac MSQ WorkerResource: Fix timeout handler for httpGetChannelData. (#17328) (#17...
• 1a7f91f add substituteCombiningFactory implementations for datasketches aggs (#17314)...
• 06c1a6a [Backport] Dart backports for (#17312) , (#17313) , (#17319) (#17322)
• f72dbfd [Backport] Window Functions : Improve performance by comparing Strings in fra...
• f2df70a BaseWorkerClientImpl: Don't attempt to recover from a closed channel. (#17052)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)