31.0.0.md

31.0.0 (2023-09-08)

Full Changelog

Breaking changes:

• Specialize Avg and Sum accumulators (#6842) #7358 (tustvold)
• Use datum arithmetic scalar value #7375 (tustvold)

Implemented enhancements:

• feat: array-empty #7313 (Weijun-H)
• Support REPLACE SQL alias syntax #7368 (berkaysynnada)
• feat: support primary key alternate syntax #7160 (parkma99)
• docs: Add Expr library developer page #7359 (tshauck)
• feat: support Binary for min/max #7397 (Weijun-H)
• feat: Add memory pool configuration to datafusion-cli #7424 (Weijun-H)
• Support Configuring Arrow RecordBatch Writers via SQL Statement Options #7390 (devinjdangelo)
• Add ROLLUP and GROUPING SETS substrait support #7382 (nseekhao)
• feat: Allow creating a ValuesExec from record batches #7444 (scsmithr)
• minor: Add ARROW to CREATE EXTERNAL TABLE docs and add example of COMPRESSION TYPE #7489 (alamb)
• Support Configuring Parquet Column Specific Options via SQL Statement Options #7466 (devinjdangelo)
• Write Multiple Parquet Files in Parallel #7483 (devinjdangelo)
• feat: explain with statistics #7459 (korowa)

Fixed bugs:

• fix: union_distinct shouldn't remove child distinct #7346 (jackwener)
• fix: inconsistent scalar types in DistinctArrayAggAccumulator state #7385 (korowa)
• fix: incorrect nullability calculation of InListExpr #7496 (jonahgao)

Merged pull requests:

• Remove redundant type check in Avg #7374 (viirya)
• feat: array-empty #7313 (Weijun-H)
• Minor: add WriteOp::name and DmlStatement::name #7329 (alamb)
• Specialize Median Accumulator #7376 (tustvold)
• Specialize Avg and Sum accumulators (#6842) #7358 (tustvold)
• Change error type of invalid argument to PlanError rather than InternalError, remove misleading comments #7355 (alamb)
• Implement array_pop_back function #7348 (tanruixiang)
• Add exec_err! error macro #7361 (comphead)
• Update sqlparser requirement from 0.36.1 to 0.37.0 #7387 (viirya)
• DML documentation #7362 (spaydar)
• Support REPLACE SQL alias syntax #7368 (berkaysynnada)
• Bug-fix/next_value() of Min/Max Scalar Values #7384 (berkaysynnada)
• Prepare 30.0.0 release #7372 (andygrove)
• fix: union_distinct shouldn't remove child distinct #7346 (jackwener)
• feat: support primary key alternate syntax #7160 (parkma99)
• Merge hash table implementations and remove leftover utilities #7366 (metesynnada)
• Minor: remove stray println! from array_expressions.rs #7389 (alamb)
• Projection Order Propagation #7364 (berkaysynnada)
• Document and scratch directory for sqllogictest and make test specific #7312 (alamb)
• Minor: Move test for Distribution and Partitioning with code #7392 (alamb)
• Minor: move datasource statistics code into its own module #7391 (alamb)
• Use datum arithmetic scalar value #7375 (tustvold)
• Fix IN expr for NaN #7378 (sarutak)
• Unnecessary to list all files during partition pruning #7395 (smallzhongfeng)
• Optimize Unnest and implement skip_nulls=true if specified #7371 (smiklos)
• Docs: Add query syntax to COPY docs #7388 (alamb)
• Clean up clippy for Rust 1.72 release #7399 (alamb)
• fix: inconsistent scalar types in DistinctArrayAggAccumulator state #7385 (korowa)
• Fix python CI #7416 (tustvold)
• docs: Add Expr library developer page #7359 (tshauck)
• Update ObjectStore 0.7.0 and Arrow 46.0.0 #7282 (tustvold)
• Fix Decimal256 scalar display string in sqllogictest #7404 (viirya)
• feat: support Binary for min/max #7397 (Weijun-H)
• Make sqllogictest distinguish NaN from -NaN #7403 (sarutak)
• Replace lazy_static with OnceLock #7409 (sarutak)
• Minor: Remove the unreached simplification rule for 0 / 0 #7405 (jonahgao)
• feat: Add memory pool configuration to datafusion-cli #7424 (Weijun-H)
• Minor: Debug log when FairPool is created #7431 (alamb)
• Support Configuring Arrow RecordBatch Writers via SQL Statement Options #7390 (devinjdangelo)
• Add ROLLUP and GROUPING SETS substrait support #7382 (nseekhao)
• Refactor sort_fuzz test to clarify what is covered #7439 (alamb)
• Use DateTime::from_naive_utc_and_offset instead of DateTime::from_utc #7451 (sarutak)
• Update substrait requirement from 0.12.0 to 0.13.1 #7443 (viirya)
• [minior fix]: adjust the projection statistics #7428 (liukun4515)
• Add new known users: Arroyo and Restate #7464 (jychen7)
• ScalarFunctionExpr Maintaining Order #7417 (berkaysynnada)
• Bug-fix/find_orderings_of_exprs #7457 (berkaysynnada)
• Update prost-derive requirement from 0.11 to 0.12 #7468 (dependabot[bot])
• Revert "Update prost-derive requirement from 0.11 to 0.12 (#7468)" #7476 (viirya)
• Return error if spill file does not exist in ExternalSorter #7479 (viirya)
• [minor fix]: Remove unused duplicate file_type.rs #7478 (sarutak)
• Minor: more flexible pool size setting for datafusion-cli #7454 (yjshen)
• Bump actions/checkout from 3 to 4 #7480 (dependabot[bot])
• Support Write Options in DataFrame::write_* methods #7435 (devinjdangelo)
• cp_solver, Duration vs Interval cases #7475 (berkaysynnada)
• feat: Allow creating a ValuesExec from record batches #7444 (scsmithr)
• Make LogicalPlan::with_new_exprs, deprecate from_plan #7396 (alamb)
• refactor: change file type logic for create table #7477 (tshauck)
• minor: do not fail analyzer if subquery plan contains extension #7455 (waynexia)
• Make IN expr work with multiple items #7449 (sarutak)
• Minor: Add doc comments and example for ScalarVaue::to_scalar #7491 (alamb)
• minor: Add ARROW to CREATE EXTERNAL TABLE docs and add example of COMPRESSION TYPE #7489 (alamb)
• Add backtrace to error messages #7434 (comphead)
• Make sqllogictest platform-independent for the sign of NaN #7462 (sarutak)
• Support Configuring Parquet Column Specific Options via SQL Statement Options #7466 (devinjdangelo)
• Minor: improve error message #7498 (alamb)
• Write Multiple Parquet Files in Parallel #7483 (devinjdangelo)
• PrimitiveGroupsAccumulator should propagate timestamp timezone information properly #7494 (sunchao)
• Minor: Add ScalarValue::data_type() for consistency with other APIs #7492 (alamb)
• feat: explain with statistics #7459 (korowa)
• fix: incorrect nullability calculation of InListExpr #7496 (jonahgao)