matt/feat/recursive ctes/config flag #3

matthewgapp · 2024-01-11T00:46:08Z

Minor: Improve TableProviderFilterPushDown docs (Minor: Improve TableProviderFilterPushDown docs apache/datafusion#7685)
FIX: Test timestamp with table (FIX: Test timestamp with table apache/datafusion#7701)
Fix bug in simplify expressions (Change SQL dialect to PostgreSQL apache/datafusion#214) (Fix bug in SimplifyExpressions apache/datafusion#7699)
Enhance Enforce Dist capabilities to fix, sub optimal bad plans (Enhance Enforce Dist capabilities to fix, sub optimal bad plans apache/datafusion#7671)
docs: add section on supports_filters_pushdown (docs: add section on supports_filters_pushdown apache/datafusion#7680)
Improve cache usage in CI (Improve cache usage in CI apache/datafusion#7678)
fix: substrait limit when fetch is None (fix: substrait limit when fetch is None apache/datafusion#7669)
minor: revert parsing precedence between Aggr and UDAF (minor: revert parsing precedence between Aggr and UDAF apache/datafusion#7682)
Minor: Move hash utils to common (Minor: Move hash utils to common apache/datafusion#7684)
Update Default Parquet Write Compression (Update Default Parquet Write Compression apache/datafusion#7692)
Stop using cache for the benchmark job (Stop using cache for the benchmark job apache/datafusion#7706)
Change rust.yml to run benchmark (Change rust.yml to run benchmark apache/datafusion#7708)
Extend infer_placeholder_types to support BETWEEN predicates (Extend infer_placeholder_types to support BETWEEN predicates apache/datafusion#7703)
Minor: Add comment explaining why verify benchmark results uses release mode (Minor: Add comment explaining why verify benchmark results uses release mode apache/datafusion#7712)
Support all the codecs supported by Avro (Support all the codecs supported by Avro apache/datafusion#7718)
Update substrait requirement from 0.14.0 to 0.15.0 (Update substrait requirement from 0.14.0 to 0.15.0 apache/datafusion#7719)
fix: coerce text to timestamps with timezones (fix: coerce text to timestamps with timezones apache/datafusion#7720)
Add LanceDB to the list of Known Users (Add LanceDB to the list of Known Users apache/datafusion#7716)
Enable avro reading/writing in datafusion-cli (Enable avro reading/writing in datafusion-cli apache/datafusion#7715)
Document crate feature flags (Document crate feature flags apache/datafusion#7713)
Minor: Consolidate UDF tests (Minor: Consolidate UDF tests apache/datafusion#7704)
Minor: fix CI failure due to Cargo.lock in cli (Minor: fix CI failure due to Cargo.lock in datafusioncli apache/datafusion#7733)
MINOR: change file to column index in page_filter trace log (MINOR: change file to column index in page_filter trace log apache/datafusion#7730)
preserve array type / timezone in date_bin and date_trunc functions (preserve array type / timezone in date_bin and date_trunc functions apache/datafusion#7729)
Remove redundant is_numeric for DataType (Remove redundant is_numeric for DataType apache/datafusion#7734)
fix: avro_to_arrow: Handle avro nested nullable struct (union) (fix: avro_to_arrow: Handle avro nested nullable struct (union) apache/datafusion#7663)
Rename SessionContext::with_config_rt to SessionContext::new_with_config_from_rt, etc (Rename SessionContext::with_config_rt to SessionContext::new_with_config_from_rt, etc apache/datafusion#7631)
Rename bounded_order_preserving_variants config to prefer_exising_sort and update docs (Rename bounded_order_preserving_variants config to prefer_exising_sort and update docs apache/datafusion#7723)
Optimize "ORDER BY + LIMIT" queries for speed / memory with special TopK operator (Optimize "ORDER BY + LIMIT" queries for speed / memory with special TopK operator apache/datafusion#7721)
Minor: Improve crate docs (Minor: Improve crate docs apache/datafusion#7740)
Resolve linter errors ([MINOR]: Resolve linter errors in the main apache/datafusion#7753)
rewrite concat_internal (Minor: Build concat_internal() with ListArray construction instead of ArrayData apache/datafusion#7748)
Minor: Add comment on input_schema from AggregateExec (Minor: Add comment on input_schema from AggregateExec apache/datafusion#7727)
Fix column name for COUNT(*) set by AggregateStatistics (Fix column name for COUNT(*) set by AggregateStatistics apache/datafusion#7757)
Add documentation about type signatures, and export TIMEZONE_WILDCARD (Add documentation about type signatures, and export TIMEZONE_WILDCARD apache/datafusion#7726)
[feat] Support cache ListFiles result cache in session level ([feat] Support cache ListFiles result cache in session level apache/datafusion#7620)
Support SHOW ALL VERBOSE to show settings description (Support SHOW ALL VERBOSE to show settings description apache/datafusion#7735)
Minor: Improve TableProvider document, and add ascii art (Minor: Improve TableProvider document, and add ascii art apache/datafusion#7759)
Prepare 32.0.0 Release (Prepare 32.0.0 Release apache/datafusion#7769)
Minor: Change all file links to GitHub in document (Minor: Change all file links to GitHub in document apache/datafusion#7768)
Minor: Improve PruningPredicate documentation (Minor: Improve PruningPredicate documentation apache/datafusion#7738)
Support InsertInto Sorted ListingTable (Support InsertInto Sorted ListingTable apache/datafusion#7743)
Minor: improve documentation to stagger_batch (Minor: improve documentation to stagger_batch apache/datafusion#7754)
External Table Primary key support (External Table Primary key support apache/datafusion#7755)
Minor: Build array_array() with ListArray construction instead of ArrayData (Minor: Build array_array() with ListArray construction instead of ArrayData apache/datafusion#7780)
Remove unnecessary cfg (Minor: Remove unnecessary #[cfg(feature = "avro")] apache/datafusion#7773)
add interval arithmetic for timestamp types (add interval arithmetic for timestamp types apache/datafusion#7758)
make tests deterministic (Minor: make tests deterministic apache/datafusion#7771)
Minor: Improve Interval Docs (Minor: Improve Interval Docs apache/datafusion#7782)
DataSink additions (DataSink additions apache/datafusion#7778)
Update substrait requirement from 0.15.0 to 0.16.0 (Update substrait requirement from 0.15.0 to 0.16.0 apache/datafusion#7783)
Move nested union optimization from plan builder to logical optimizer (Move nested union optimization from plan builder to logical optimizer apache/datafusion#7695)
Minor: comments that explain the schema used in simply_expressions (Minor: comments that explain the schema used in simply_expressions apache/datafusion#7747)
Update regex-syntax requirement from 0.7.1 to 0.8.0 (Update regex-syntax requirement from 0.7.1 to 0.8.0 apache/datafusion#7784)
Minor: Add sql test for UNION / UNION ALL + plans (Minor: Add sql test for UNION / UNION ALL + plans apache/datafusion#7787)
fix: preserve column qualifier for DataFrame::with_column (fix: preserve column qualifier for DataFrame::with_column apache/datafusion#7792)
Interval Arithmetic NegativeExpr Support (Interval Arithmetic NegativeExpr Support apache/datafusion#7804)
Exactness Indicator of Parameters: Precision (Exactness Indicator of Parameters: Precision apache/datafusion#7809)
add LogicalPlanBuilder::join_on (add LogicalPlanBuilder::join_on apache/datafusion#7805)
Fix window sort removal wrong operator. (Fix SortPreservingRepartition with no existing ordering. apache/datafusion#7811)
Update zstd requirement from 0.12 to 0.13 (Update zstd requirement from 0.12 to 0.13 apache/datafusion#7806)
[Minor]: Remove input_schema field from window executor ([Minor]: Remove input_schema field from window executor apache/datafusion#7810)
refactor(7181): move streaming_merge() into separate mod from the merge mod. (refactor(7181): move streaming_merge() into separate mod from the merge node apache/datafusion#7799)
Improve update error (Improve update error apache/datafusion#7777)
Minor: Update LogicalPlan::join_on API, use it more (Minor: Update LogicalPlan::join_on API, use it more apache/datafusion#7814)
Add distinct union optimization (Add distinct union optimization apache/datafusion#7788)
Make CI fail on any occurrence of rust-tomlfmt failed (Make CI fail on any occurrence of rust-tomlfmt failed apache/datafusion#7774)
Encode all join conditions in a single expression field (Encode all join conditions in a single expression field apache/datafusion#7612)
Update substrait requirement from 0.16.0 to 0.17.0 (Update substrait requirement from 0.16.0 to 0.17.0 apache/datafusion#7808)
Minor: include sort expressions in SortPreservingRepartitionExec explain plan (Minor: include sort expressions in SortPreservingRepartitionExec explain plan apache/datafusion#7796)
minor: add more document to Wildcard expr (minor: add more document to Wildcard expr apache/datafusion#7822)
Minor: Move Monotonicity to expr crate (Minor: Move Monotonicity to expr crate apache/datafusion#7820)
Use code block for better formatting of rustdoc for PhysicalGroupBy (Use code block for better formatting of rustdoc for PhysicalGroupBy apache/datafusion#7823)
Update explain plan to show TopK operator (Update explain plan to show TopK operator apache/datafusion#7826)
Extract ReceiverStreamBuilder (Extract ReceiverStreamBuilder apache/datafusion#7817)
Extend backtrace coverage for DatafusionError::Plan errors errors (Extend backtrace coverage for DatafusionError::Plan errors errors apache/datafusion#7803)
Add documentation and usability for prepared parameters (Add documentation and usability for prepared parameters apache/datafusion#7785)
Implement GetIndexedField for map-typed columns (Implement GetIndexedField for map-typed columns apache/datafusion#7825)
Minor: Assert streaming_merge has non empty sort exprs (Minor: Assert streaming_merge has non empty sort exprs apache/datafusion#7795)
Minor: Upgrade docs for PhysicalExpr::{propagate_constraints, evaluate_bounds} (Minor: Upgrade docs for PhysicalExpr::{propagate_constraints, evaluate_bounds} apache/datafusion#7812)
Change ScalarValue::List to store ArrayRef (Change ScalarValue::List to store ArrayRef apache/datafusion#7629)
[MINOR]:Do not introduce unnecessary repartition when row count is 1. ([MINOR]:Do not introduce unnecessary repartition when row count is 1. apache/datafusion#7832)
Minor: Add tests for binary / utf8 coercion (Minor: Add tests for binary / utf8 coercion apache/datafusion#7839)
Avoid panics on error while encoding/decoding ListValue::Array as protobuf (Avoid panics on error while encoding/decoding ListValue::Array as protobuf apache/datafusion#7837)
Refactor Statistics, introduce precision estimates (Exact, Inexact, Absent) (Refactor Statistics, introduce precision estimates (Exact, Inexact, Absent) apache/datafusion#7793)
Refactor ScalarValue::new_primitive to return Result (Remove redundant unwrap in ScalarValue::new_primitive, return a Result apache/datafusion#7830)
Fix precision loss when coercing date_part utf8 argument (Fix precision loss when coercing date_part utf8 argument apache/datafusion#7846)
Add operator section to user guide, Add std::ops operations to prelude, and add not() expr_fn (Add operator section to user guide, Add std::ops operations to prelude, and add not() expr_fn apache/datafusion#7732)
Expose arrow-schema serde crate feature flag (Expose arrow-schema serde crate feature flag apache/datafusion#7829)
Improve ContextProvider naming: rename get_table_provider --> get_table_source, deprecate get_table_provider (Improve ContextProvider naming: rename get_table_provider --> get_table_source, deprecate get_table_provider apache/datafusion#7831)
DataSink Dynamic Execution Time Demux (DataSink Dynamic Execution Time Demux apache/datafusion#7791)
Add small column on empty projection (Add small column on empty projection apache/datafusion#7833)
feat(7849): coerce TIMESTAMP to TIMESTAMPTZ (feat(7849): coerce TIMESTAMP to TIMESTAMPTZ apache/datafusion#7850)
Support binary argumnets (via coercion) for like and ilike and string functions (Support Binary/LargeBinary --> Utf8/LargeUtf8 in ilike and string functions apache/datafusion#7840)
fix typo (Minor: fix typo in comments apache/datafusion#7856)
Minor: improve join / join_on docs (Minor: improve join / join_on docs apache/datafusion#7813)
Support Decimal256 on AVG aggregate expression (Support Decimal256 on AVG aggregate expression apache/datafusion#7853)
Minor: fix typo in comments (Minor: fix typo in comments apache/datafusion#7861)
Minor: fix typo in GreedyMemoryPool documentation (Minor: fix typo in GreedyMemoryPool documentation apache/datafusion#7864)
Minor: fix multiple typos (Minor: fix multiple typos apache/datafusion#7863)
Minor: Fix docstring typos (Minor: Fix docstring typos apache/datafusion#7873)
Add CursorValues Decoupling Cursor Data from Cursor Position (Add CursorValues Decoupling Cursor Data from Cursor Position apache/datafusion#7855)
Support Decimal256 column in create external table (Support Decimal256 column in create external table apache/datafusion#7866)
Support Decimal256 in Min/Max aggregate expressions (Support Decimal256 in Min/Max aggregate expressions apache/datafusion#7881)
Implement Hive-Style Partitioned Write Support (Implement Hive-Style Partitioned Write Support apache/datafusion#7801)
Minor: fix config typo (Minor: fix config typo apache/datafusion#7874)
Add Decimal256 sqllogictests for SUM, MEDIAN and COUNT aggregate expressions (Add Decimal256 sqllogictests for SUM, MEDIAN and COUNT aggregate expressions apache/datafusion#7889)
[test] add fuzz test for topk ([test] add fuzz test for topk apache/datafusion#7772)
Allow Setting Minimum Parallelism with RowCount Based Demuxer (Allow Setting Minimum Parallelism with RowCount Based Demuxer apache/datafusion#7841)
Drop single quotes to make warnings for parquet options not confusing (Drop single quotes to make warnings for parquet options not confusing apache/datafusion#7902)
Add multi-column topk fuzz tests (Add multi-column topk fuzz tests apache/datafusion#7898)
Change FileScanConfig.table_partition_cols from (String, DataType) to Fields (Change FileScanConfig.table_partition_cols from (String, DataType) to Fields apache/datafusion#7890)
Maintain time_zone in new_list (Maintain time zone in ScalarValue::new_list apache/datafusion#7899)
[MINOR]: Move joinside struct to common ([MINOR]: Move joinside struct to common apache/datafusion#7908)
fix ExecutionContext to SessionContext (doc: fix ExecutionContext to SessionContext in custom-table-providers.md apache/datafusion#7903)
Update arrow 48.0.0 (Update arrow 48.0.0 apache/datafusion#7854)
feat: support Decimal256 for the abs function (feat: support Decimal256 for the abs function apache/datafusion#7904)
Simplifications ([MINOR] Simplify Aggregate, and Projection output_partitioning implementation apache/datafusion#7907)
Bump actions/setup-node from 3 to 4 (Bump actions/setup-node from 3 to 4 apache/datafusion#7915)
Fix bug, first last reverse ([Bug Fix]: Fix bug, first last reverse apache/datafusion#7914)
Minor: provide default implementation for ExecutionPlan::statistics (Minor: provide default implementation for ExecutionPlan::statistics apache/datafusion#7911)
Update substrait requirement from 0.17.0 to 0.18.0 (Update substrait requirement from 0.17.0 to 0.18.0 apache/datafusion#7916)
Remove unnecessary clone in datafusion_proto (Minor: Remove unnecessary clone in datafusion_proto apache/datafusion#7921)
[MINOR]: Simplify code, change requirement from PhysicalSortExpr to PhysicalSortRequirement ([MINOR]: Simplify code, change requirement from PhysicalSortExpr to PhysicalSortRequirement apache/datafusion#7913)
[Minor] Move combine_join util to under equivalence.rs ([Minor] Move combine_join util to under equivalence.rs apache/datafusion#7917)
support scan empty projection (support scan empty projection apache/datafusion#7920)
Cleanup logical optimizer rules. (Cleanup logical optimizer rules. apache/datafusion#7919)
Parallelize Serialization of Columns within Parquet RowGroups (Parallelize Serialization of Columns within Parquet RowGroups apache/datafusion#7655)
feat: Use bloom filter when reading parquet to skip row groups (feat: Use bloom filter when reading parquet to skip row groups apache/datafusion#7821)
fix: don't push down volatile predicates in projection (fix: don't push down volatile predicates in projection apache/datafusion#7909)
Add parquet feature flag, enabled by default, and make parquet conditional (Add parquet feature flag, enabled by default, and make parquet conditional apache/datafusion#7745)
[MINOR]: Simplify enforce_distribution, minor changes ([MINOR]: Simplify enforce_distribution, minor changes apache/datafusion#7924)
Add simple window query to sqllogictest (Add simple window query to sqllogictest apache/datafusion#7928)
ci: upgrade node to version 20 (ci: upgrade node to version 20 apache/datafusion#7918)
Change input for to_timestamp function to be seconds rather than nanoseconds, add to_timestamp_nanos (Change input for to_timestamp function to be seconds rather than nanoseconds, add to_timestamp_nanos apache/datafusion#7844)
Minor: Document parquet crate feature (Minor: Document parquet crate feature apache/datafusion#7927)
Minor: reduce some #cfg(feature = "parquet") (Minor: reduce some #cfg(feature = "parquet") apache/datafusion#7929)
Minor: reduce use of cfg(parquet) in tests (Minor: reduce use of #cfg(feature = "parquet") in tests apache/datafusion#7930)
Fix CI failures on to_timestamp() calls (Fix CI failures on to_timestamp() calls apache/datafusion#7941)
minor: add a datatype casting for the updated value (minor: add a datatype casting for the updated value apache/datafusion#7922)
fix (Minor：add avro feature in datafusion-examples to make avro_sql run apache/datafusion#7946)
Add simple exclude all columns test to sqllogictest (Add simple exclude all columns test to sqllogictest apache/datafusion#7945)
Support Partitioning Data by Dictionary Encoded String Array Types (Support Partitioning Data by Dictionary Encoded String Array Types apache/datafusion#7896)
Minor: Remove array() in array_expression (Minor: Remove array() in array_expression apache/datafusion#7961)
Minor: simplify update code (Minor: simplify update code apache/datafusion#7943)
Add some initial content about creating logical plans (Add some initial content about creating logical plans apache/datafusion#7952)
Minor: Change from &mut SessionContext to &SessionContext in substrait (Minor: Change from &mut SessionContext to &SessionContext in substrait apache/datafusion#7965)
Fix crate READMEs (Fix crate READMEs apache/datafusion#7964)
Minor: Improve HashJoinExec documentation (Minor: Improve HashJoinExec documentation apache/datafusion#7953)
chore: clean useless clone baesd on clippy (chore: clean useless clone baesd on clippy apache/datafusion#7973)
Add README.md to core, execution and physical-plan crates (Add README.md to core, execution and physical-plan crates apache/datafusion#7970)
Move source repartitioning into ExecutionPlan::repartition (Move source repartitioning into ExecutionPlan::repartition apache/datafusion#7936)
minor: fix broken links in README.md (minor: fix broken links in README.md apache/datafusion#7986)
Minor: Upate the sqllogictest crate README (Minor: Upate the sqllogictest crate README apache/datafusion#7971)
Improve MemoryCatalogProvider default impl block placement (Improve MemoryCatalogProvider default impl block placement apache/datafusion#7975)
Fix ScalarValue handling of NULL values for ListArray (Fix ScalarValue handling of NULL values for ListArray apache/datafusion#7969)
Refactor of Ordering and Prunability Traversals and States (Refactor of Ordering and Prunability Traversals and States apache/datafusion#7985)
Keep output as scalar for scalar function if all inputs are scalar (Keep output as scalar for scalar function if all inputs are scalar apache/datafusion#7967)
Fix crate READMEs for core, execution, physical-plan (Fix crate READMEs for core, execution, physical-plan apache/datafusion#7990)
Update sqlparser requirement from 0.38.0 to 0.39.0 (Update sqlparser requirement from 0.38.0 to 0.39.0 apache/datafusion#7983)
Fix panic in multiple distinct aggregates by fixing ScalarValue::new_list (Fix panic in multiple distinct aggregates by fixing ScalarValue::new_list apache/datafusion#7989)
MemoryReservation exposes MemoryConsumer (Minor: Add MemoryReservation::consumer getter apache/datafusion#8000)
fix: generate logical plan for UPDATE SET FROM statement (fix: generate logical plan for UPDATE SET FROM statement apache/datafusion#7984)
Create temporary files for reading or writing (Create temporary files for reading or writing apache/datafusion#8005)
doc: minor fix to SortExec::with_fetch comment (Minor: fix comment on SortExec::with_fetch method apache/datafusion#8011)
Fix: dataframe_subquery example Optimizer rule common_sub_expression_eliminate failed (Fix: dataframe_subquery example Optimizer rule common_sub_expression_eliminate failed apache/datafusion#8016)
Percent Decode URL Paths (ListingTableUrl Inconsistent Percent Encoding apache/datafusion#8009) (Percent Decode URL Paths (#8009) apache/datafusion#8012)
Minor: Extract common deps into workspace (Minor: Extract common deps into workspace apache/datafusion#7982)
minor: change some plan_err to exec_err (minor: change some plan_err to exec_err apache/datafusion#7996)
Minor: error on unsupported RESPECT NULLs syntax (Minor: error on unsupported RESPECT NULLs syntax apache/datafusion#7998)
GroupedHashAggregateStream breaks spill batch (Break GroupedHashAggregateStream spill batch into smaller chunks apache/datafusion#8004)
Minor: Add implementation examples to ExecutionPlan::execute (Minor: Add implementation examples to ExecutionPlan::execute apache/datafusion#8013)
address comment (Minor: Extend wrap_into_list_array to accept multiple args apache/datafusion#7993)
GroupedHashAggregateStream should register spillable consumer (GroupedHashAggregateStream should register spillable consumer apache/datafusion#8002)
fix: single_distinct_aggretation_to_group_by fail (fix: single_distinct_aggretation_to_group_by fail apache/datafusion#7997)
Read only enough bytes to infer Arrow IPC file schema via stream (Read only enough bytes to infer Arrow IPC file schema via stream apache/datafusion#7962)
Minor: remove a strange char (Minor: remove a strange char apache/datafusion#8030)
Minor: Improve documentation for Filter Pushdown (Minor: Improve documentation for Filter Pushdown apache/datafusion#8023)
Minor: Improve ExecutionPlan documentation (Minor: Improve ExecutionPlan documentation apache/datafusion#8019)
fix: clippy warnings from nightly rust 1.75 (fix: clippy warnings from nightly rust 1.75 apache/datafusion#8025)
Minor: Avoid recomputing compute_array_ndims in align_array_dimensions (Minor: Avoid recomputing compute_array_ndims in align_array_dimensions apache/datafusion#7963)
Minor: fix doc check (Minor: fix doc and fmt CI check apache/datafusion#8037)
Minor: remove uncessary #cfg test (Minor: remove uncessary #cfg test apache/datafusion#8036)
Minor: Improve documentation for PartitionStream and StreamingTableExec (Minor: Improve documentation for PartitionStream and StreamingTableExec apache/datafusion#8035)
Combine Equivalence and Ordering equivalence to simplify state (Combine Equivalence and Ordering equivalence to simplify state apache/datafusion#8006)
Encapsulate ProjectionMapping as a struct (Encapsulate ProjectionMapping as a struct apache/datafusion#8033)
Minor: Fix bugs in docs for to_timestamp, to_timestamp_seconds, ... (Minor: Fix bugs in docs for to_timestamp, to_timestamp_seconds, ... apache/datafusion#8040)
Improve comments for PartitionSearchMode struct (Improve comments for PartitionSearchMode struct apache/datafusion#8047)
General approach for Array replace (General approach for Array replace apache/datafusion#8050)
Minor: Remove the irrelevant note from the Expression API doc (Minor: Remove the irrelevant note from the Expression API doc apache/datafusion#8053)
Minor: Add more documentation about Partitioning (Minor: Add more documentation about Partitioning apache/datafusion#8022)
Minor: improve documentation for IsNotNull, DISTINCT, etc (Minor: improve documentation for IsNotNull, DISTINCT, etc apache/datafusion#8052)
Prepare 33.0.0 Release (Prepare 33.0.0 Release apache/datafusion#8057)
Minor: improve error message by adding types to message (Minor: improve error message by adding types to message apache/datafusion#8065)
Minor: Remove redundant BuiltinScalarFunction::supports_zero_argument() (Minor: Remove redundant BuiltinScalarFunction::supports_zero_argument() apache/datafusion#8059)
Add example to ci (Add example to ci apache/datafusion#8060)
Update substrait requirement from 0.18.0 to 0.19.0 (Update substrait requirement from 0.18.0 to 0.19.0 apache/datafusion#8076)
Fix incorrect results in COUNT(*) queries with LIMIT (Fix incorrect results in COUNT(*) queries with LIMIT apache/datafusion#8049)
feat: Support determining extensions from names like foo.parquet.snappy as well as foo.parquet (feat: Support determining extensions from names like foo.parquet.snappy as well as foo.parquet apache/datafusion#7972)
Use FairSpillPool for TaskContext with spillable config (Use FairSpillPool for TaskContext with spillable config apache/datafusion#8072)
Minor: Improve HashJoinStream docstrings (Minor: Improve HashJoinStream docstrings apache/datafusion#8070)
Fixing broken link (Fixing broken link apache/datafusion#8085)
fix: DataFusion suggests invalid functions (fix: DataFusion suggests invalid functions apache/datafusion#8083)
Replace macro with function for array_repeat (Replace macro with function for array_repeat apache/datafusion#8071)
Minor: remove unnecessary projection in single_distinct_to_group_by rule (Minor: remove unnecessary projection in single_distinct_to_group_by rule apache/datafusion#8061)
minor: Remove duplicate version numbers for arrow, object_store, and parquet dependencies (minor: Remove duplicate version numbers for arrow, object_store, and parquet dependencies apache/datafusion#8095)
fix: add match encode/decode scalar function type (fix: add encode/decode to protobuf encoding apache/datafusion#8089)
feat: Protobuf serde for Json file sink (feat: Protobuf serde for Json file sink apache/datafusion#8062)
Minor: use Expr::alias in a few places to make the code more concise (Minor: use Expr::alias in a few places to make the code more concise apache/datafusion#8097)
Minor: Cleanup BuiltinScalarFunction::return_type() (Minor: Cleanup BuiltinScalarFunction::return_type() apache/datafusion#8088)
Update sqllogictest requirement from 0.17.0 to 0.18.0 (Update sqllogictest requirement from 0.17.0 to 0.18.0 apache/datafusion#8102)
Projection Pushdown rule and test changes (Projection Pushdown in PhysicalPlan apache/datafusion#8073)
Push limit into aggregation for DISTINCT ... LIMIT queries (Push limit into aggregation for DISTINCT ... LIMIT queries apache/datafusion#8038)
Bug-fix in Filter and Limit statistics (Bug-fix in Filter and Limit statistics apache/datafusion#8094)
feat: support target table alias in update statement (feat: support target table alias in update statement apache/datafusion#8080)
Simlify downcast functions in cast.rs. (Minor: Simlify downcast functions in cast.rs. apache/datafusion#8103)
Fix ArrayAgg schema mismatch issue (Fix ArrayAgg schema mismatch issue apache/datafusion#8055)
Minor: Support nulls in array_replace, avoid a copy (Minor: Support nulls in array_replace, avoid a copy apache/datafusion#8054)
Minor: Improve the document format of JoinHashMap (Minor: Improve the document format of JoinHashMap apache/datafusion#8090)
Simplify ProjectionPushdown and make it more general (Simplify ProjectionPushdown and make it more general apache/datafusion#8109)
Minor: clean up the code regarding clippy (Minor: clean up the code regarding clippy apache/datafusion#8122)
Support remaining functions in protobuf serialization, add expr_fn for StructFunction (Support remaining functions in protobuf serialization, add expr_fn for StructFunction apache/datafusion#8100)
cleanup scalar function impl (Minor: Cleanup BuiltinScalarFunction's phys-expr creation apache/datafusion#8114)
rewrite array_append/array_prepend to remove deplicate codes (rewrite array_append/array_prepend to remove deplicate codes apache/datafusion#8108)
Implementation of array_intersect (Implementation of array_intersect apache/datafusion#8081)
fix build by adding ScalarFunction::ArrayIntersect match (Minor: fix ci break apache/datafusion#8136)
Improve documentation for calculate_prune_length method in SymmetricHashJoin (Improve documentation for calculate_prune_length method in SymmetricHashJoin apache/datafusion#8125)
Minor: remove duplicated array_replace tests (Minor: remove duplicated array_replace tests apache/datafusion#8066)
use tmp directory in test (Minor: Fix temporary files created but not deleted during testing apache/datafusion#8115)
chore: remove panics in datafusion-common::scalar (chore: remove panics in datafusion-common::scalar by making more operations return Result apache/datafusion#7901)
Fix join order for TPCH Q17 & Q18 by improving FilterExec statistics (Fix join order for TPCH Q17 & Q18 by improving FilterExec statistics apache/datafusion#8126)
Fix: Do not try and preserve order when there is no order to preserve in RepartitionExec (Fix: Do not try and preserve order when there is no order to preserve in RepartitionExec apache/datafusion#8127)
feat: add column statistics into explain (feat: add column statistics into explain apache/datafusion#8112)
Add subtrait support for IS NULL and IS NOT NULL (Add subtrait support for IS NULL and IS NOT NULL apache/datafusion#8093)
Combine Wildcard and QualifiedWildcard, add wildcard() expr fn (Combine Expr::Wildcard and Wxpr::QualifiedWildcard, add wildcard() expr fn apache/datafusion#8105)
docs: show creation of DFSchema (docs: show creation of DFSchema apache/datafusion#8132)
feat: support UDAF in substrait producer/consumer (feat: support UDAF in substrait producer/consumer apache/datafusion#8119)
Improve documentation site to make it easier to find communication on Slack/Discord (Improve documentation site to make it easier to find communication on Slack/Discord apache/datafusion#8138)
Fix typo in partitioning.rs (Fix typo in partitioning.rs apache/datafusion#8134)
Implement DISTINCT ON from Postgres (Implement DISTINCT ON from Postgres apache/datafusion#7981)
Prepare 33.0.0-rc2 (Prepare 33.0.0-rc2 apache/datafusion#8144)
Avoid concat in array_append (Avoid concat in array_append apache/datafusion#8137)
Replace macro with function for array_remove (Replace macro with function for array_remove apache/datafusion#8106)
Implement array_union (Implement array_union apache/datafusion#7897)
Minor: Document ExecutionPlan::equivalence_properties more thoroughly (Minor: Document ExecutionPlan::equivalence_properties more thoroughly apache/datafusion#8128)
feat: show statistics in explain verbose (feat: show statistics in explain verbose apache/datafusion#8113)
feat:implement postgres style 'overlay' string function (feat:implement postgres style 'overlay' string function apache/datafusion#8117)
Minor: Encapsulate LeftJoinData into a struct (rather than anonymous enum) and add comments (Minor: Encapsulate LeftJoinData into a struct (rather than anonymous enum) and add comments apache/datafusion#8153)
Update sqllogictest requirement from 0.18.0 to 0.19.0 (Update sqllogictest requirement from 0.18.0 to 0.19.0 apache/datafusion#8163)
feat: fill missing values with NULLs while inserting (feat: fill missing values with NULLs while inserting apache/datafusion#8146)
Introduce return type for aggregate sum (Introduce return type for aggregate sum apache/datafusion#8141)
implement range/generate_series func (implement range/generate_series func apache/datafusion#8140)
Encapsulate EquivalenceClass into a struct (Encapsulate EquivalenceClass into a struct apache/datafusion#8034)
Revert "Minor: remove unnecessary projection in `single_distinct_to_g… (Revert "Minor: remove unnecessary projection in `single_distinct_to_g… apache/datafusion#8176)
Preserve all of the valid orderings during merging. (Preserve all of the valid orderings during merging. apache/datafusion#8169)
Make fields of ScalarUDF , AggregateUDF and WindowUDF non pub (Make fields of ScalarUDF , AggregateUDF and WindowUDF non pub apache/datafusion#8079)
Fix logical conflicts (Fix logical conflicts apache/datafusion#8187)
Minor: Update JoinHashMap example to make it clearer (Minor: Update JoinHashMap comment example to make it clearer apache/datafusion#8154)
Implement StreamTable and StreamTableProvider (Decouple Streaming Use-Case from File IO Abstractions apache/datafusion#7994) (Implement StreamTable and StreamTableProvider (#7994) apache/datafusion#8021)
Remove unused Results ([MINOR]: Remove unused Results apache/datafusion#8189)
Minor: clean up the code based on clippy (Minor: clean up the code based on clippy apache/datafusion#8179)
Minor: simplify filter statistics code (Minor: simplify filter statistics code apache/datafusion#8174)
Replace macro with function for array_position and array_positions (Replace macro with function for array_position and array_positions apache/datafusion#8170)
Add Library Guide for User Defined Functions: Window/Aggregate ( Add Library Guide for User Defined Functions: Window/Aggregate apache/datafusion#8171)
Add more stream docs (Add more stream docs apache/datafusion#8192)
Implement func array_pop_front (Implement func array_pop_front apache/datafusion#8142)
Moving arrow_files SQL tests to sqllogictest (Moving arrow_files SQL tests to sqllogictest apache/datafusion#8217)
fix use of name in Column (fix regression in the use of name in ProjectionPushdown apache/datafusion#8219)
Fix column indices in the planning tests ([MINOR]: Fix column indices in the planning tests apache/datafusion#8191)
Remove unnecessary reassignment (Remove unnecessary reassignment apache/datafusion#8232)
Update itertools requirement from 0.11 to 0.12 (Update itertools requirement from 0.11 to 0.12 apache/datafusion#8233)
Port tests in subqueries.rs to sqllogictest (Port tests in subqueries.rs to sqllogictest apache/datafusion#8231)
feat: make FSL scalar also an arrayref (feat: make FixedSizeList scalar also an ArrayRef apache/datafusion#8221)
Add versions to datafusion dependencies (Add versions to datafusion dependencies apache/datafusion#8238)
feat: support eq_array and to_array_of_size for FSL (feat: to_array_of_size for ScalarValue::FixedSizeList apache/datafusion#8225)
feat:implement calcite style 'levenshtein' string function (feat:implement calcite style 'levenshtein' string function apache/datafusion#8168)
feat: roundtrip FixedSizeList Scalar to protobuf (feat: roundtrip FixedSizeList Scalar to protobuf apache/datafusion#8239)
Update prost-build requirement from =0.12.1 to =0.12.2 (Update prost-build requirement from =0.12.1 to =0.12.2 apache/datafusion#8244)
Minor: Port tests in displayable.rs to sqllogictest (Minor: Port tests in displayable.rs to sqllogictest apache/datafusion#8246)
Minor: add with_estimated_selectivity to Precision (Minor: add with_estimated_selectivity to Precision apache/datafusion#8177)
fix: Timestamp with timezone not considered join on (fix: Timestamp with timezone not considered join on apache/datafusion#8150)
Replace macro in array_array to remove duplicate codes (Replace macro in array_array to remove duplicate codes apache/datafusion#8252)
Port tests in projection.rs to sqllogictest (Port tests in projection.rs to sqllogictest apache/datafusion#8240)
Introduce array_except function (Introduce array_except function apache/datafusion#8135)
Port tests in describe.rs to sqllogictest (Port tests in describe.rs to sqllogictest apache/datafusion#8242)
Remove FileWriterMode and ListingTableInsertMode (Decouple Streaming Use-Case from File IO Abstractions apache/datafusion#7994) (Remove FileWriterMode and ListingTableInsertMode (#7994) apache/datafusion#8017)
Minor: clean up the code based on Clippy (Minor: clean up the code based on Clippy apache/datafusion#8257)
Update arrow 49.0.0 and object_store 0.8.0 (Update arrow 49.0.0 and object_store 0.8.0 apache/datafusion#8029)
feat: impl the basic string_agg function (feat: impl the basic string_agg function apache/datafusion#8148)
Minor: Make schema of grouping set columns nullable (Minor: Make schema of grouping set columns nullable apache/datafusion#8248)
feat: support arbitrary binaryexpr simplifications (feat: support simplifying BinaryExpr with arbitrary guarantees in GuaranteeRewriter apache/datafusion#8256)
Making stream joins extensible: A new Trait implementation for SHJ (Making stream joins extensible: A new Trait implementation for SHJ apache/datafusion#8234)
Don't canonicalize ListingTableUrl (Don't Canonicalize Filesystem Paths in ListingTableUrl / support new external tables for files that do not (yet) exist apache/datafusion#8014)
Minor: Add sql level test for inserting into non-existent directory (Minor: Add sql level test for inserting into non-existent directory apache/datafusion#8278)
Replace array_has/array_has_all/array_has_any macro to remove duplicate code (Replace array_has/array_has_all/array_has_any macro to remove duplicate code apache/datafusion#8263)
Fix bug in field level metadata matching code (Fix bug in field level metadata matching code apache/datafusion#8286)
Interval Arithmetic Updates (Refactor Interval Arithmetic Updates apache/datafusion#8276)
[MINOR]: Remove unecessary orderings from the final plan ([MINOR]: Remove unecessary orderings from the final plan apache/datafusion#8289)
consistent logical & physical NTILE return types (consistent logical & physical NTILE return types apache/datafusion#8270)
make array_union/array_except/array_intersect handle empty/null arrays rightly (make array_union/array_except/array_intersect handle empty/null arrays rightly apache/datafusion#8269)
improve file path validation when reading parquet (improve file path validation when reading parquet apache/datafusion#8267)
[Benchmarks] Make partitions default to number of cores instead of 2 ([Benchmarks] Make partitions default to number of cores instead of 2 apache/datafusion#8292)
Update prost-build requirement from =0.12.2 to =0.12.3 (Update prost-build requirement from =0.12.2 to =0.12.3 apache/datafusion#8298)
Fix Display for List (Fix Display for List apache/datafusion#8261)
feat: support customizing column default values for inserting (feat: support customizing column default values for inserting apache/datafusion#8283)
support LargeList for arrow_cast, support ScalarValue::LargeList (support LargeList for arrow_cast, support ScalarValue::LargeList apache/datafusion#8290)
Minor: remove useless clone based on Clippy (Minor: remove useless clone based on Clippy apache/datafusion#8300)
Calculate ordering equivalence for expressions (rather than just columns) (Calculate ordering equivalence for expressions (rather than just columns) apache/datafusion#8281)
Fix sqllogictests links in contributor-guide/index.md and README.md (Fix sqllogictests link in contributor-guide/index.md apache/datafusion#8314)
Refactor: Unify Expr::ScalarFunction and Expr::ScalarUDF, introduce unresolved functions by name (Refactor: Unify Expr::ScalarFunction and Expr::ScalarUDF, introduce unresolved functions by name apache/datafusion#8258)
Support no distinct aggregate sum/min/max in single_distinct_to_group_by rule (Support no distinct aggregate sum/min/max in single_distinct_to_group_by rule apache/datafusion#8266)
feat:implement sql style 'substr_index' string function (feat:implement sql style 'substr_index' string function apache/datafusion#8272)
Fixing issues with for timestamp literals (Fixing issues with for timestamp literals apache/datafusion#8193)
Projection Pushdown over StreamingTableExec (Projection Pushdown over StreamingTableExec apache/datafusion#8299)
minor: fix documentation (minor: fix documentation apache/datafusion#8323)
fix: wrong result of range function (fix: wrong result of range function apache/datafusion#8313)
Minor: rename parquet.rs to parquet/mod.rs (Minor: rename parquet.rs to parquet/mod.rs apache/datafusion#8301)
refactor: output ordering (refactor: output ordering apache/datafusion#8304)
Update substrait requirement from 0.19.0 to 0.20.0 (Update substrait requirement from 0.19.0 to 0.20.0 apache/datafusion#8339)
First iteration of aggregates.rs tests to sqllogictests (Port tests in aggregates.rs to sqllogictest apache/datafusion#8316)
Library Guide: Add Using the DataFrame API (Library Guide: Add Using the DataFrame API apache/datafusion#8319)
Port tests in limit.rs to sqllogictest (Port tests in limit.rs to sqllogictest apache/datafusion#8315)
move array function unit_tests to sqllogictest (move array function unit_tests to sqllogictest apache/datafusion#8332)
NTH_VALUE reverse support (NTH_VALUE reverse support apache/datafusion#8327)
Refactor optimize projections rule, combines (eliminate, merge, pushdown projections) (Optimize Projections during Logical Plan apache/datafusion#8340)
Move merge projections tests to under optimize projections ([MINOR]: Move merge projections tests to under optimize projections apache/datafusion#8352)
Add quote and escape attributes to create csv external table (Add quote and escape attributes to create csv external table apache/datafusion#8351)
Minor: Add DataFrame test (Minor: Add DataFrame test apache/datafusion#8341)
clean up the code based on Clippy (Minor: clean up the code based on Clippy apache/datafusion#8359)
Minor: Make it easier to work with Expr::ScalarFunction (Minor: Make it easier to work with Expr::ScalarFunction apache/datafusion#8350)
Move some datafusion-optimizer::utils down to datafusion-expr::utils (Minor: Move some datafusion-optimizer::utils down to datafusion-expr::utils apache/datafusion#8354)
Minor: Make BuiltInScalarFunction::alias a method (Minor: Make BuiltInScalarFunction::alias a method apache/datafusion#8349)
Extract parquet statistics to its own module, add tests (Extract parquet statistics to its own module, add tests apache/datafusion#8294)
feat:implement sql style 'find_in_set' string function (feat:implement sql style 'find_in_set' string function apache/datafusion#8328)
largeutf to temporal (Support LargeUtf8 to Temporal Coercion apache/datafusion#8357)
Refactor aggregate function handling (Refactor aggregate function handling apache/datafusion#8358)
Implement Aliases for ScalarUDF (Implement Aliases for ScalarUDF apache/datafusion#8360)
Minor: Remove uncessary name field in ScalarFunctionDefintion (Minor: Remove unnecessary name field in ScalarFunctionDefintion apache/datafusion#8365)
feat: support LargeList in array_empty (feat: support LargeList in array_empty apache/datafusion#8321)
Double type argument for to_timestamp function (Double type argument for to_timestamp function apache/datafusion#8159)
Support User Defined Table Function (Support User Defined Table Function apache/datafusion#8306)
Document timestamp input limits (Document timestamp input limits apache/datafusion#8369)
fix: make ntile work in some corner cases (fix: make ntile work in some corner cases apache/datafusion#8371)
Refactor array_union function to use a generic (Minor: Refactor array_union function to use a generic union_arrays function apache/datafusion#8381)
Refactor function argument handling in (Minor: Refactor function argument handling in ScalarFunctionDefinition apache/datafusion#8387)
Materialize dictionaries in group keys (Materialize Dictionaries in Group Keys apache/datafusion#7647) (Materialize dictionaries in group keys apache/datafusion#8291)
Rewrite array_ndims to fix List(Null) handling (Rewrite array_ndims to fix List(Null) handling apache/datafusion#8320)
Docs: Improve the documentation on ScalarValue (Docs: Improve the documentation on ScalarValue apache/datafusion#8378)
Avoid concat for array_replace (Avoid concat for array_replace apache/datafusion#8337)
add a summary table to benchmark compare output (add a summary table to benchmark compare output apache/datafusion#8399)
Refactors on TreeNode Implementations (Refactors on TreeNode Implementations apache/datafusion#8395)
feat: support LargeList in make_array and array_length (feat: support LargeList in make_array and array_length apache/datafusion#8121)
remove unalias() TableScan filters when create Physical Filter (remove unalias TableScan filters when create Physical Filter apache/datafusion#8404)
Update custom-table-providers.md (Update custom-table-providers.md apache/datafusion#8409)
fix transforming LogicalPlan::Explain use TreeNode::transform fails (fix transforming LogicalPlan::Explain use TreeNode::transform fails apache/datafusion#8400)
Docs: Fix array_except documentation example (Docs: Fix array_except documentation example error apache/datafusion#8407)
Support named query parameters (Support named query parameters apache/datafusion#8384)
Minor: Add installation link to README.md (Minor: Add installation link to README.md apache/datafusion#8389)
Update code comment for the cases of regularized RANGE frame and add tests for ORDER BY cases with RANGE frame (Update code comment for the cases of regularized RANGE frame and add tests for ORDER BY cases with RANGE frame apache/datafusion#8410)
Minor: Add example with parameters to LogicalPlan (Minor: Add example with parameters to LogicalPlan apache/datafusion#8418)
Minor: Improve PruningPredicate documentation (Minor: Improve PruningPredicate documentation apache/datafusion#8394)
feat: ScalarValue from String (feat: ScalarValue from String apache/datafusion#8411)
Bump actions/labeler from 4.3.0 to 5.0.0 (Bump actions/labeler from 4.3.0 to 5.0.0 apache/datafusion#8422)
Update sqlparser requirement from 0.39.0 to 0.40.0 (Update sqlparser requirement from 0.39.0 to 0.40.0 apache/datafusion#8338)
feat: support LargeList for array_has, array_has_all and array_has_any (feat: support LargeList for array_has, array_has_all and array_has_any apache/datafusion#8322)
Union schema can't be a subset of the child schema (Union schema can't be a subset of the child schema apache/datafusion#8408)
Move PartitionSearchMode into datafusion_physical_plan, rename to InputOrderMode (Move PartitionSearchMode into datafusion_physical_plan, rename to InputOrderMode apache/datafusion#8364)
Make filter selectivity for statistics configurable (Make filter selectivity for statistics configurable apache/datafusion#8243)
fix: Changed labeler.yml to latest format (fix: Changed labeler.yml to latest format apache/datafusion#8431)
Minor: Use ScalarValue::from impl for strings (Minor: Use ScalarValue::from impl for strings apache/datafusion#8429)
Support crossjoin in substrait. (Support crossjoin in substrait. apache/datafusion#8427)
Fix ambiguous reference when aliasing in combination with ORDER BY (Fix ambiguous reference when aliasing in combination with ORDER BY apache/datafusion#8425)
Minor: convert marcro list-slice and slice to function (Minor: convert marcro list-slice and slice to function apache/datafusion#8424)
Remove macro in iter_to_array for List (Remove macro in iter_to_array for List apache/datafusion#8414)
fix: Literal in ORDER BY window definition should not be an ordinal referring to relation column (fix: Literal in ORDER BY window definition should not be an ordinal referring to relation column apache/datafusion#8419)
feat: customize column default values for external tables (feat: customize column default values for external tables apache/datafusion#8415)
feat: Support array_sort(list_sort) (feat: Support array_sort(list_sort) apache/datafusion#8279)
Bugfix: Remove df-cli specific SQL statment options before executing with DataFusion (Bugfix: Remove df-cli specific SQL statment options before executing with DataFusion apache/datafusion#8426)
Detect when filters make subqueries scalar (Detect when filters on unique constraints make subqueries scalar apache/datafusion#8312)
Add alias check to optimize projections merge (Add alias check to optimize projections merge apache/datafusion#8438)
Fix PartialOrd for ScalarValue::List/FixSizeList/LargeList (Fix PartialOrd for ScalarValue::List/FixSizeList/LargeList apache/datafusion#8253)
Support parquet_metadata for datafusion-cli (Support parquet_metadata for datafusion-cli apache/datafusion#8413)
Fix bug in optimizing a nested count (Fix bug in optimizing a nested count apache/datafusion#8459)
Bump actions/setup-python from 4 to 5 (Bump actions/setup-python from 4 to 5 apache/datafusion#8449)
fix: ORDER BY window definition should work on null (fix: ORDER BY window definition should work on null literal apache/datafusion#8444)
flx clippy warnings (flx clippy warnings apache/datafusion#8455)
fix: RANGE frame for corner cases with empty ORDER BY clause should be treated as constant sort (fix: RANGE frame for corner cases with empty ORDER BY clause should be treated as constant sort apache/datafusion#8445)
Preserve dict_id on Field during serde roundtrip (Preserve dict_id on Field during serde roundtrip apache/datafusion#8457)
support inter leave node (feat: support InterleaveExecNode in the proto apache/datafusion#8460)
Not fail when window input is empty record batch ([BUG FIX]: Proper Empty Batch handling in window execution apache/datafusion#8466)
update cast (Minor: update cast apache/datafusion#8458)
fix: don't unifies projection if expr is non-trival (fix: don't unifies projection if expr is non-trival apache/datafusion#8454)
Minor: Add new bloom filter predicate tests (Minor: Add new bloom filter predicate tests apache/datafusion#8433)
Add PRIMARY KEY Aggregate support to dataframe API (Add PRIMARY KEY Aggregate support to dataframe API apache/datafusion#8356)
Minor: refactor data_trunc to reduce duplicated code (Minor: refactor data_trunc to reduce duplicated code apache/datafusion#8430)
Support array_distinct function. (Support array_distinct function. apache/datafusion#8268)
Add primary key support to stream table (Add primary key support to stream table apache/datafusion#8467)
Add evaluate_demo and range_analysis_demo to Expr examples (Add evaluate_demo and range_analysis_demo to Expr examples apache/datafusion#8377)
fix typo (Minor: fix function name typo apache/datafusion#8473)
Fix comment typo in table.rs: s/indentical/identical/ (Minor: Fix comment typo in table.rs: s/indentical/identical/ apache/datafusion#8469)
Remove define_array_slice and reuse array_slice for array_pop_front/back (Remove define_array_slice and reuse array_slice for array_pop_front/back apache/datafusion#8401)
Minor: refactor trim to clean up duplicated code (Minor: refactor trim to clean up duplicated code apache/datafusion#8434)
Split EmptyExec into PlaceholderRowExec (Split EmptyExec into PlaceholderRowExec apache/datafusion#8446)
Enable non-uniform field type for structs created in DataFusion (Enable non-uniform field type for structs created in DataFusion apache/datafusion#8463)
add multi ordering test case (Minor: Add multi ordering test for array agg order apache/datafusion#8439)
Sort filenames when reading parquet to ensure consistent schema (Sort filenames when reading parquet to ensure consistent schema apache/datafusion#6629)
Minor: Improve comments in EnforceDistribution tests (Minor: Improve comments in EnforceDistribution tests apache/datafusion#8474)
fix: support uppercase when parsing Interval (fix: support uppercase when parsing Interval apache/datafusion#8478)
Better Equivalence (ordering and exact equivalence) Propagation through ProjectionExec (Better Equivalence (ordering and exact equivalence) Propagation through ProjectionExec apache/datafusion#8484)
Add today alias for current_date (Add today alias for current_date apache/datafusion#8423)
remove useless clone (Minor: remove useless clone in array_expression apache/datafusion#8495)
fix: incorrect set preserve_partitioning in SortExec (fix: incorrect set preserve_partitioning in SortExec apache/datafusion#8485)
Explicitly mark parquet for tests in datafusion-common (Explicitly mark parquet for tests in datafusion-common apache/datafusion#8497)
Minor/Doc: Clarify DataFrame::write_table Documentation (Minor/Doc: Clarify DataFrame::write_table Documentation apache/datafusion#8519)
fix: Pull stats in IdentVisitor/GraphvizVisitor only when requested (fix: Pull stats in IdentVisitor/GraphvizVisitor only when requested apache/datafusion#8514)
Change display of RepartitionExec from SortPreservingRepartitionExec to RepartitionExec preserve_order=true (Change display of RepartitionExec from SortPreservingRepartitionExec to RepartitionExec preserve_order=true apache/datafusion#8521)
Fix DataFrame::cache errors with Plan("Mismatch between schema and batches") (Fix DataFrame::cache errors with Plan("Mismatch between schema and batches") apache/datafusion#8510)
Minor: update pbjson_dependency (Minor: update pbjson_dependency apache/datafusion#8470)
Minor: Update prost-derive dependency (Minor: Update prost-derive dependency apache/datafusion#8471)
Add write_table to dataframe actions in user guide (Minor/Doc: Add DataFrame::write_table to DataFrame user guide apache/datafusion#8527)
Minor: Add repartition_file.slt end to end test for repartitioning files, and supporting tweaks (Minor: Add repartition_file.slt end to end test for repartitioning files, and supporting tweaks apache/datafusion#8505)
Prepare version 34.0.0 (Prepare version 34.0.0 apache/datafusion#8508)
refactor: use ExprBuilder to consume substrait expr and use macro to generate error (refactor: use ExprBuilder to consume substrait expr and use macro to generate error apache/datafusion#8515)
Make tests deterministic ([MINOR]: Make some slt tests deterministic apache/datafusion#8525)
fix: volatile expressions should not be target of common subexpt elimination (fix: volatile expressions should not be target of common subexpt elimination apache/datafusion#8520)
Add LakeSoul to the list of Known Users (Minor: Add LakeSoul to the list of Known Users apache/datafusion#8536)
Fix regression with Incorrect results when reading parquet files with different schemas and statistics (Fix regression with Incorrect results when reading parquet files with different schemas and statistics apache/datafusion#8533)
feat: improve string statistics display (feat: improve string statistics display in datafusion-cli parquet_metadata function apache/datafusion#8535)
Defer file creation to write (Defer file creation to write apache/datafusion#8539)
Minor: Improve error handling in sqllogictest runner (Minor: Improve error handling in sqllogictest runner apache/datafusion#8544)
Remove order_bys from AggregateExec state (Remove order_bys from AggregateExec state apache/datafusion#8537)
Fix count(null) and count(distinct null) (Fix count(null) and count(distinct null) apache/datafusion#8511)
Minor: reduce code duplication in date_bin_impl (Minor: reduce code duplication in date_bin_impl apache/datafusion#8528)
Add metrics for UnnestExec (Add metrics for UnnestExec apache/datafusion#8482)
regenerate changelog (Prepare 34.0.0-rc3 apache/datafusion#8549)
fix: make sure CASE WHEN pick first true branch when WHEN clause is true (fix: make sure CASE WHEN pick first true branch when WHEN clause is true apache/datafusion#8477)
Minor: make SubqueryAlias::try_new take Arc (Minor: make SubqueryAlias::try_new take Arc<LogicalPlan> apache/datafusion#8542)
Fallback on null empty value in ExprBoundaries::try_from_column (Fallback on null empty value in ExprBoundaries::try_from_column apache/datafusion#8501)
Add test for DataFrame::write_table (Add test for DataFrame::write_table apache/datafusion#8531)
Generate empty column at placeholder exec ([MINOR]: Generate empty column at placeholder exec apache/datafusion#8553)
Minor: Remove now dead SUPPORTED_STRUCT_TYPES (Minor: Remove now dead SUPPORTED_STRUCT_TYPES apache/datafusion#8480)
[MINOR]: Add getter methods to first and last value ([MINOR]: Add getter methods to first and last value apache/datafusion#8555)
[MINOR]: Some code changes and a new empty batch guard for SHJ ([MINOR]: Some code changes and a new empty batch guard for SHJ apache/datafusion#8557)
docs: update udf docs for udtf (docs: update udf docs for udtf apache/datafusion#8546)
feat: implement Unary Expr in substrait (feat: implement Unary Expr in substrait apache/datafusion#8534)
Fix compute_record_batch_statistics wrong with projection (Fix compute_record_batch_statistics wrong with projection apache/datafusion#8489)
cleanup parquet flag (Minor: Cleanup warning in scalar.rs test apache/datafusion#8563)
Minor: move some invariants out of the loop (Minor: move some invariants out of the loop apache/datafusion#8564)
feat: implement Repartition plan in substrait (feat: implement Repartition plan in substrait apache/datafusion#8526)
Fix sort order aware file group parallelization (Fix sort order aware file group parallelization apache/datafusion#8517)
feat: support largelist in array_slice (feat: support largelist in array_slice apache/datafusion#8561)
minor: fix to support scalars (minor: fix to support scalars apache/datafusion#8559)
refactor: HashJoinStream state machine (refactor: HashJoinStream state machine apache/datafusion#8538)
Remove ListingTable and FileScanConfig Unbounded (Remove ListingTable and FileScanConfig Unbounded apache/datafusion#8540) (Remove ListingTable and FileScanConfig Unbounded (#8540) apache/datafusion#8573)
Update substrait requirement from 0.20.0 to 0.21.0 (Update substrait requirement from 0.20.0 to 0.21.0 apache/datafusion#8574)
[minor]: Fix rank calculation bug when empty order by is seen ([minor]: Fix rank calculation bug when empty order by is seen apache/datafusion#8567)
Add LiteralGuarantee on columns to extract conditions required for PhysicalExpr expressions to evaluate to true (Add LiteralGuarantee on columns to extract conditions required for PhysicalExpr expressions to evaluate to true apache/datafusion#8437)
[MINOR]: Parametrize sort-preservation tests to exercise all situations (unbounded/bounded sources and flag behavior) ([MINOR]: Parametrize sort-preservation tests to exercise all situations (unbounded/bounded sources and flag behavior) apache/datafusion#8575)
Minor: Add some comments to scalar_udf example (Minor: Add some comments to scalar_udf example apache/datafusion#8576)
Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend (Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend apache/datafusion#8317)
support LargeList in array_positions (feat: support LargeList in array_positions apache/datafusion#8571)
support LargeList in array_element (feat: support LargeList in array_element apache/datafusion#8570)
Increase test coverage for unbounded and bounded cases (Increase test coverage for unbounded and bounded cases apache/datafusion#8581)
Port tests in parquet.rs to sqllogictest (Port tests in parquet.rs to sqllogictest apache/datafusion#8560)
Minor: avoid a copy in Expr::unalias (Minor: avoid a copy in Expr::unalias apache/datafusion#8588)
Minor: support complex expr as the arg in the ApproxPercentileCont function (Minor: support complex expr as the arg in the ApproxPercentileCont function apache/datafusion#8580)
Bugfix: Add functional dependency check and aggregate try_new schema (Bugfix: Add functional dependency check and aggregate try_new schema apache/datafusion#8584)
Remove GroupByOrderMode (Remove GroupByOrderMode apache/datafusion#8593)
replace not-impl-err (Minor: replace not-impl-err in array_expression apache/datafusion#8589)
Substrait insubquery (Substrait insubquery apache/datafusion#8363)
Minor: port last test from parquet.rs (Minor: port last test from parquet.rs apache/datafusion#8587)
Minor: consolidate map sqllogictest tests (Minor: consolidate map sqllogictest tests apache/datafusion#8550)
feat: support LargeList in array_dims (feat: support LargeList in array_dims apache/datafusion#8592)
Fix regression in regenerating protobuf source (Fix regression in regenerating protobuf source apache/datafusion#8603)
Remove unbounded_input from FileSinkOptions (Remove unbounded_input from FileSinkOptions apache/datafusion#8605)
Add arrow_err! macros, optional backtrace to ArrowError (Add arrow_err! macros, optional backtrace to ArrowError apache/datafusion#8586)
Add examples of DataFrame::write* methods without S3 dependency (Add examples of DataFrame::write* methods without S3 dependency apache/datafusion#8606)
Implement logical plan serde for CopyTo (Implement logical plan serde for CopyTo apache/datafusion#8618)
Fix InListExpr to return the correct number of rows (Fix InListExpr to return the correct number of rows apache/datafusion#8601)
Remove ListingTable single_file option (Remove ListingTable single_file option apache/datafusion#8604)
support LargeList in array_remove (feat: support LargeList in array_remove apache/datafusion#8595)
Rename ParamValues::{LIST -> List,MAP -> Map} (Rename ParamValues::{LIST -> List,MAP -> Map} apache/datafusion#8611)
Support binary temporal coercion for Date64 and Timestamp types
Add new configuration item listing_table_ignore_subdirectory (Add new configuration item listing_table_ignore_subdirectory apache/datafusion#8565)
Optimize the parameter types of ParamValues's methods (Optimize the parameter types of ParamValues's methods apache/datafusion#8613)
Don't panic on zero placeholder in ParamValues::get_placeholders_with_values (Do not panic on zero placeholders in ParamValues::get_placeholders_with_values apache/datafusion#8615)
Fix non-null sub-field on nullable struct-field has wrong nullity. apache/datafusion#8507: Non-null sub-field on nullable struct-field has wrong nullity (Fix #8507: Non-null sub-field on nullable struct-field has wrong nullity apache/datafusion#8623)
Implement contained API in PruningPredicate (Implement contained API in PruningPredicate apache/datafusion#8440)
Add partial serde support for ParquetWriterOptions (Add partial serde support for ParquetWriterOptions apache/datafusion#8627)
add arguments length check (Minor: add arguments length check in array_expressions apache/datafusion#8622)
Improve DataFrame functional tests (Minor: improve dataframe functional dependency tests apache/datafusion#8630)
Improve regexp_match performance by avoiding cloning Regex (Improve regexp_match performance by avoiding cloning Regex apache/datafusion#8631)
Minor: improve listing_table_ignore_subdirectory config documentation (Minor: improve listing_table_ignore_subdirectory config documentation apache/datafusion#8634)
Support Writing Arrow files (Support Writing Arrow files apache/datafusion#8608)
Filter pushdown into cross join (Filter pushdown into cross join apache/datafusion#8626)
[MINOR] Remove duplicate test utility and move one utility function for better organization ([MINOR] Remove duplicate test utility and move one utility function for better organization apache/datafusion#8652)
[MINOR]: Add new test for filter pushdown into cross join ([MINOR]: Add new test for filter pushdown into cross join apache/datafusion#8648)
Rewrite bloom filters to use contains API (Rewrite bloom filters to use contains API apache/datafusion#8442)
Split equivalence code into smaller modules. (Split equivalence code into smaller modules. apache/datafusion#8649)
Move parquet_schema.rs from sql to parquet tests (Move parquet_schema.rs from sql to parquet tests apache/datafusion#8644)
Fix group by aliased expression in LogicalPLanBuilder::aggregate (Fix group by aliased expression in LogicalPLanBuilder::aggregate apache/datafusion#8629)
Refactor array_union and array_intersect functions to one general function (Refactor array_union and array_intersect functions to one general function apache/datafusion#8516)
Avoid extra clone in datafusion-proto::physical_plan (Minor: avoid extra clone in datafusion-proto::physical_plan apache/datafusion#8650)
Minor: name some constant values in arrow writer, parquet writer (Minor: name some constant values in arrow writer, parquet writer apache/datafusion#8642)
TreeNode Refactor Part 2 (TreeNode Refactor Part 2 apache/datafusion#8653)
feat: support inlist in LiteralGurantee for pruning (feat: support inlist in LiteralGurantee for pruning apache/datafusion#8654)
Streaming CLI support (Streaming CLI support apache/datafusion#8651)
Add serde support for CSV FileTypeWriterOptions (Add serde support for CSV FileTypeWriterOptions apache/datafusion#8641)
Add trait based ScalarUDF API (Add trait based ScalarUDF API apache/datafusion#8578)
Handle ordering of first last aggregation inside aggregator (Handle ordering of first last aggregation inside aggregator apache/datafusion#8662)
feat: support 'LargeList' in array_pop_front and array_pop_back (feat: support 'LargeList' in array_pop_front and array_pop_back apache/datafusion#8569)
chore: rename ceresdb to apache horaedb (chore: rename ceresdb to apache horaedb apache/datafusion#8674)
clean code (Minor: clean up code apache/datafusion#8671)
remove tz with modified offset from tests (fix: Antarctica/Vostok tz offset changed in chrono-tz 0.8.5 apache/datafusion#8677)
Make the BatchSerializer behind Arc to avoid unnecessary struct creation (Make the BatchSerializer behind Arc to avoid unnecessary struct creation apache/datafusion#8666)
Implement serde for CSV and Parquet FileSinkExec (Implement serde for CSV and Parquet FileSinkExec apache/datafusion#8646)
[pruning] Add shortcut when all units have been pruned ([pruning] Add shortcut when all units have been pruned apache/datafusion#8675)
Change first/last implementation to prevent redundant comparisons when data is already sorted (Change first/last implementation to prevent redundant comparisons when data is already sorted apache/datafusion#8678)
minor: remove unused conversion (minor: remove useless conversion apache/datafusion#8684)
refactor: modified JoinHashMap build order for HashJoinStream (refactor: modified JoinHashMap build order for HashJoinStream apache/datafusion#8658)
Start setting up tpch planning benchmarks (Start setting up tpch planning benchmarks apache/datafusion#8665)
update doc (Doc: Clarify When Limit is Pushed Down to TableProvider::Scan apache/datafusion#8686)
Closes Parallel NDSON file reading apache/datafusion#8502: Parallel NDJSON file reading (Closes #8502: Parallel NDJSON file reading apache/datafusion#8659)
init draft (Improve array_prepend signature for null and empty array apache/datafusion#8625)
Cleanup TreeNode implementations (Cleanup TreeNode implementations apache/datafusion#8672)
Update sqlparser requirement from 0.40.0 to 0.41.0 (Update sqlparser requirement from 0.40.0 to 0.41.0 apache/datafusion#8647)
Update scalar functions doc for extract/datepart (Update scalar functions doc for extract/datepart apache/datafusion#8682)
Remove DescribeTableStmt in parser in favour of existing functionality from sqlparser-rs (Remove DescribeTableStmt in parser in favour of existing functionality from sqlparser-rs apache/datafusion#8703)
feat: simplify null in list (Simplify NULL [NOT] IN (..) expressions apache/datafusion#8691)
Rename expr::window_function::WindowFunction to WindowFunctionDefinition, make structure consistent with ScalarFunction (Rename expr::window_function::WindowFunction to WindowFunctionDefinition, make structure consistent with ScalarFunction apache/datafusion#8382)
Deprecate duplicate function LogicalPlan::with_new_inputs (Deprecate duplicate function LogicalPlan::with_new_inputs apache/datafusion#8707)
Minor: refactor bloom filter tests to reduce duplication (Minor: refactor bloom filter tests to reduce duplication apache/datafusion#8435)
clean up code (Minor: clean up code based on Clippy apache/datafusion#8715)
Update analyze.rs (Minor: Unbounded Output of AnalyzeExec apache/datafusion#8717)
support LargeList in array_position (feat: support LargeList in array_position apache/datafusion#8714)
support LargeList in array_ndims (feat: support LargeList in array_ndims apache/datafusion#8716)
feat: remove filters with null constants (feat: remove filters with null constants apache/datafusion#8700)
support LargeList in array_prepend and array_append (support LargeList in array_prepend and array_append apache/datafusion#8679)
Support for extract(epoch from date) for Date32 and Date64 (Support for extract(epoch from date) for Date32 and Date64 apache/datafusion#8695)
Implement trait based API for defining WindowUDF (Implement trait based API for defining WindowUDF apache/datafusion#8719)
Minor: Introduce utils::hash for StructArray (Minor: Introduce utils::hash for StructArray apache/datafusion#8552)
[CI] Improve windows machine CI test time ([CI] Improve windows machine CI test time apache/datafusion#8730)
fix guarantees in allways_true of PruningPredicate (fix guarantees in allways_true of PruningPredicate apache/datafusion#8732)
[Minor] Avoid mem copy in generate window exprs (Minor: Avoid memory copy in construct window exprs apache/datafusion#8718)
support LargeList in array_repeat (feat: support LargeList in array_repeat apache/datafusion#8725)
ctrl+c termination (Minor: Ctrl+C Termination in CLI apache/datafusion#8739)
Add support for functional dependency for ROW_NUMBER window function. (Add support for functional dependency for ROW_NUMBER window function. apache/datafusion#8737)
Minor: reduce code duplication in PruningPredicate test (Minor: reduce code duplication in PruningPredicate test apache/datafusion#8441)
feat: native types in DistinctCountAccumulator for primitive types (feat: native types in DistinctCountAccumulator for primitive types apache/datafusion#8721)
Add test case for unnecessary hash when target is 1 ([MINOR]: Add a test case for when target partition is 1, no hash repartition is added to the plan. apache/datafusion#8757)
Minor: Improve PruningPredicate docstrings more (Minor: Improve PruningPredicate docstrings apache/datafusion#8748)
feat: support LargeList in cardinality (feat: support LargeList in cardinality apache/datafusion#8726)
Add reproducer for Regression "RowConverter column schema mismatch, expected Utf8 got Dictionary(Int32, Utf8)" after upgrade apache/datafusion#8738 (Add reproducer for #8738 apache/datafusion#8750)
Update checking columns in schema merge (Minor: Use faster check for column name in schema merge apache/datafusion#8765)
Minor: Add documentation about stream cancellation (Minor: Add documentation about stream cancellation apache/datafusion#8747)
Move repartition_file_scans out of enable_round_robin check in EnforceDistribution rule (Move repartition_file_scans out of enable_round_robin check in EnforceDistribution rule apache/datafusion#8731)
Clean internal implementation of WindowUDF (Clean internal implementation of WindowUDF apache/datafusion#8746)
feat: support largelist in array_to_string (feat: support largelist in array_to_string apache/datafusion#8729)
Error handling. ([MINOR] CLI error handling on streaming use cases apache/datafusion#8761)
Convert Binary Operator StringConcat to Function for array_concat, array_append and array_prepend (Convert Binary Operator StringConcat to Function for array_concat, array_append and array_prepend apache/datafusion#8636)
Minor: Fix incorrect indices for hashing struct (Minor: Fix incorrect indices for hashing struct apache/datafusion#8775)
Minor: Improve library docs to mention TreeNode, ExprSimplifier, PruningPredicate and cp_solver (Minor: Improve library docs to mention TreeNode, ExprSimplifier, PruningPredicate and cp_solver apache/datafusion#8749)
[MINOR] Add logo source files ([MINOR] Add logo source files apache/datafusion#8762)
Add Apache attribution to site footer (Add Apache attribution to site footer apache/datafusion#8760)
ci: speed up win64 test (ci: speed up win64 test apache/datafusion#8728)
Add schema_err! error macros with optional backtrace (Add schema_err! error macros with optional backtrace apache/datafusion#8620)
Fix regression by reverting Materialize dictionaries in group keys (Fix regression by reverting Materialize dictionaries in group keys apache/datafusion#8740)
fix: struct field don't push down to TableScan (fix: struct field don't push down to TableScan apache/datafusion#8774)
Implement ScalarUDF in terms of ScalarUDFImpl trait (Implement ScalarUDF in terms of ScalarUDFImpl trait apache/datafusion#8713)
Minor: Fix error messages in array expressions (Minor: Fix error messages in array expressions apache/datafusion#8781)
Move tests from expr.rs to sqllogictests. Part1 (Move tests from expr.rs to sqllogictests. Part1 apache/datafusion#8773)
add config flag for recursive ctes

* move array function unit_tests to sqllogictest Signed-off-by: veeupup <[email protected]> * add comment for array_expression internal test --------- Signed-off-by: veeupup <[email protected]>

Co-authored-by: Mehmet Ozan Kabak <[email protected]>

…own projections) (apache#8340)

…che#8351) * Minor: Improve the document format of JoinHashMap * sql csv_with_quote_escape * fix

* Minor: restore DataFrame test * Move test to a better location * simplify test

…pache#8354) These utils manipulate `LogicalPlan`s and `Expr`s and may be useful in projects that only depend on `datafusion-expr`

* Extract parquet statistics to its own module, add tests * Update datafusion/core/src/datasource/physical_plan/parquet/statistics.rs Co-authored-by: Raphael Taylor-Davies <[email protected]> * rename enum * Improve API * Add test for reading struct array statistics * Add test for column after statistics * improve tests * simplify * clippy * Update datafusion/core/src/datasource/physical_plan/parquet/statistics.rs * Update datafusion/core/src/datasource/physical_plan/parquet/statistics.rs * Add test showing incorrect statistics * Rework statistics * Fix clippy * Update documentation and make it clear the statistics are not publically accessable * Add link to upstream arrow ticket --------- Co-authored-by: Raphael Taylor-Davies <[email protected]> Co-authored-by: Raphael Taylor-Davies <[email protected]>

* feat:implement sql style 'find_in_set' string function * format code * modify test case

Signed-off-by: jayzhan211 <[email protected]>

* Refactor aggregate function handling * fix ci * update comment * fix ci * simplify the code * fix fmt * fix ci * fix clippy

* Implement Aliases for ScalarUDF Signed-off-by: veeupup <[email protected]> * fix comments Signed-off-by: veeupup <[email protected]> --------- Signed-off-by: veeupup <[email protected]>

…#8365)

* support LargeList in array_empty * update err info

* feat: test queries for to_timestamp(float) WIP * feat: Float64 input for to_timestamp * cargo fmt * clippy * docs: double input type for to_timestamp * feat: cast floats to timestamp * style: cargo fmt * fix: float64 cast for timestamp nanos only

* Support User Defined Table Function Signed-off-by: veeupup <[email protected]> * fix comments Signed-off-by: veeupup <[email protected]> * add udtf test Signed-off-by: veeupup <[email protected]> * add file header * Simply table function example, add some comments * Simplfy exprs * make clippy happy * Update datafusion/core/tests/user_defined/user_defined_table_functions.rs --------- Signed-off-by: veeupup <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

* document timestamp input limis * fix text * prettier * remove doc for nanoseconds * Update datafusion/physical-expr/src/datetime_expressions.rs Co-authored-by: Andrew Lamb <[email protected]> --------- Co-authored-by: Andrew Lamb <[email protected]>

* fix: make ntile work in some corner cases * fix comments * minor * Update datafusion/sqllogictest/test_files/window.slt Co-authored-by: Mustafa Akur <[email protected]> --------- Co-authored-by: Mustafa Akur <[email protected]>

union_arrays function

ScalarFunctionDefinition

Given that group keys inherently have few repeated values, especially when grouping on a single column, the use of dictionary encoding is unlikely to be yielding significant returns

* done Signed-off-by: jayzhan211 <[email protected]> * add more test Signed-off-by: jayzhan211 <[email protected]> * cleanup Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

* Minor: Improve the documentation on `ScalarValue` * Update datafusion/common/src/scalar.rs Co-authored-by: Liang-Chi Hsieh <[email protected]> * Update datafusion/common/src/scalar.rs Co-authored-by: Liang-Chi Hsieh <[email protected]> --------- Co-authored-by: Liang-Chi Hsieh <[email protected]>

* add benchmark Signed-off-by: jayzhan211 <[email protected]> * fmt Signed-off-by: jayzhan211 <[email protected]> * address clippy Signed-off-by: jayzhan211 <[email protected]> * cleanup Signed-off-by: jayzhan211 <[email protected]> * fix comment Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

* minor changes * PipelineStatePropagator tree refactor * Remove duplications by children_unbounded() * Remove on-the-fly tree construction * Minor changes --------- Co-authored-by: Mustafa Akur <[email protected]>

…8121) * feat: support LargeList in make_array and array_length * chore: add tests * fix: update tests for nested array * use usise_as * add new_large_list * refactor array_length * add comment * update test in sqllogictest * fix ci * fix macro * use usize_as * update comment * return based on data_type in make_array

…apache#8737) * Add primary key support for row_number window function * Add comments, minor changes * Add new test * Review --------- Co-authored-by: Mehmet Ozan Kabak <[email protected]>

…pache#8721) * DistinctCountGroupsAccumulator * test coverage * clippy warnings * count distinct for primitive types * revert hashset to std * fixed accumulator size estimation

* support LargeList in cardinality

…nforceDistribution` rule (apache#8731) * Cleanup * More * Restore add_roundrobin_on_top * Restore test files * More * Restore * More * More * Make test stable * For review * Add test

* Clean internal implementation of WindowUDF * fix doc

* support largelist in array_to_string * reduce code duplication

…, `array_append` and `array_prepend` (apache#8636) * reuse function for string concat Signed-off-by: jayzhan211 <[email protected]> * remove casting in string concat Signed-off-by: jayzhan211 <[email protected]> * add test Signed-off-by: jayzhan211 <[email protected]> * operator to function rewrite Signed-off-by: jayzhan211 <[email protected]> * fix explain Signed-off-by: jayzhan211 <[email protected]> * add more test Signed-off-by: jayzhan211 <[email protected]> * add column cases Signed-off-by: jayzhan211 <[email protected]> * cleanup Signed-off-by: jayzhan211 <[email protected]> * presever name Signed-off-by: jayzhan211 <[email protected]> * Update datafusion/optimizer/src/analyzer/rewrite_expr.rs Co-authored-by: Andrew Lamb <[email protected]> * rename Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

* fix bug Signed-off-by: jayzhan211 <[email protected]> * fmt Signed-off-by: jayzhan211 <[email protected]> * add rowsort Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]>

…ingPredicate and cp_solver (apache#8749) * Minor: Improve library docs to mention TreeNode, ExprSimplifier, PruningPredicate and cp_solver * fix link

* Add logo source files * add another file

* Add `schema_err!` error macros with optional backtrace

…pache#8740) * revert eb8aff7 / Materialize dictionaries in group keys * Update tests * Update tests

* fix: struct don't push down to TableScan * add similar to test and apply comment * remove catch all in outer_columns_helper * minor * fix clippy --------- Co-authored-by: Andrew Lamb <[email protected]>

* Fix error messages in array expressions * fix fmt

* move tests from to sqllogictests part1 * Update datafusion/sqllogictest/test_files/expr.slt Co-authored-by: Andrew Lamb <[email protected]> * Update datafusion/sqllogictest/test_files/expr.slt Co-authored-by: Andrew Lamb <[email protected]> --------- Co-authored-by: Andrew Lamb <[email protected]>

Veeupup and others added 30 commits November 28, 2023 10:02

move array function unit_tests to sqllogictest (apache#8332)

2a69244

* move array function unit_tests to sqllogictest Signed-off-by: veeupup <[email protected]> * add comment for array_expression internal test --------- Signed-off-by: veeupup <[email protected]>

NTH_VALUE reverse support (apache#8327)

e21b031

Co-authored-by: Mehmet Ozan Kabak <[email protected]>

Refactor optimize projections rule, combines (eliminate, merge, pushd…

19bdcdc

…own projections) (apache#8340)

Move merge projections tests to under optimize projections (apache#8352)

4c914ea

Add quote and escape attributes to create csv external table (apa…

aeb012e

…che#8351) * Minor: Improve the document format of JoinHashMap * sql csv_with_quote_escape * fix

Minor: Add DataFrame test (apache#8341)

d22403a

* Minor: restore DataFrame test * Move test to a better location * simplify test

clean up the code based on Clippy (apache#8359)

e93f8e1

Minor: Make it easier to work with Expr::ScalarFunction (apache#8350)

bbec787

Move some datafusion-optimizer::utils down to datafusion-expr::utils (a…

11f164c

…pache#8354) These utils manipulate `LogicalPlan`s and `Expr`s and may be useful in projects that only depend on `datafusion-expr`

Minor: Make BuiltInScalarFunction::alias a method (apache#8349)

167b5b7

feat:implement sql style 'find_in_set' string function (apache#8328)

c079a92

* feat:implement sql style 'find_in_set' string function * format code * modify test case

largeutf to temporal (apache#8357)

a588123

Signed-off-by: jayzhan211 <[email protected]>

Refactor aggregate function handling (apache#8358)

a49740f

* Refactor aggregate function handling * fix ci * update comment * fix ci * simplify the code * fix fmt * fix ci * fix clippy

Implement Aliases for ScalarUDF (apache#8360)

d45cf00

* Implement Aliases for ScalarUDF Signed-off-by: veeupup <[email protected]> * fix comments Signed-off-by: veeupup <[email protected]> --------- Signed-off-by: veeupup <[email protected]>

Minor: Remove uncessary name field in ScalarFunctionDefintion (apache…

513fd05

…#8365)

feat: support LargeList in array_empty (apache#8321)

e52d150

* support LargeList in array_empty * update err info

fix: make ntile work in some corner cases (apache#8371)

eb5aa22

* fix: make ntile work in some corner cases * fix comments * minor * Update datafusion/sqllogictest/test_files/window.slt Co-authored-by: Mustafa Akur <[email protected]> --------- Co-authored-by: Mustafa Akur <[email protected]>

Refactor array_union function to use a generic (apache#8381)

8882f1b

union_arrays function

Refactor function argument handling in (apache#8387)

a6e6d3f

ScalarFunctionDefinition

Materialize dictionaries in group keys (apache#7647) (apache#8291)

eb8aff7

Given that group keys inherently have few repeated values, especially when grouping on a single column, the use of dictionary encoding is unlikely to be yielding significant returns

add a summary table to benchmark compare output (apache#8399)

bb2ea4b

Refactors on TreeNode Implementations (apache#8395)

075ff3d

* minor changes * PipelineStatePropagator tree refactor * Remove duplications by children_unbounded() * Remove on-the-fly tree construction * Minor changes --------- Co-authored-by: Mustafa Akur <[email protected]>

Ted-Jiang and others added 29 commits January 3, 2024 17:02

[Minor] Avoid mem copy in generate window exprs (apache#8718)

881d03f

support LargeList in array_repeat (apache#8725)

ca260d9

ctrl+c termination (apache#8739)

e6b9f52

Add support for functional dependency for ROW_NUMBER window function. (…

819d357

…apache#8737) * Add primary key support for row_number window function * Add comments, minor changes * Add new test * Review --------- Co-authored-by: Mehmet Ozan Kabak <[email protected]>

Minor: reduce code duplication in PruningPredicate test (apache#8441)

e5036d0

feat: native types in DistinctCountAccumulator for primitive types (a…

561d941

…pache#8721) * DistinctCountGroupsAccumulator * test coverage * clippy warnings * count distinct for primitive types * revert hashset to std * fixed accumulator size estimation

Add test case for unnecessary hash when target is 1 (apache#8757)

05e3d45

Minor: Improve PruningPredicate docstrings more (apache#8748)

4173070

feat: support LargeList in cardinality (apache#8726)

af20d2d

* support LargeList in cardinality

Add reproducer for apache#8738 (apache#8750)

93b4a4c

Update checking columns in schema merge (apache#8765)

6e75297

Minor: Add documentation about stream cancellation (apache#8747)

0208755

Move repartition_file_scans out of enable_round_robin check in `E…

29f23eb

…nforceDistribution` rule (apache#8731) * Cleanup * More * Restore add_roundrobin_on_top * Restore test files * More * Restore * More * More * Make test stable * For review * Add test

Clean internal implementation of WindowUDF (apache#8746)

98f02ff

* Clean internal implementation of WindowUDF * fix doc

feat: support largelist in array_to_string (apache#8729)

b2e8848

* support largelist in array_to_string * reduce code duplication

Error handling. (apache#8761)

821db54

Minor: Fix incorrect indices for hashing struct (apache#8775)

7316274

* fix bug Signed-off-by: jayzhan211 <[email protected]> * fmt Signed-off-by: jayzhan211 <[email protected]> * add rowsort Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]>

Minor: Improve library docs to mention TreeNode, ExprSimplifier, Prun…

4dd09f3

…ingPredicate and cp_solver (apache#8749) * Minor: Improve library docs to mention TreeNode, ExprSimplifier, PruningPredicate and cp_solver * fix link

[MINOR] Add logo source files (apache#8762)

d6c891e

* Add logo source files * add another file

Add Apache attribution to site footer (apache#8760)

40c674b

ci: speed up win64 test (apache#8728)

4289737

Add schema_err! error macros with optional backtrace (apache#8620)

dd4263f

* Add `schema_err!` error macros with optional backtrace

Fix regression by reverting Materialize dictionaries in group keys (a…

ff27d90

…pache#8740) * revert eb8aff7 / Materialize dictionaries in group keys * Update tests * Update tests

fix: struct field don't push down to TableScan (apache#8774)

cc42894

* fix: struct don't push down to TableScan * add similar to test and apply comment * remove catch all in outer_columns_helper * minor * fix clippy --------- Co-authored-by: Andrew Lamb <[email protected]>

Implement ScalarUDF in terms of ScalarUDFImpl trait (apache#8713)

746988a

Minor: Fix error messages in array expressions (apache#8781)

dd58c5a

* Fix error messages in array expressions * fix fmt

add config flag for recursive ctes

128b2c6

matthewgapp closed this Jan 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

matt/feat/recursive ctes/config flag #3

matt/feat/recursive ctes/config flag #3

matthewgapp commented Jan 11, 2024

matt/feat/recursive ctes/config flag #3

matt/feat/recursive ctes/config flag #3

Conversation

matthewgapp commented Jan 11, 2024