Release notes for 361

[martint]

Ashhar Hasan

• all checked
• 08-02 10:13:04 9e3a1db Provide more information to connectors to control aggregation pushdown
• 08-02 10:13:04 ee57029 Prevent aggregation pushdown for textual types for some connectors
• 08-03 22:49:25 406e521 Include JDBC type mapping fragment in all applicable connectors
• 08-03 22:49:25 4394ce9 Split general configuration properties to their own fragment
• 08-04 22:43:41 7d7df75 Resolve issue with OAuth2 product tests and reenable
• 08-05 00:02:18 858f839 Document non-transactional-insert property for applicable connectors
• 08-05 22:26:44 20f9c48 Update code not to use QueryAssert.containsExactly and use containsExactlyInOrder instead
• 08-09 02:26:06 3cfa652 Expose stats for caches in CachingJdbcClient
• 08-09 02:26:06 8d5b360 Add metadata.cache-maximum-size config option to CachingJdbcClient
• 08-09 02:26:06 e8ce319 Add flushCache method to CachingJdbcClient to invalidate all caches
• 08-09 03:30:24 6b86046 Refactor pushdown markings in CassandraClusteringPredicatesExtractor
• 08-09 03:30:24 750000c Fix Cassandra OrderedRangeSet pushdown
• 08-09 22:54:00 9d1e6ac Avoid hard-coded http scheme for Pinot controller url
• 08-10 22:31:54 7344d9d Add link for MongoDB objectid_timestamp in function list
• 08-11 05:31:33 3eaf4bb Fix incorrect comment
• 08-13 00:18:14 bcf9d32 Update path to run-launcher script
• 08-13 00:18:14 cf1f32a Add instructions for attaching a debugger to product tests
• 08-13 12:36:48 dfa2048 Add and update to consistent anchors

David Phillips

• all checked
• 08-02 18:02:46 cf0ae11 Iceberg: add new fields in files system table
• 08-04 12:21:57 55811f2 Use FluentFuture in QueuedStatementResource
• 08-04 12:22:49 dbaaaac Add CI variable for Maven command
• 08-13 11:02:40 22fef6a Document JDBC sessionUser parameter
• 08-16 14:51:43 9d73101 Refactor IcebergMetadata to support multiple catalogs

GitHub

• all checked
• 08-04 10:12:14 cb33b01 Update PostgreSQL driver to 42.2.23
• 08-14 18:33:15 6d290fd Correct spill-window-operator ending in documentation

Grzegorz Kokosiński

• all checked
• 08-02 05:32:26 480a621 Update error-prone to 2.8.0
• 08-02 23:17:35 ca55aac Disable oauth2 product tests
• 08-07 12:53:02 4da67db Pin selenium/standalone-chrome-debug docker image version
• 08-10 02:42:44 58832bc Clean up Optional usage in BigQueryConnectorModule
• 08-10 02:42:44 5d63f07 Use Optional::or instead of Stream::findFirst on a temporary stream
• 08-13 05:33:12 3ec5790 Add merge strategy to DEVELOPMENT.md

Karol Sobczak

• all checked
• 07-31 03:49:51 61077e6 Update ConnectorMetadata#listTables contract

Martin Traverso

• all checked
• 08-16 14:05:15 0215166 Improve Decimal Aggregation State Serializer Performance
• 08-16 14:05:15 3791bae Improve BenchmarkDecimalAggregation
• 08-16 14:05:15 5783c8c Use unbound method references for input functions
• 08-16 14:05:15 5b3c384 Lazily allocate overflows array in grouped decimal aggregations

Piotr Findeisen

• all checked
• 07-30 15:13:46 1f896ea Don't use raw Metric type outside of Metrics class
• 08-02 02:14:08 33d8410 Support reading from Iceberg table after partition field dropped
• 08-02 02:14:08 4bb71c5 Support writing to Iceberg table after partition field dropped
• 08-02 02:14:08 adb45f3 Update naming in Iceberg Trino/Spark compatibility test
• 08-02 05:25:26 c3cd438 Bind container ports to host ports by default in product tests
• 08-02 23:08:39 760e2d2 Reject Glue and Alluxio metastores from Iceberg
• 08-02 23:08:39 e927b4c Remove redundant test resource from TestIcebergPlugin
• 08-03 03:39:38 4312479 Rename varchar/varbinary Parquet writer
• 08-03 03:39:38 89b40ce Correct type used to get varbinary value
• 08-03 03:39:38 8fbd7fa Convert to Iceberg value strictly
• 08-03 03:39:59 d1571c0 Move teardown next to setup method
• 08-03 03:39:59 e99b53b Remove left over file
• 08-03 03:39:59 f18a0a5 Fix typo in test constant name
• 08-03 03:46:11 cf24fa2 Add test counting accesses to files for iceberg
• 08-03 06:23:36 bf5a1fa Test file access without JMX in Iceberg
• 08-03 06:23:36 c88077a Remove redundant test schema
• 08-03 06:33:25 bea74d5 rebase: Added ability to have unique table location for each iceberg table
• 08-03 06:56:02 722c9fa Test Iceberg and Hive table compatibility
• 08-03 12:57:55 dd113a3 Use table name as cache key explicitly
• 08-03 13:03:22 0f8d702 Fix usage of raw Java type
• 08-03 13:03:22 216d81e Avoid redundant metastore calls for Iceberg system tables
• 08-03 13:03:22 4e60758 Test metastore invocations for Iceberg system table access
• 08-04 08:22:49 ba05c00 Remove overhead from ANALYZE since SQL info is now in connectors

Yuya Ebihara

• all checked
• 08-07 16:45:25 74806c5 Fix ORC session property casing typo
• 08-08 08:01:04 e581f4f Add test for MongoDB ObjectId
• 08-09 18:08:30 2f3889e Run Kafka and Pinot docker in Pinot query runner
• 08-09 18:12:17 a46e7cd Add timestamp_objectid function in MongoDB connector
• 08-10 04:59:00 663355b Enable mongodb.socket-keep-alive by default
• 08-10 05:41:37 71ce3cb Upgrade google-http-client and google-oauth-client version
• 08-10 07:42:33 f9aac40 Expose $properties table for views in Hive connector
• 08-10 22:43:32 570c647 Fix supported SQL statements for Oracle connector
• 08-13 07:31:26 2663e49 Check table existence in Pinot connector

kasiafi

• all checked
• 08-06 13:21:44 236a6d4 Support subqueries in pattern matching context in window
• 08-06 13:21:44 3318d46 Analyze IN-predicate with subquery in pattern recognition context
• 08-06 13:21:44 41d147b Support subqueries in MATCH_RECOGNIZE
• 08-06 13:21:44 5048dda Add analyzer check for lambda in pattern recognition context
• 08-06 13:21:44 b3b15ca Test IN-predicate without subquery in pattern matching context

Łukasz Osipiuk

• all checked
• 08-11 03:52:45 33be493 Add support for from_utc_timestamp in Hive views translation
• 08-11 23:28:36 dad2092 Fix assertions for old Hive versions in TestHiveViews for from_utc_timestamp
• 08-12 01:37:16 d815688 Fix reading of specific Iceberg snapshots
• 08-12 12:50:16 765eb2e Block deletion of Spark Iceberg table with path override properties
• 08-12 12:50:16 ba89f41 Use Iceberg LocationProvider instead of hardcoded path
• 08-13 08:34:30 05382b5 Refactor table initialization in testFromUtcTimestamp
• 08-13 08:34:30 6443176 Use proper test value in testFromUtcTimestamp
• 08-16 00:06:35 bd2d372 Properly delimit identifiers in DataType expression created from Type
• 08-16 09:41:53 38937db Support reading multiple Glue partition statistics in parallel
• 08-16 09:41:53 c10cd01 Allow for batch updates of Glue column statistics

[findepi]

Iceberg
* Support Iceberg table after partition field dropped. ({issue}`8730`)

#8730

[findepi]

Iceberg
* Use randomized location when creating a table, so that later table rename and drop does not interfere with new tables created with the same name.
  This can be enabled using `iceberg.unique-table-location` configuration property. ({issue}`6063`)

#6063

[hashhar]

ClickHouse
* Fix incorrect results for aggregation functions applied to columns of type VARCHAR and CHAR. ({issue}`7320`)

MySQL
* Fix incorrect results for aggregation functions applied to columns of type VARCHAR and CHAR. ({issue}`7320`)

PostgreSQL
* Fix incorrect results for aggregation functions applied to columns of type VARCHAR and CHAR. ({issue}`7320`)

SQL Server
* Fix incorrect results for aggregation functions applied to columns of type VARCHAR and CHAR. ({issue}`7320`)

#7320, #8551

[ebyhr]

MongoDB
* Add {func}`timestamp_objectid`. ({issue}`8824`)

#8824

[hashhar]

I've unrolled this already to make it easier when adding to release notes - we can condense into a single entry if needed.

ClickHouse
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

Druid
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

MemSQL
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

MySQL
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

Oracle
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

Phoenix
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

PostgreSQL
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

Redshift
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

SQL Server
* Add new configuration property `metadata.cache-maximum-size` to allow limiting the size of the metadata cache. ({issue}`8652`)

#8652

[hashhar]

Cassandra
* Fix incorrect results for predicates with inequality on clustering key. ({issue}`401`)

#401, #8629

[hashhar]

Pinot
* Fix handling of https URLs in `pinot.controller-urls`. ({issue}`8617`)

#8617, #8837

[ebyhr]

MongoDB
* Enable `mongodb.socket-keep-alive` config property by default. ({issue}`8832`)

#8832

[ebyhr]

Hive
* Allow access to `$properties` table for Trino and Hive views. ({issue}`8805`)

#8805, #8810

[losipiuk]

Hive
* Add support for translating Hive views which use `from_utc_timestamp` function. ({issue}`8502`)

#8502 #8860

[homar]

Iceberg
* Return proper query results for queries accessing multiple snapshots of single Iceberg table. ({issue} `8868`)

#8868

[losipiuk]

Iceberg
* Allow reading from Iceberg tables which specify `write.object-storage.path` storage property.  ({issue} `8573`)

cc: @jackye1995

[ebyhr]

Pinot
* Fix failures when querying `information_schema.columns` with a table name condition. ({issue}`8307`)

#8307, #8859

[losipiuk]

General
* Fix failures of specific queries accessing ROW columns with fields requiring a name quoting.  ({issue} `8845`)

#8845

[losipiuk]

Hive
* Increase performance of updating Glue table statistics for partitioned tables. Also change default Glue statistics read/write parallelism from 1 to 5. ({issue} `8839`)

#8839

[martint]

* Improve performance of {func}`sum` and {func}`avg` aggregations involving `decimal` values. ({issue}`8878`)

#8878

[phd3]

General
* Support table redirection for `INSERT`, `DELETE` and `UPDATE` operations. ({issue}`8683`)

#8683

[ebyhr]

Cassandra
* Add support for pushdown of `smallint`, `tinyint` and `date` types on partition columns. ({issue}`3763`)

#5150

[raunaqmorarka]

General
* Performance improvement for queries using `IN` predicate with moderate to large number of constants. ({issue}`8833`)

#8833

[losipiuk]

Hive
* Fix query failure when inserting data into Hive ACID table which was not explicitly bucketed. ({issue} `8899`)

#8899

[11xor6]

General
* Add support for OAuth2/OIDC opaque access tokens. The property `http-server.authentication.oauth2.audience` has been removed in favor of using `http-server.authentication.oauth2.client-id` as expected by OIDC. To support audiences which are not the `client-id` the new property `http-server.authentication.oauth2.additional-audiences` has been added. Additionally the new property `http-server.authentication.oauth2.issuer` is required; tokens which do not have this issuer will be rejected. ({issue}`8641`)

#8641

[sopel39]

General
* Prevent dropping or renaming object with incompatible SQL command (e.g dropping a view using ``DROP TABLE`` syntax). ({issue}`8869`)

#8869

[phd3]

Elasticsearch
* Add support for reading fields as `JSON` values. ({issue}`7308`)  

#7308, #7373

[hashhar]

Pinot
* Implement aggregation pushdown for `count`, `avg`, `min`, `max`, `sum`, `count(DISTINCT)` and `approx_distinct`.
  It is enabled by default and can be disabled using the configuration property `pinot.aggregation-pushdown.enabled`
  or the session property `aggregation_pushdown_enabled`. ({issue}`4140`)

#4140, #6069

cc: @elonazoulay Does the release note look good to you? Anything I missed?

[sopel39]

General:
* Add `system.metadata.materialized_views` table that list detailed information about materialized views. ({issue}`8796`)

#8796

[sopel39]

General:
* Fix invalid result when two decimals of are added together.
  This happened in certain queries where decimals had different precision.
  ({issue}`8973`)

#8973

[sopel39]

Hive:
* Prevent creating of tables that have column names containing commas, leading or trailing spaces. ({issue}`8954`)

#8954

[ebyhr]

Hive
* Add support for translating Hive views which contain common table expressions. ({issue}`5977`)
* Add support for translating Hive views which contain outer parentheses. ({issue}`8789`)
* Add support for translating Hive views which use `date` function. ({issue}`8789`)
* Add support for translating Hive views which use `pmod` function. ({issue}`8935`)

#5977 #8789 #8935

[kokosing]

JDBC
- Fix getting a list of columns where table or schema name patterns contain upper case value. Also query against `system.jdbc.column` with predicate where table table or schema name with uppercase values was fixed too. ({issue}`8978`

#8978

[ebyhr]

BigQuery
* Fix incorrect result when using BigQuery `TIME` type. ({issue}`8999`)

#8999

[findepi]

General
* Fix incorrect results for queries involving comparison between `varchar` column and a `char` constant. ({issue}`8984`)

#8984

[kokosing]

JDBC
 * Implement `PreparedStatement#getParameterMetaData` method. ({issue}`2978`)

#2978