Release 2.16.0

CHANGELOG.md

@@ -4,6 +4,106 @@
 `psql` with the `-X` flag to prevent any `.psqlrc` commands from
 accidentally triggering the load of a previous DB version.**
 
+
+## 2.16.0 (2024-07-25)
+
+This release contains significant performance improvements when working with compressed data, extended join
+support in continuous aggregates, and the ability to define foreign keys from regular tables towards hypertables.
+We recommend that you upgrade at the next available opportunity.
+
+In TimescaleDB v2.16.0 we:
+
+* Introduce multiple performance focused optimizations for data manipulation operations (DML) over compressed chunks.
+
+  Improved upsert performance by more than 100x in some cases and more than 1000x in some update/delete scenarios.
+
+* Add the ability to define sparse indexes on non-partitioning columns of compressed hypertables
+
+  TimescaleDB v2.16.0 extends chunk exclusion to use those sparse indexes when queries filter on the relevant columns,
+  and prune chunks that do not include any relevant data for calculating the query response.
+
+* Offer new options for use cases that require foreign keys defined.
+
+  You can now add foreign keys from regular tables towards hypertables. We have also removed
+  some really annoying locks in the reverse direction that blocked access to referenced tables
+  while compression was running.
+
+* Extend Continuous Aggregates to support more types of analytical queries.
+
+  More types of joins are supported, additional equality operators on join clauses, and
+  support for joins between multiple regular tables.
+
+**Highlighted features in this release**
+
+* Improved query performance through chunk exclusion on compressed hypertables.
+
+  You can now define sparse indexes on compressed chunks for any column with one of the following
+  integer data types: `smallint`, `int`, `bigint`, `serial`, `bigserial`, `date`, `timestamp`, `timestamptz`.
+
+  After you call `enable_column_stats` on a column, TimescaleDB tracks the min and max values for
+  that column. TimescaleDB uses that information to exclude chunks for queries that filter on that
+  column, and would not find any data in those chunks.
+
+* Improved upsert performance on compressed hypertables.
+
+  By using index scans to verify constraints during inserts on compressed chunks, TimescaleDB speeds
+  up some ON CONFLICT clauses by more than 100x.
+
+* Improved performance of updates, deletes, and inserts on compressed hypertables.
+
+  By filtering data while accessing the compressed data and before decompressing, TimescaleDB has
+  improved performance for updates and deletes on all types of compressed chunks, as well as inserts
+  into compressed chunks with unique constraints.
+
+  By signaling constraint violations without decompressing, or decompressing only when matching
+  records are found in the case of updates, deletes and upserts, TimescaleDB v2.16.0 speeds
+  up those operations more than 1000x in some update/delete scenarios, and 10x for upserts.
+
+* You can add foreign keys from regular tables to hypertables, with support for all types of cascading options.
+  This is useful for hypertables that partition using sequential IDs, and need to reference those IDs from other tables.
+
+* Lower locking requirements during compression for hypertables with foreign keys
+
+  Advanced foreign key handling removes the need for locking referenced tables when new chunks are compressed.
+  DML is no longer blocked on referenced tables while compression runs on a hypertable.
+
+* Improved support for queries on Continuous Aggregates
+
+  `INNER/LEFT` and `LATERAL` joins are now supported. Plus, you can now join with multiple regular tables,
+  and you can have more than one equality operator on join clauses.
+
+**PostgreSQL 13 support removal announcement**
+
+Following the deprecation announcement for PostgreSQL 13 in TimescaleDB v2.13,
+PostgreSQL 13 is no longer supported in TimescaleDB v2.16.
+
+The Currently supported PostgreSQL major versions are 14, 15 and 16.
+
+**Features**
+* #6880: Add support for the array operators used for compressed DML batch filtering.
+* #6895: Improve the compressed DML expression pushdown.
+* #6897: Add support for replica identity on compressed hypertables.
+* #6918: Remove support for PG13.
+* #6920: Rework compression activity wal markers.
+* #6989: Add support for foreign keys when converting plain tables to hypertables.
+* #7020: Add support for the chunk column statistics tracking.
+* #7048: Add an index scan for INSERT DML decompression.
+* #7075: Reduce decompression on the compressed INSERT.
+* #7101: Reduce decompressions for the compressed UPDATE/DELETE.
+* #7108 Reduce decompressions for INSERTs with UNIQUE constraints
+* #7116 Use DELETE instead of TRUNCATE after compression
+* #7134 Refactor foreign key handling for compressed hypertables
+
+**Bugfixes**
+* #7018: Fix `search_path` quoting in the compression defaults function.
+* #7046: Prevent locking for compressed tuples.
+* #7055: Fix the `scankey` for `segment by` columns, where the type `constant` is different to `variable`.
+* #7064: Fix the bug in the default `order by` calculation in compression.
+* #7069: Fix the index column name usage.
+* #7074: Fix the bug in the default `segment by` calculation in compression.
+
+**Thanks**
+
 ## 2.15.3 (2024-07-02)
 
 This release contains bug fixes since the 2.15.2 release.

sql/CMakeLists.txt

@@ -40,11 +40,12 @@ set(MOD_FILES
     updates/2.14.2--2.15.0.sql
     updates/2.15.0--2.15.1.sql
     updates/2.15.1--2.15.2.sql
-    updates/2.15.2--2.15.3.sql)
+    updates/2.15.2--2.15.3.sql
+    updates/2.15.3--2.16.0.sql)
 
 # The downgrade file to generate a downgrade script for the current version, as
 # specified in version.config
-set(CURRENT_REV_FILE reverse-dev.sql)
+set(CURRENT_REV_FILE 2.16.0--2.15.3.sql)
 # Files for generating old downgrade scripts. This should only include files for
 # downgrade from one version to its previous version since we do not support
 # skipping versions when downgrading.
@@ -80,7 +81,8 @@ set(OLD_REV_FILES
     2.15.0--2.14.2.sql
     2.15.1--2.15.0.sql
     2.15.2--2.15.1.sql
-    2.15.3--2.15.2.sql)
+    2.15.3--2.15.2.sql
+    2.16.0--2.15.3.sql)
 
 set(MODULE_PATHNAME "$libdir/timescaledb-${PROJECT_VERSION_MOD}")
 set(LOADER_PATHNAME "$libdir/timescaledb")

sql/updates/2.15.3--2.16.0.sql

@@ -0,0 +1,86 @@
+-- Enable tracking of statistics on a column of a hypertable.
+--
+-- hypertable - OID of the table to which the column belongs to
+-- column_name - The column to track statistics for
+-- if_not_exists - If set, and the entry already exists, generate a notice instead of an error
+CREATE FUNCTION @[email protected]_column_stats(
+    hypertable              REGCLASS,
+    column_name             NAME,
+    if_not_exists           BOOLEAN = FALSE
+) RETURNS TABLE(column_stats_id INT, enabled BOOL)
+AS 'SELECT NULL,NULL' LANGUAGE SQL VOLATILE SET search_path = pg_catalog, pg_temp;
+
+-- Disable tracking of statistics on a column of a hypertable.
+--
+-- hypertable - OID of the table to remove from
+-- column_name - NAME of the column on which the stats are tracked
+-- if_not_exists - If set, and the entry does not exist,
+-- generate a notice instead of an error
+CREATE FUNCTION @[email protected]_column_stats(
+    hypertable              REGCLASS,
+    column_name             NAME,
+    if_not_exists           BOOLEAN = FALSE
+) RETURNS TABLE(hypertable_id INT, column_name NAME, disabled BOOL)
+AS 'SELECT NULL,NULL,NULL' LANGUAGE SQL VOLATILE SET search_path = pg_catalog, pg_temp;
+
+-- Track statistics for columns of chunks from a hypertable.
+-- Currently, we track the min/max range for a given column across chunks.
+-- More statistics (like bloom filters) will be added in the future.
+--
+-- A "special" entry for a column with invalid chunk_id, PG_INT64_MAX,
+-- PG_INT64_MIN indicates that min/max ranges could be computed for this column
+-- for chunks.
+--
+-- The ranges can overlap across chunks. The values could be out-of-date if
+-- modifications/changes occur in the corresponding chunk and such entries
+-- should be marked as "invalid" to ensure that the chunk is in
+-- appropriate state to be able to use these values. Thus these entries
+-- are different from dimension_slice which is used for tracking partitioning
+-- column ranges which have different characteristics.
+--
+-- Currently this catalog supports datatypes like INT, SERIAL, BIGSERIAL,
+-- DATE, TIMESTAMP etc. by storing the ranges in bigint columns. In the
+-- future, we could support additional datatypes (which support btree style
+-- >, <, = comparators) by storing their textual representation.
+--
+CREATE TABLE _timescaledb_catalog.chunk_column_stats (
+  id serial NOT NULL,
+  hypertable_id integer NOT NULL,
+  chunk_id integer NOT NULL,
+  column_name name NOT NULL,
+  range_start bigint NOT NULL,
+  range_end bigint NOT NULL,
+  valid boolean NOT NULL,
+  -- table constraints
+  CONSTRAINT chunk_column_stats_pkey PRIMARY KEY (id),
+  CONSTRAINT chunk_column_stats_ht_id_chunk_id_colname_key UNIQUE (hypertable_id, chunk_id, column_name),
+  CONSTRAINT chunk_column_stats_range_check CHECK (range_start <= range_end),
+  CONSTRAINT chunk_column_stats_hypertable_id_fkey FOREIGN KEY (hypertable_id) REFERENCES _timescaledb_catalog.hypertable (id),
+  CONSTRAINT chunk_column_stats_chunk_id_fkey FOREIGN KEY (chunk_id) REFERENCES _timescaledb_catalog.chunk (id)
+);
+
+SELECT pg_catalog.pg_extension_config_dump('_timescaledb_catalog.chunk_column_stats', '');
+
+SELECT pg_catalog.pg_extension_config_dump(pg_get_serial_sequence('_timescaledb_catalog.chunk_column_stats', 'id'), '');
+
+GRANT SELECT ON _timescaledb_catalog.chunk_column_stats TO PUBLIC;
+GRANT SELECT ON _timescaledb_catalog.chunk_column_stats_id_seq TO PUBLIC;
+
+-- Remove foreign key constraints from compressed chunks
+DO $$
+DECLARE
+  conrelid regclass;
+  conname name;
+BEGIN
+  FOR conrelid, conname IN
+  SELECT
+    con.conrelid::regclass,
+    con.conname
+  FROM _timescaledb_catalog.chunk ch
+  JOIN pg_constraint con ON con.conrelid = format('%I.%I',schema_name,table_name)::regclass AND con.contype='f'
+  WHERE NOT ch.dropped AND EXISTS(SELECT FROM _timescaledb_catalog.chunk ch2 WHERE NOT ch2.dropped AND ch2.compressed_chunk_id=ch.id)
+  LOOP
+    EXECUTE format('ALTER TABLE %s DROP CONSTRAINT %I', conrelid, conname);
+  END LOOP;
+END $$;
+

sql/updates/2.16.0--2.15.3.sql

@@ -0,0 +1,24 @@
+DROP FUNCTION IF EXISTS _timescaledb_functions.cagg_get_bucket_function_info(INTEGER);
+-- remove chunk column statistics related objects
+DROP FUNCTION IF EXISTS @[email protected]_column_stats(REGCLASS, NAME, BOOLEAN);
+DROP FUNCTION IF EXISTS @[email protected]_column_stats(REGCLASS, NAME, BOOLEAN);
+ALTER EXTENSION timescaledb DROP TABLE _timescaledb_catalog.chunk_column_stats;
+ALTER EXTENSION timescaledb DROP SEQUENCE _timescaledb_catalog.chunk_column_stats_id_seq;
+DROP TABLE IF EXISTS _timescaledb_catalog.chunk_column_stats;
+
+-- Add foreign key constraints back to compressed chunks
+DO $$
+DECLARE
+  chunkrelid regclass;
+  conname name;
+  conoid oid;
+BEGIN
+  FOR chunkrelid, conname, conoid IN
+  SELECT format('%I.%I',ch.schema_name,ch.table_name)::regclass, con.conname, con.oid
+  FROM _timescaledb_catalog.hypertable ht
+  JOIN pg_constraint con ON con.contype = 'f' AND con.conrelid=format('%I.%I',ht.schema_name,ht.table_name)::regclass
+  JOIN _timescaledb_catalog.chunk ch on ch.hypertable_id=ht.compressed_hypertable_id and not ch.dropped
+  LOOP
+    EXECUTE format('ALTER TABLE %s ADD CONSTRAINT %I %s', chunkrelid, conname, pg_get_constraintdef(conoid));
+  END LOOP;
+END $$;

sql/updates/latest-dev.sql

@@ -1,86 +0,0 @@
--- Enable tracking of statistics on a column of a hypertable.
---
--- hypertable - OID of the table to which the column belongs to
--- column_name - The column to track statistics for
--- if_not_exists - If set, and the entry already exists, generate a notice instead of an error
-CREATE FUNCTION @[email protected]_column_stats(
-    hypertable              REGCLASS,
-    column_name             NAME,
-    if_not_exists           BOOLEAN = FALSE
-) RETURNS TABLE(column_stats_id INT, enabled BOOL)
-AS 'SELECT NULL,NULL' LANGUAGE SQL VOLATILE SET search_path = pg_catalog, pg_temp;
-
--- Disable tracking of statistics on a column of a hypertable.
---
--- hypertable - OID of the table to remove from
--- column_name - NAME of the column on which the stats are tracked
--- if_not_exists - If set, and the entry does not exist,
--- generate a notice instead of an error
-CREATE FUNCTION @[email protected]_column_stats(
-    hypertable              REGCLASS,
-    column_name             NAME,
-    if_not_exists           BOOLEAN = FALSE
-) RETURNS TABLE(hypertable_id INT, column_name NAME, disabled BOOL)
-AS 'SELECT NULL,NULL,NULL' LANGUAGE SQL VOLATILE SET search_path = pg_catalog, pg_temp;
-
--- Track statistics for columns of chunks from a hypertable.
--- Currently, we track the min/max range for a given column across chunks.
--- More statistics (like bloom filters) will be added in the future.
---
--- A "special" entry for a column with invalid chunk_id, PG_INT64_MAX,
--- PG_INT64_MIN indicates that min/max ranges could be computed for this column
--- for chunks.
---
--- The ranges can overlap across chunks. The values could be out-of-date if
--- modifications/changes occur in the corresponding chunk and such entries
--- should be marked as "invalid" to ensure that the chunk is in
--- appropriate state to be able to use these values. Thus these entries
--- are different from dimension_slice which is used for tracking partitioning
--- column ranges which have different characteristics.
---
--- Currently this catalog supports datatypes like INT, SERIAL, BIGSERIAL,
--- DATE, TIMESTAMP etc. by storing the ranges in bigint columns. In the
--- future, we could support additional datatypes (which support btree style
--- >, <, = comparators) by storing their textual representation.
---
-CREATE TABLE _timescaledb_catalog.chunk_column_stats (
-  id serial NOT NULL,
-  hypertable_id integer NOT NULL,
-  chunk_id integer NOT NULL,
-  column_name name NOT NULL,
-  range_start bigint NOT NULL,
-  range_end bigint NOT NULL,
-  valid boolean NOT NULL,
-  -- table constraints
-  CONSTRAINT chunk_column_stats_pkey PRIMARY KEY (id),
-  CONSTRAINT chunk_column_stats_ht_id_chunk_id_colname_key UNIQUE (hypertable_id, chunk_id, column_name),
-  CONSTRAINT chunk_column_stats_range_check CHECK (range_start <= range_end),
-  CONSTRAINT chunk_column_stats_hypertable_id_fkey FOREIGN KEY (hypertable_id) REFERENCES _timescaledb_catalog.hypertable (id),
-  CONSTRAINT chunk_column_stats_chunk_id_fkey FOREIGN KEY (chunk_id) REFERENCES _timescaledb_catalog.chunk (id)
-);
-
-SELECT pg_catalog.pg_extension_config_dump('_timescaledb_catalog.chunk_column_stats', '');
-
-SELECT pg_catalog.pg_extension_config_dump(pg_get_serial_sequence('_timescaledb_catalog.chunk_column_stats', 'id'), '');
-
-GRANT SELECT ON _timescaledb_catalog.chunk_column_stats TO PUBLIC;
-GRANT SELECT ON _timescaledb_catalog.chunk_column_stats_id_seq TO PUBLIC;
-
--- Remove foreign key constraints from compressed chunks
-DO $$
-DECLARE
-  conrelid regclass;
-  conname name;
-BEGIN
-  FOR conrelid, conname IN
-  SELECT
-    con.conrelid::regclass,
-    con.conname
-  FROM _timescaledb_catalog.chunk ch
-  JOIN pg_constraint con ON con.conrelid = format('%I.%I',schema_name,table_name)::regclass AND con.contype='f'
-  WHERE NOT ch.dropped AND EXISTS(SELECT FROM _timescaledb_catalog.chunk ch2 WHERE NOT ch2.dropped AND ch2.compressed_chunk_id=ch.id)
-  LOOP
-    EXECUTE format('ALTER TABLE %s DROP CONSTRAINT %I', conrelid, conname);
-  END LOOP;
-END $$;
-

sql/updates/reverse-dev.sql

@@ -1,24 +0,0 @@
-DROP FUNCTION IF EXISTS _timescaledb_functions.cagg_get_bucket_function_info(INTEGER);
--- remove chunk column statistics related objects
-DROP FUNCTION IF EXISTS @[email protected]_column_stats(REGCLASS, NAME, BOOLEAN);
-DROP FUNCTION IF EXISTS @[email protected]_column_stats(REGCLASS, NAME, BOOLEAN);
-ALTER EXTENSION timescaledb DROP TABLE _timescaledb_catalog.chunk_column_stats;
-ALTER EXTENSION timescaledb DROP SEQUENCE _timescaledb_catalog.chunk_column_stats_id_seq;
-DROP TABLE IF EXISTS _timescaledb_catalog.chunk_column_stats;
-
--- Add foreign key constraints back to compressed chunks
-DO $$
-DECLARE
-  chunkrelid regclass;
-  conname name;
-  conoid oid;
-BEGIN
-  FOR chunkrelid, conname, conoid IN
-  SELECT format('%I.%I',ch.schema_name,ch.table_name)::regclass, con.conname, con.oid
-  FROM _timescaledb_catalog.hypertable ht
-  JOIN pg_constraint con ON con.contype = 'f' AND con.conrelid=format('%I.%I',ht.schema_name,ht.table_name)::regclass
-  JOIN _timescaledb_catalog.chunk ch on ch.hypertable_id=ht.compressed_hypertable_id and not ch.dropped
-  LOOP
-    EXECUTE format('ALTER TABLE %s ADD CONSTRAINT %I %s', chunkrelid, conname, pg_get_constraintdef(conoid));
-  END LOOP;
-END $$;

version.config

@@ -1,3 +1,3 @@
-version = 2.16.0-dev
+version = 2.16.0
 update_from_version = 2.15.3
 downgrade_to_version = 2.15.3