Fix now() plantime constification with BETWEEN #6491

konskov · 2024-01-03T10:30:04Z

Previously when using BETWEEN ... AND additional constraints
in a WHERE clause, the BETWEEN was not handled correctly because
it was wrapped in a BoolExpr node, which prevented plantime
exclusion.
This commit fixes the handling of this case to allow chunk exclusion
to take place at planning time.
Also, makes sure we use our mock timestamp in all places in tests.
Previously we were using a mix of current_timestamp_mock and now(),
which was returning unexpected/incorrect results.

codecov · 2024-01-03T10:37:04Z

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (92e9009) 79.74% compared to head (755ae7a) 79.74%.

Files	Patch %	Lines
src/planner/planner.c	87.50%	0 Missing and 3 partials ⚠️
src/time_utils.c	57.14%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6491      +/-   ##
==========================================
- Coverage   79.74%   79.74%   -0.01%     
==========================================
  Files         188      188              
  Lines       36739    36722      -17     
  Branches     9290     9278      -12     
==========================================
- Hits        29297    29283      -14     
- Misses       3110     3112       +2     
+ Partials     4332     4327       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-01-03T15:52:29Z

@gayyappan, @erimatnor: please review this pull request.

Powered by pull-review

akuzm · 2024-01-03T16:06:04Z

tsl/test/shared/sql/constify_now.sql.in

+SET timescaledb.enable_chunk_append TO true;
+SET timescaledb.enable_constraint_aware_append TO true;
+
+-- no startup exclusion should be happening for any of the queries below


It sounds like if the chunks shouldn't be excluded at all, might be more clear to say that they should be excluded at planning time.

akuzm · 2024-01-03T16:14:00Z

tsl/test/shared/sql/constify_now.sql.in

+timestamp BETWEEN now() AND now() - interval '1 day'
+AND rawtag_id = 1 ORDER BY "timestamp" ASC;


So what are you getting here, something like

(BoolExprAnd (OpExpr (= rawtag_id 1)) (BoolExprAnd (>= time now()) (<= time ...)))

Top-level AND should normally be flattened into a list, but ts_constify_now is called before it happens, so this is why we have this problem?

I haven’t yet found where the flattening happens, but yes that is how the expression looks: (pprint(node) in ts_constify_now)

{BOOLEXPR :boolop and :args ( {BOOLEXPR :boolop and :args ( {OPEXPR :opno 1325 :opfuncid 1156 :opresulttype 16 :opretset false :opcollid 0 :inputcollid 0 :args ( {VAR :varno 1 :varattno 1 :vartype 1184 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 1 :varattnosyn 1 :location 26 } {OPEXPR :opno 1329 :opfuncid 1190 :opresulttype 1184 :opretset false :opcollid 0 :inputcollid 0 :args ( {FUNCEXPR :funcid 1299 :funcresulttype 1184 :funcretset false :funcvariadic false :funcformat 0 :funccollid 0 :inputcollid 0 :args <> :location 39 } {CONST :consttype 1186 :consttypmod -1 :constcollid 0 :constlen 16 :constbyval false :constisnull false :location 56 :constvalue 16 [ 0 14 39 7 0 0 0 0 0 0 0 0 0 0 0 0 ] } ) :location 45 } ) :location 31 } {OPEXPR :opno 1323 :opfuncid 1155 :opresulttype 16 :opretset false :opcollid 0 :inputcollid 0 :args ( {VAR :varno 1 :varattno 1 :vartype 1184 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 1 :varattnosyn 1 :location 26 } {FUNCEXPR :funcid 1299 :funcresulttype 1184 :funcretset false :funcvariadic false :funcformat 0 :funccollid 0 :inputcollid 0 :args <> :location 68 } ) :location 31 } ) :location 31 } {OPEXPR :opno 96 :opfuncid 65 :opresulttype 16 :opretset false :opcollid 0 :inputcollid 0 :args ( {VAR :varno 1 :varattno 2 :vartype 23 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 1 :varattnosyn 2 :location 78 } {CONST :consttype 23 :consttypmod -1 :constcollid 0 :constlen 4 :constbyval true :constisnull false :location 82 :constvalue 4 [ 1 0 0 0 0 0 0 0 ] } ) :location 80 } ) :location 74 }

I believe the flattening happens in eval_const_expressions, which gets called after ts_constify_now.

akuzm · 2024-01-03T16:16:08Z

src/planner/constify_now.c

-			{
-				be->args = list_concat(be->args, additions);
-			}
+				be->args = additions;


Is this correct? We should keep the old conditions as well. Can you please try to construct a test: first prepare the statement with the mock now() in the middle of the table, so that not all chunks are excluded. Then move the mock now() to be in the future, and check that all the chunks are excluded at run time when you execute the prepared statement. They won't be excluded if you don't keep the original conditions.

you are right, this test currently fails...

However, it seems that this is also the current behavior wrt prepared statements in the main branch, even for the simplest condition (where time > now()) : looking at the constify_now test output, where we test prepared statements https://github.com/timescale/timescaledb/blob/main/tsl/test/shared/expected/constify_now-15.out#L396-L433 I see that two rows are returned when the current timestamp is moved into the future and only one row should actually be returned

the issue appears to be our mock timestamp. When testing with an example actually using now(), the results returned are the expected ones.
Here is the example I used:

create table test2 (time timestamptz, a int, b int); select create_hypertable('test2', 'time', chunk_time_interval => interval '2 min'); insert into test2 (time, a, b) select t, a, b from generate_series(now() - interval '15 min', now() + interval '14 min', interval '30 sec') g(t) cross join generate_series(1,3,1) a cross join generate_series(1,2,1) b; prepare p2 as select * from test2 where time between now() - interval '2 min' and now() and a = 1; -- execute p2 immediately, then after two minutes: explain execute p2; QUERY PLAN ---------------------------------------------------------------------------------------------------- Custom Scan (ChunkAppend) on test2 (cost=4.27..119.60 rows=8 width=16) Chunks excluded during startup: 6 -> Bitmap Heap Scan on _hyper_1_8_chunk (cost=4.27..14.95 rows=1 width=16) Recheck Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) Filter: (a = 1) -> Bitmap Index Scan on _hyper_1_8_chunk_test2_time_idx (cost=0.00..4.27 rows=9 width=0) Index Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) -> Bitmap Heap Scan on _hyper_1_9_chunk (cost=4.27..14.95 rows=1 width=16) Recheck Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) Filter: (a = 1) -> Bitmap Index Scan on _hyper_1_9_chunk_test2_time_idx (cost=0.00..4.27 rows=9 width=0) Index Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) (12 rows) execute p2; time | a | b -------------------------------+---+--- 2024-01-05 13:03:52.465237+02 | 1 | 1 2024-01-05 13:03:52.465237+02 | 1 | 2 2024-01-05 13:04:22.465237+02 | 1 | 1 2024-01-05 13:04:22.465237+02 | 1 | 2 2024-01-05 13:04:52.465237+02 | 1 | 1 2024-01-05 13:04:52.465237+02 | 1 | 2 2024-01-05 13:05:22.465237+02 | 1 | 1 2024-01-05 13:05:22.465237+02 | 1 | 2 (8 rows) -- after a bit: explain execute p2; QUERY PLAN ----------------------------------------------------------------------------------------------------- Custom Scan (ChunkAppend) on test2 (cost=4.27..119.60 rows=8 width=16) Chunks excluded during startup: 6 -> Bitmap Heap Scan on _hyper_1_9_chunk (cost=4.27..14.95 rows=1 width=16) Recheck Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) Filter: (a = 1) -> Bitmap Index Scan on _hyper_1_9_chunk_test2_time_idx (cost=0.00..4.27 rows=9 width=0) Index Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) -> Bitmap Heap Scan on _hyper_1_10_chunk (cost=4.27..14.95 rows=1 width=16) Recheck Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) Filter: (a = 1) -> Bitmap Index Scan on _hyper_1_10_chunk_test2_time_idx (cost=0.00..4.27 rows=9 width=0) Index Cond: (("time" >= (now() - '00:02:00'::interval)) AND ("time" <= now())) (12 rows) Time: 4.078 ms postgres@postgres[63583]=# execute p2; time | a | b -------------------------------+---+--- 2024-01-05 13:04:22.465237+02 | 1 | 1 2024-01-05 13:04:22.465237+02 | 1 | 2 2024-01-05 13:04:52.465237+02 | 1 | 1 2024-01-05 13:04:52.465237+02 | 1 | 2 2024-01-05 13:05:22.465237+02 | 1 | 1 2024-01-05 13:05:22.465237+02 | 1 | 2 2024-01-05 13:05:52.465237+02 | 1 | 1 2024-01-05 13:05:52.465237+02 | 1 | 2 (8 rows)

testing with actual now() instead of our mock timestamp, it seems the changes in this PR work as expected, but our current_timestamp_mock doesn’t play well with prepared statements. This needs to be fixed. @akuzm do you think I should do this as part of this PR?

@akuzm do you think I should do this as part of this PR?

I think it would be good to fix it here because it is needed to properly test your changes.

akuzm · 2024-01-22T17:25:25Z

sql/updates/latest-dev.sql

@@ -397,3 +397,6 @@ CREATE FUNCTION _timescaledb_functions.constraint_clone(constraint_oid OID,targe
 DROP FUNCTION IF EXISTS _timescaledb_functions.chunks_in;
 DROP FUNCTION IF EXISTS _timescaledb_internal.chunks_in;

+
+CREATE FUNCTION _timescaledb_functions.ts_replace_now_mock()


This should probably be "ts_now_mock()"? The current "ts_replace_now_mock()" sounds like the function that does the replacement, I was staring at it trying to understand why you need it in SQL 😃

Also probably we don't need it in the production schema, but not sure where to put it.

We have a file debug_build_utils.sql that is used for utilities that should only be available in debug builds.

Not sure you need to have this function in the upgrade/downgrade scripts since it is only going to be used for regression test runs in debug builds.

mkindahl

Very nice addition, but suggest that you only add the mock function to debug builds and configure the tests accordingly.

mkindahl · 2024-01-24T08:33:31Z

sql/updates/latest-dev.sql

@@ -397,3 +397,6 @@ CREATE FUNCTION _timescaledb_functions.constraint_clone(constraint_oid OID,targe
 DROP FUNCTION IF EXISTS _timescaledb_functions.chunks_in;
 DROP FUNCTION IF EXISTS _timescaledb_internal.chunks_in;

+
+CREATE FUNCTION _timescaledb_functions.ts_replace_now_mock()


We have a file debug_build_utils.sql that is used for utilities that should only be available in debug builds.

Not sure you need to have this function in the upgrade/downgrade scripts since it is only going to be used for regression test runs in debug builds.

mkindahl · 2024-01-24T08:34:01Z

sql/updates/reverse-dev.sql

@@ -770,3 +770,4 @@ AS 'BEGIN END' LANGUAGE PLPGSQL SET search_path TO pg_catalog,pg_temp;

 DROP FUNCTION IF EXISTS _timescaledb_functions.get_orderby_defaults(regclass,text[]);
 DROP FUNCTION IF EXISTS _timescaledb_functions.get_segmentby_defaults(regclass);
+DROP FUNCTION IF EXISTS _timescaledb_functions.ts_replace_now_mock();


Not sure you need this in the upgrade/downgrade scripts since it will only be used in regression tests for debug builds.

We don't need this function in latest/reverse-dev, removed.

mkindahl · 2024-01-24T08:34:45Z

sql/util_time.sql

+CREATE OR REPLACE FUNCTION _timescaledb_functions.ts_replace_now_mock()
+RETURNS TIMESTAMPTZ AS '@MODULE_PATHNAME@', 'ts_replace_now_mock' LANGUAGE C STABLE STRICT PARALLEL SAFE;


Suggest to move this to debug_build_utils.sql.

Previously when using BETWEEN ... AND additional constraints in a WHERE clause, the BETWEEN was not handled correctly because it was wrapped in a BoolExpr node, which prevented plantime exclusion. The flattening of such expressions happens in `eval_const_expressions` which gets called after our constify_now code. This commit fixes the handling of this case to allow chunk exclusion to take place at planning time. Also, makes sure we use our mock timestamp in all places in tests. Previously we were using a mix of current_timestamp_mock and now(), which was returning unexpected/incorrect results.

@coney21

This release contains performance improvements, an improved hypertable DDL API and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations (100k limit, configurable) * Helper functions for determining compression settings **Removal notice: Multi-node support** TimescaleDB 2.13 is the last version that includes multi-node support. Multi-node support is effectively removed in 2.14 version. Learn more about it [here](docs/MultiNodeDeprecation.md). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include recompress_chunk procedure. Its function will be replaced by compress_chunk function which should work on uncompressed and partially compressed chunk. It should be used to fully compress all chunks. **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * timescale#6325 Add plan-time chunk exclusion for real-time CAggs * timescale#6360 Remove support for creating Continuous Aggregates with old format * timescale#6386 Add functions for determining compression defaults * timescale#6410 Remove multinode public API * timescale#6440 Allow SQLValueFunction pushdown into compressed scan * timescale#6463 Support approximate hypertable size * timescale#6513 Make compression settings per chunk * timescale#6529 Remove reindex_relation from recompression * timescale#6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * timescale#6545 Remove restrictions for changing compression settings * timescale#6566 Limit tuple decompression during DML operations * timescale#6579 Change compress_chunk and decompress_chunk to idempotent version by default * timescale#6608 Add LWLock for OSM usage in loader * timescale#6609 Deprecate recompress_chunk * timescale#6609 Add optional recompress argument to compress_chunk **Bugfixes** * timescale#6541 Inefficient join plans on compressed hypertables. * timescale#6491 Enable now() plantime constification with BETWEEN * timescale#6494 Fix create_hypertable referenced by fk succeeds * timescale#6498 Suboptimal query plans when using time_bucket with query parameters * timescale#6507 time_bucket_gapfill with timezones doesn't handle daylight savings * timescale#6509 Make extension state available through function * timescale#6512 Log extension state changes * timescale#6522 Disallow triggers on CAggs * timescale#6523 Reduce locking level on compressed chunk index during segmentwise recompression * timescale#6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * timescale#6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * timescale#6575 Fix compressed chunk not found during upserts * timescale#6592 Fix recompression policy ignoring partially compressed chunks * timescale#6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

github-actions bot assigned konskov Jan 3, 2024

konskov force-pushed the fix_constify_between branch 5 times, most recently from ee71889 to a07e4b2 Compare January 3, 2024 15:44

konskov changed the title ~~[JUST TESTING] Fix constify between~~ Fix now() plantime constification with BETWEEN Jan 3, 2024

konskov marked this pull request as ready for review January 3, 2024 15:52

github-actions bot requested review from erimatnor and gayyappan January 3, 2024 15:52

akuzm reviewed Jan 3, 2024

View reviewed changes

konskov force-pushed the fix_constify_between branch 13 times, most recently from cb6e424 to 184b796 Compare January 22, 2024 13:27

konskov requested a review from svenklemm January 22, 2024 13:27

konskov force-pushed the fix_constify_between branch from 184b796 to 8856c4b Compare January 22, 2024 13:28

konskov requested a review from akuzm January 22, 2024 15:41

akuzm reviewed Jan 22, 2024

View reviewed changes

akuzm approved these changes Jan 22, 2024

View reviewed changes

gayyappan approved these changes Jan 23, 2024

View reviewed changes

mkindahl reviewed Jan 24, 2024

View reviewed changes

konskov force-pushed the fix_constify_between branch from 8856c4b to d0d5130 Compare January 24, 2024 14:50

konskov force-pushed the fix_constify_between branch from d0d5130 to 755ae7a Compare January 24, 2024 15:04

konskov requested a review from mkindahl January 24, 2024 15:06

mkindahl approved these changes Jan 24, 2024

View reviewed changes

konskov merged commit 929ad41 into timescale:main Jan 24, 2024
44 checks passed

konskov added this to the TimescaleDB 2.14 milestone Jan 29, 2024

antekresic mentioned this pull request Feb 7, 2024

Release 2.14.0 #6613

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix now() plantime constification with BETWEEN #6491

Fix now() plantime constification with BETWEEN #6491

konskov commented Jan 3, 2024 •

edited

Loading

codecov bot commented Jan 3, 2024 •

edited

Loading

github-actions bot commented Jan 3, 2024

akuzm Jan 3, 2024

akuzm Jan 3, 2024

konskov Jan 5, 2024

konskov Jan 8, 2024

akuzm Jan 3, 2024 •

edited

Loading

konskov Jan 4, 2024

konskov Jan 4, 2024

konskov Jan 5, 2024

konskov Jan 8, 2024

akuzm Jan 8, 2024

akuzm Jan 22, 2024

akuzm Jan 22, 2024

mkindahl Jan 24, 2024

mkindahl left a comment

mkindahl Jan 24, 2024

mkindahl Jan 24, 2024

konskov Jan 24, 2024

mkindahl Jan 24, 2024

konskov Jan 24, 2024

		timestamp BETWEEN now() AND now() - interval '1 day'
		AND rawtag_id = 1 ORDER BY "timestamp" ASC;

		CREATE OR REPLACE FUNCTION _timescaledb_functions.ts_replace_now_mock()
		RETURNS TIMESTAMPTZ AS '@MODULE_PATHNAME@', 'ts_replace_now_mock' LANGUAGE C STABLE STRICT PARALLEL SAFE;

Fix now() plantime constification with BETWEEN #6491

Fix now() plantime constification with BETWEEN #6491

Conversation

konskov commented Jan 3, 2024 • edited Loading

codecov bot commented Jan 3, 2024 • edited Loading

Codecov Report

github-actions bot commented Jan 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akuzm Jan 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkindahl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konskov commented Jan 3, 2024 •

edited

Loading

codecov bot commented Jan 3, 2024 •

edited

Loading

akuzm Jan 3, 2024 •

edited

Loading