[xCluster] Replication timeouts due to copy commands #23173
Assignees
Labels
2024.1_blocker
2024.1.2_blocker
2024.2_blocker
area/docdb
YugabyteDB core features
kind/bug
This issue is a bug
priority/medium
Medium priority issue
qa_automation
Bugs identified via itest-system, LST, Stress automation or causing automation failures
qa_itest-system
Bugs identified in itest-system automation
QA
QA filed bugs
Comments
agsh-yb
added
area/docdb
yugabyte-ci
added
kind/bug
hari90
changed the title
|
Why does this happen? |
jasonyb
pushed a commit
that referenced
this issue
Jul 19, 2024
Summary: a64fbfc [#23120] YSQL: Do not wait for a safe snapshot in serializable read only deferrable mode. 9f82c01 [#23188] DocDB: Persist new colocated_id mapping discovered as part of processing CHANGE_METADATA_OP in xCluster ClusterConfig. 1522522 Update third-party dependencies to remove CentOS 7 cd7410c 2.18.1.1 release notes (#23241) 550458d [#23047] docdb: Fix cotable ids in flushed frontier at restore 225ddfe [PLAT-14700] Make node-agent error message on installation to be more precise 9c9a059 [PLAT-14563] Fixing disk mounting logic to use mount points defined in the config 84eefbd [DOC-368] Azure workload identity docs changes (#22881) 781af0d [#23183] xCluster: Move Setup, Bootstrap, Alter and Delete Target Replication functions out of CatalogManager 1d646b1 [docs] Add ysql_output_buffer_size to yb-tserver config reference. (#23233) 2d95dd2 Fix the issue yaml parsing dd9e85b [PLAT-14534]Add regex match for GCP Instance template d3bba18 update diagram (#23245) 78b317c [/PLAT-14708] Fix JSON field name in TaskInfo query ac9164b [#23173] DocDB: Allow large bytes to be passed to RateLimiter Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36723
hari90
added a commit
that referenced
this issue
Jul 19, 2024
…Limiter Summary: Original commit: ac9164b / D36703 RateLimiter has a debug assert that you cannot `Request` more than `GetSingleBurstBytes`. In release mode we do not perform this check and any call gets stuck forever. This change allows large bytes to be requested on RateLimiter. It does so by breaking requests larger than `GetSingleBurstBytes` into multiple smaller requests. This change is a temporary fix to allow xCluster to operate without any issues. RocksDB RateLimiter has multiple enhancements over the years that would help avoid this and more starvation issues. Ex: facebook/rocksdb@cb2476a. We should consider pulling in those changes. Fixes #23173 Jira: DB-12112 Test Plan: RateLimiterTest.LargeRequests Reviewers: slingam Reviewed By: slingam Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36720
hari90
added a commit
that referenced
this issue
Jul 19, 2024
…Limiter Summary: Original commit: ac9164b / D36703 RateLimiter has a debug assert that you cannot `Request` more than `GetSingleBurstBytes`. In release mode we do not perform this check and any call gets stuck forever. This change allows large bytes to be requested on RateLimiter. It does so by breaking requests larger than `GetSingleBurstBytes` into multiple smaller requests. This change is a temporary fix to allow xCluster to operate without any issues. RocksDB RateLimiter has multiple enhancements over the years that would help avoid this and more starvation issues. Ex: facebook/rocksdb@cb2476a. We should consider pulling in those changes. Fixes #23173 Jira: DB-12112 Test Plan: RateLimiterTest.LargeRequests Reviewers: slingam Reviewed By: slingam Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36721
hari90
added a commit
that referenced
this issue
Jul 19, 2024
…teLimiter Summary: Original commit: ac9164b / D36703 RateLimiter has a debug assert that you cannot `Request` more than `GetSingleBurstBytes`. In release mode we do not perform this check and any call gets stuck forever. This change allows large bytes to be requested on RateLimiter. It does so by breaking requests larger than `GetSingleBurstBytes` into multiple smaller requests. This change is a temporary fix to allow xCluster to operate without any issues. RocksDB RateLimiter has multiple enhancements over the years that would help avoid this and more starvation issues. Ex: facebook/rocksdb@cb2476a. We should consider pulling in those changes. Fixes #23173 Jira: DB-12112 Test Plan: RateLimiterTest.LargeRequests Reviewers: slingam Reviewed By: slingam Subscribers: ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36722
vaibhav-yb
pushed a commit
to vaibhav-yb/yugabyte-db
that referenced
this issue
Jul 23, 2024
Summary: RateLimiter has a debug assert that you cannot `Request` more than `GetSingleBurstBytes`. In release mode we do not perform this check and any call gets stuck forever. This change allows large bytes to be requested on RateLimiter. It does so by breaking requests larger than `GetSingleBurstBytes` into multiple smaller requests. This change is a temporary fix to allow xCluster to operate without any issues. RocksDB RateLimiter has multiple enhancements over the years that would help avoid this and more starvation issues. Ex: facebook/rocksdb@cb2476a. We should consider pulling in those changes. Fixes yugabyte#23173 Jira: DB-12112 Test Plan: RateLimiterTest.LargeRequests Reviewers: slingam Reviewed By: slingam Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D36703
jasonyb
pushed a commit
that referenced
this issue
Jul 23, 2024
…ter-merge Merge YB master commit ac9164b titled [#23173] DocDB: Allow large bytes to be passed to RateLimiter and committed 2024-07-19T11:00:31-07:00 into YB pg15. YB pg15 initial merge refers to 55782d5. - BasePgSQLTest.java: - MAX_ATTEMPTS_TO_DESTROY_CONTROL_CONN: adjacent conflict between YB master 58b746e adding this variable and YB pg15 6e887a9 removing FailOnConflictTestGflags. - TestYsqlUpgrade.java: - migratingIsEquivalentToReinitdb: YB master 2ec9224 moves this test lower while YB pg15 4638d7d adds @ignore line. - indexam.c: - yb_index_getbitmap: adjacent conflict between YB master b8fedf6 removing third argument to function call and upstream PG 346ed70b0ad7efc574711a97812692dab4542712 changing rd_amroutine to rd_indam. - xact.c: - TopTransactionStateData: YB master 3494424 adds false ybIsInternalRcSubTransaction at the end of the struct. Upstream PG 98afa68d935215e8ab2c7de7fb45972a03e3da26 changes the style. As part of merging, remove the redundant code comments. - bootstrap.c: - BootstrapModeMain: YB pg15 3e44ae9 changes YbFirstBootstrapObjectId to FirstGenbkiObjectId and reindents all arguments of the YBCCreateDatabase function call. YB master 8becd35 removes "template0" and changes the last argument of the same function call. Merge. - tablecmds.c: - declarations: YB master 19ab966 adds a new parameter yb_tab to the function ATExecDropColumn. Adjacent lines conflict with changes to surrounding functions by upstream PG. - ATExecDropColumn: upstream PG commit 1df5875d39383b3981b804666ee1f4b0ff65943f changes to use performMultipleDeletions instead of performDeletion. YB master commit 7c8343d changes the performDeletion call to use YB_SKIP_YB_DROP_COLUMN as the flag parameter. Resolve the conflict by using YB_SKIP_YB_DROP_COLUMN as the flag parameter in the performMultipleDeletions call. - nodeValuesscan.c: - ExecInitValuesScan: conflict with YB master 471c7c4 which is an upstream PG point import of an old commit already present in REL_15_2. Discard the incoming master change. - nodeYbBitmapIndexscan.c: - MultiExecYbBitmapIndexScan: YB pg15 6368931 changes ybscan->relation to ybscan->rs_base.rs_rd while YB master b8fedf6 removes this code. Remove it. - misc.c: - YB master 0a73bdd is an import of upstream PG e082734c8e78e6622a0422e612a870278721e83f on REL_11_STABLE. The REL_15_STABLE equivalent is 595c988c90024acdc56479d0a631a232a9d19e36, and it does not affect this file since the functions have moved to signalfuncs.c. - signalfuncs.c: - pg_signal_backend: apply the 595c988c90024acdc56479d0a631a232a9d19e36 patch here (see misc.c explanation). - syscache.c: - includes: adjacent conflict. YB pg15 initial merge moves genam.h and heapam.h includes to new Yugabyte includes section. YB master 968e831 adds assert.h include. - yb_cache_index_name_table: - Adjacent conflict between YB pg15 bea1ffb removing YbPinnedObjectKey, YbPinnedObjectsCacheData, YbPinnedObjectsCache and YB master 968e831 adding yb_cache_index_name_table. - Between PG 11.2 and 15.2, six additional catcache entries need to be added into the yb_cache_index_name_table array: - "pg_parameter_acl_parname_index", - "pg_parameter_acl_oid_index", - "pg_publication_namespace_oid_index", - "pg_publication_namespace_pnnspid_pnpubid_index", - "pg_range_rngmultitypid_index", - "pg_statistic_ext_data_stxoid_inh_index", - pg_dump.c: - help: adjacent conflict between YB master eb77547 adding --read-time option and YB pg15 7e413a0f82c8f8e9abb3981839191414de3aae08 adding --rows-per-insert option. - execnodes.h: - YbBitmapIndexScanState: adjacent conflict between YB master 7e413a0f82c8f8e9abb3981839191414de3aae08 adding biss_requires_recheck and YB pg15 cd1df46 changing the type biss_ScanDesc. - ValuesScanState: YB master 471c7c4 is an upstream PG point import of an old commit already present in REL_15_2. Discard the incoming master change. - catcache.h: - SetCatCacheTuple: adjacent conflict between YB pg15 initial merge adding SetCatCacheTuple, SetCatCacheList, RelationHasCachedLists and YB master 968e831 renaming GetCatCacheMisses to YbGetCatCacheMisses and adding YbGetCatCacheIdMisses. - syscache.h - YbInitPinnedCacheIfNeeded: adjacent conflict between YB pg15 bea1ffb removing YbInitPinnedCacheIfNeeded, YbResetPinnedCache, YbIsObjectPinned, YbPinObjectIfNeeded and YB master 968e831 adding YbCheckCatalogCacheIndexNameTable and YbGetCatalogCacheIndexName. - privileges.out: - pg_signal_backend can't signal superusers: YB master 0a73bdd is an import of upstream PG e082734c8e78e6622a0422e612a870278721e83f on REL_11_STABLE. The REL_15_STABLE equivalent is 595c988c90024acdc56479d0a631a232a9d19e36. Update the test in accordance with that, which should apply cleanly. - yb_pg_privileges.out: (similar to privileges.out with trivial conflict) - privileges.sql: (similar to privileges.out) - yb_pg_privileges.sql: (similar to yb_pg_privileges.out) - yb_pg_dml_serial_schedule: - YB pg15 4f17a04 splits yb_pg_dml_serial_schedule to that and yb_dml_serial_schedule. YB master 471c7c4 adds test yb_dml to yb_pg_dml_serial_schedule. Place it in yb_dml_serial_schedule instead. - yb_dml_serial_schedule: (see above) - source_ddl_end_handler.c: - ProcessSourceEventTriggerDDLCommands: YB master 8142fdc renames some variables from camel case to snake case. YB pg15 07f3c7d adapts this code in accordance with upstream PG 2f9661311b83dc481fc19f6e3bda015392010a40 changing char* to CommandTag enum. - ybc_pg_typedefs.h: - YBCPgGFlagsAccessor: YB master 1773ae2 adds ysql_use_fast_backward_scan while both YB pg15 fedbdac and YB master 56d2d2d add TEST_generate_ybrowid_sequentially. Take both. - ybc_pggate.cc: (similar to ybc_pg_typedefs.h) - yb_pg_metrics.c: - statementType: YB master 968e831 adds CatCacheMisses_# entries. Due to six additional cache entries between PG 11.2 and 15.2 (see syscache.c explanation), add six more entries here: - CatCacheMisses_79, - CatCacheMisses_80, - CatCacheMisses_81, - CatCacheMisses_82, - CatCacheMisses_83, - CatCacheMisses_84, - pg_proc.dat: - yb_reset_analyze_statistics: YB master 9f2f70e introduces this code. Upstream PG 814c9019aa7b0ce3a4f19415023a256fdd717b20 changes prolang => '14' to prolang => 'sql' Do the same here to avoid the following initdb error: unresolved OID reference "14" in pg_proc.dat field prolang line 12041 - yb_system_views.sql: - yb_reset_analyze_statistics: YB master 9f2f70e introduces this function. - Upstream PG 6cbfb784c3c91146148a76d50cda6f69ae6a79fb splits pg_statistic_ext to that and pg_statistic_ext_data, so adapt the function accordingly. - Upstream PG 3d351d916b20534f973eda760cde17d96545d4c4 makes reltuples = -1 by default instead of 0. Adapt the function accordingly. - YB master appears to miss the case for handling partitioned tables. This likely didn't surface in tests on master because reltuples = 0 by default in master. - V52__22028__yb_reset_analyze_statistics.sql: - yb_reset_analyze_statistics: (same) - yb_reset_analyze.sql, yb_reset_analyze.out: - This test is introduced by YB master 9f2f70e. Adapt to PG 15: - Add GRANT ALL ON SCHEMA public for non-super users. - Adapt to pg_statistic_ext_data change by upstream PG 6cbfb784c3c91146148a76d50cda6f69ae6a79fb similarly to the above resolution. - Adapt to reltuples = -1 change by upstream PG 3d351d916b20534f973eda760cde17d96545d4c4 similar to the above resolution. - Update an expected output for pg_database ncolstats from 13 to 16 since pg_database now has more columns. - explain.c: - yb_fix_indexpr_mutator: YB master de8c392 adds this function. Change varnoold to varnosyn in accordance with upstream PG 9ce77d75c5ab094637cc4a446296dc3be6e3c221. - ybccmds.c: - YBCDropColumn: YB master 19ab966 adds a raw_parser call. Upstream PG 844fe9f159a948377907a63d0ef3fb16dc51ce50 adds a new argument for raw_parser. Pass the appropriate argument RAW_PARSE_DEFAULT. - yb_alter_table_rewrite.out: - ALTER TABLE test_drop_pk_column_part DROP COLUMN id: YB master 19ab966 adds this test. Change the error message according to upstream PG a0555ddab9b672a04681ce0d9f6c94104c01b15f. - nodeYbBitmapTablescan.c: - YbBitmapTableNext: YB master b8fedf6 adds CreateTemplateTupleDesc and ExecInitScanTupleSlot calls. The signatures of those functions changes between PG 11.2 and 15.2. Follow a similar transformation as in nodeYbSeqscan.c YbSeqNext. - procarray.c: - GetSnapshotDataReuse: YB master edbd06b sets yb_read_time_point_handle in GetSnapshotData. Upstream PG 623a9ba79bbdd11c5eccb30b8bd5c446130e521c adds an additional GetSnapshotDataReuse alongside GetSnapshotData. Set yb_read_time_point_handle there as well. - yb_query_diagnostics.c: - BuiltinTrancheNames: YB master 47f40bb adds LWLockRegisterTranche call. Upstream PG 29c3e2dd5a6aeaf1a23d7d83d665501e2dcc6955 replaces that with a BuiltinTrancheNames array. Translate to lwlock.c BuiltinTrancheNames. - DumpToFile: YB master 47f40bb adds FileWrite function call. Upstream PG c24dcd0cfd949bdf245814c4c2b3df828ee7db36 adds extra offset parameter. Pass 0. - lwlock.c: - BuiltinTrancheNames: (see yb_query_diagnostics.c BuiltinTrancheNames) - hba.c: - get_hba_options: YB master 18bb9b8 adds if (hba->yb_jwt_jwks_url) block to this function. The function moves to hbafuncs.c by upstream PG d4781d8873f8c3fc8b0957cc03ce91627576cf36. - hbafuncs.c: (see hba.c) - wait_event.h: - WaitEventActivity: YB master 47f40bb adds YB_WAIT_EVENT_QUERY_DIAGNOSTICS_MAIN to WaitEventActivity in pgstat.h. Upstream PG a333476b925134f6185037eaff3424c07a9f466f moves this code to wait_event.h. - wait_event.c: - pgstat_get_wait_activity: (similar to wait_event.h) - yb_index_schedule: - YB master 56d2d2d removes yb_index_serial_schedule and yb_index_serial2_schedule and makes yb_index_schedule and yb_pg_index_schedule. YB pg15 9690419 adds yb_index_including test to yb_index_serial_schedule. Move that test to yb_index_schedule. - ipci.c - CreateSharedMemoryAndSemaphores: upstream PG 0bd305ee1d427ef29f5fa4fa20567e3b3f5ff792 moves code into CalculateShmemSize. YB 47f40bb adds code there. Move that code to the new location. Note also upstream PG 4f2400cb3f10aa79f99fba680c198237da28dd38 modifying an adjacent line. - backend/utils/misc/MakeFile: - OBJS: YB master 47f40bb adds yb_query_diagnostics.o to the list. YB pg15 changes the style. - snapmgr.c: - RestoreSnapshot: adjacent conflict between YB master edbd06b defining yb_read_time_point_handle and upstream PG 623a9ba79bbdd11c5eccb30b8bd5c446130e521c defining snapXactCompletionCount. - dbcommands.c: - createdb: upstream PG aa01051418f10afbdfa781b8dc109615ca785ff9 creates if (OidIsValid(dboid)) and else case, moving existing code into the else case. YB master dee7691 adds "CREATE DATABASE using templates other than template0 and template1" code in the same place. Order the YB master added code before the if-else. Secondly, the same YB master commit modifies a YBCCreateDatabase function call. There are two such calls here, one for each branch of the if-else (for history, follow YB pg15 07f3c7d commit message). Modify each call. - snapshot.h: - struct YbReadTimePointHandle: adjacent conflict between upstream PG 63746189b23815415cacc715fdc4f6b991f1a5e7 removing SnapshotSatisfiesFunc code and YB master edbd06b adding YbReadTimePointHandle code. - struct SnapshotData: adjacent conflict between upstream PG 623a9ba79bbdd11c5eccb30b8bd5c446130e521c adding snapXactCompletionCount field and YB master edbd06b adding yb_read_time_point_handle field. - pg_dml.cc: - BindColumn: adjacent lines conflict between YB pg15 0df60d3 changing code in function and YB master 47da28c changing function signature style. - has_aggregate_targets: adjacent lines conflict between YB pg15 5e6edbf removing has_system_targets and YB master 47da28c removing both has_aggregate_targets and has_system_targets. - pg_dml.h: - has_secondary_index_with_doc_op: adjacent lines conflict betewen YB pg15 5e6edbf removing has_system_targets and YB master 47da28c adding [[nodiscard]] to multiple functions (including has_system_targets) and filling in definitions for them. - pg_sample.cc: - PgSample::InitRandomState: YB pg15 055f04e splits rand_state to rand_state_s0 and rand_state_s1, and YB master 47da28c makes other changes to the function. - PrepareSamplingState: YB pg15 055f04e splits rand_state to rand_state_s0 and rand_state_s1, and YB master 47da28c moves this code into class PgSamplePicker. - pg_sample.h: - PrepareSamplingState: YB pg15 055f04e splits rand_state to rand_state_s0 and rand_state_s1, and YB master 47da28c deletes the code since pg_sample.cc now has the function defined in the class. - pg15_tests/passing_tests.tsv: - TestPgRegressWaitQueues: YB master 3494424 causes this test to crash, so remove it from the list to investigate later.
jasonyb
pushed a commit
that referenced
this issue
Jul 23, 2024
…a854215' into pg15 Summary: Merge YB master commit ac9164b titled [#23173] DocDB: Allow large bytes to be passed to RateLimiter and committed 2024-07-19T11:00:31-07:00 into YB pg15. YB pg15 initial merge refers to 55782d5. - BasePgSQLTest.java: - MAX_ATTEMPTS_TO_DESTROY_CONTROL_CONN: adjacent conflict between YB master 58b746e adding this variable and YB pg15 6e887a9 removing FailOnConflictTestGflags. - TestYsqlUpgrade.java: - migratingIsEquivalentToReinitdb: YB master 2ec9224 moves this test lower while YB pg15 4638d7d adds @ignore line. - indexam.c: - yb_index_getbitmap: adjacent conflict between YB master b8fedf6 removing third argument to function call and upstream PG 346ed70b0ad7efc574711a97812692dab4542712 changing rd_amroutine to rd_indam. - xact.c: - TopTransactionStateData: YB master 3494424 adds false ybIsInternalRcSubTransaction at the end of the struct. Upstream PG 98afa68d935215e8ab2c7de7fb45972a03e3da26 changes the style. As part of merging, remove the redundant code comments. - bootstrap.c: - BootstrapModeMain: YB pg15 3e44ae9 changes YbFirstBootstrapObjectId to FirstGenbkiObjectId and reindents all arguments of the YBCCreateDatabase function call. YB master 8becd35 removes "template0" and changes the last argument of the same function call. Merge. - tablecmds.c: - declarations: YB master 19ab966 adds a new parameter yb_tab to the function ATExecDropColumn. Adjacent lines conflict with changes to surrounding functions by upstream PG. - ATExecDropColumn: upstream PG commit 1df5875d39383b3981b804666ee1f4b0ff65943f changes to use performMultipleDeletions instead of performDeletion. YB master commit 7c8343d changes the performDeletion call to use YB_SKIP_YB_DROP_COLUMN as the flag parameter. Resolve the conflict by using YB_SKIP_YB_DROP_COLUMN as the flag parameter in the performMultipleDeletions call. - nodeValuesscan.c: - ExecInitValuesScan: conflict with YB master 471c7c4 which is an upstream PG point import of an old commit already present in REL_15_2. Discard the incoming master change. - nodeYbBitmapIndexscan.c: - MultiExecYbBitmapIndexScan: YB pg15 6368931 changes ybscan->relation to ybscan->rs_base.rs_rd while YB master b8fedf6 removes this code. Remove it. - misc.c: - YB master 0a73bdd is an import of upstream PG e082734c8e78e6622a0422e612a870278721e83f on REL_11_STABLE. The REL_15_STABLE equivalent is 595c988c90024acdc56479d0a631a232a9d19e36, and it does not affect this file since the functions have moved to signalfuncs.c. - signalfuncs.c: - pg_signal_backend: apply the 595c988c90024acdc56479d0a631a232a9d19e36 patch here (see misc.c explanation). - syscache.c: - includes: adjacent conflict. YB pg15 initial merge moves genam.h and heapam.h includes to new Yugabyte includes section. YB master 968e831 adds assert.h include. - yb_cache_index_name_table: - Adjacent conflict between YB pg15 bea1ffb removing YbPinnedObjectKey, YbPinnedObjectsCacheData, YbPinnedObjectsCache and YB master 968e831 adding yb_cache_index_name_table. - Between PG 11.2 and 15.2, six additional catcache entries need to be added into the yb_cache_index_name_table array: - "pg_parameter_acl_parname_index", - "pg_parameter_acl_oid_index", - "pg_publication_namespace_oid_index", - "pg_publication_namespace_pnnspid_pnpubid_index", - "pg_range_rngmultitypid_index", - "pg_statistic_ext_data_stxoid_inh_index", - pg_dump.c: - help: adjacent conflict between YB master eb77547 adding --read-time option and YB pg15 7e413a0f82c8f8e9abb3981839191414de3aae08 adding --rows-per-insert option. - execnodes.h: - YbBitmapIndexScanState: adjacent conflict between YB master 7e413a0f82c8f8e9abb3981839191414de3aae08 adding biss_requires_recheck and YB pg15 cd1df46 changing the type biss_ScanDesc. - ValuesScanState: YB master 471c7c4 is an upstream PG point import of an old commit already present in REL_15_2. Discard the incoming master change. - catcache.h: - SetCatCacheTuple: adjacent conflict between YB pg15 initial merge adding SetCatCacheTuple, SetCatCacheList, RelationHasCachedLists and YB master 968e831 renaming GetCatCacheMisses to YbGetCatCacheMisses and adding YbGetCatCacheIdMisses. - syscache.h - YbInitPinnedCacheIfNeeded: adjacent conflict between YB pg15 bea1ffb removing YbInitPinnedCacheIfNeeded, YbResetPinnedCache, YbIsObjectPinned, YbPinObjectIfNeeded and YB master 968e831 adding YbCheckCatalogCacheIndexNameTable and YbGetCatalogCacheIndexName. - privileges.out: - pg_signal_backend can't signal superusers: YB master 0a73bdd is an import of upstream PG e082734c8e78e6622a0422e612a870278721e83f on REL_11_STABLE. The REL_15_STABLE equivalent is 595c988c90024acdc56479d0a631a232a9d19e36. Update the test in accordance with that, which should apply cleanly. - yb_pg_privileges.out: (similar to privileges.out with trivial conflict) - privileges.sql: (similar to privileges.out) - yb_pg_privileges.sql: (similar to yb_pg_privileges.out) - yb_pg_dml_serial_schedule: - YB pg15 4f17a04 splits yb_pg_dml_serial_schedule to that and yb_dml_serial_schedule. YB master 471c7c4 adds test yb_dml to yb_pg_dml_serial_schedule. Place it in yb_dml_serial_schedule instead. - yb_dml_serial_schedule: (see above) - source_ddl_end_handler.c: - ProcessSourceEventTriggerDDLCommands: YB master 8142fdc renames some variables from camel case to snake case. YB pg15 07f3c7d adapts this code in accordance with upstream PG 2f9661311b83dc481fc19f6e3bda015392010a40 changing char* to CommandTag enum. - ybc_pg_typedefs.h: - YBCPgGFlagsAccessor: YB master 1773ae2 adds ysql_use_fast_backward_scan while both YB pg15 fedbdac and YB master 56d2d2d add TEST_generate_ybrowid_sequentially. Take both. - ybc_pggate.cc: (similar to ybc_pg_typedefs.h) - yb_pg_metrics.c: - statementType: YB master 968e831 adds CatCacheMisses_# entries. Due to six additional cache entries between PG 11.2 and 15.2 (see syscache.c explanation), add six more entries here: - CatCacheMisses_79, - CatCacheMisses_80, - CatCacheMisses_81, - CatCacheMisses_82, - CatCacheMisses_83, - CatCacheMisses_84, - pg_proc.dat: - yb_reset_analyze_statistics: YB master 9f2f70e introduces this code. Upstream PG 814c9019aa7b0ce3a4f19415023a256fdd717b20 changes prolang => '14' to prolang => 'sql' Do the same here to avoid the following initdb error: unresolved OID reference "14" in pg_proc.dat field prolang line 12041 - yb_system_views.sql: - yb_reset_analyze_statistics: YB master 9f2f70e introduces this function. - Upstream PG 6cbfb784c3c91146148a76d50cda6f69ae6a79fb splits pg_statistic_ext to that and pg_statistic_ext_data, so adapt the function accordingly. - Upstream PG 3d351d916b20534f973eda760cde17d96545d4c4 makes reltuples = -1 by default instead of 0. Adapt the function accordingly. - YB master appears to miss the case for handling partitioned tables. This likely didn't surface in tests on master because reltuples = 0 by default in master. - V52__22028__yb_reset_analyze_statistics.sql: - yb_reset_analyze_statistics: (same) - yb_reset_analyze.sql, yb_reset_analyze.out: - This test is introduced by YB master 9f2f70e. Adapt to PG 15: - Add GRANT ALL ON SCHEMA public for non-super users. - Adapt to pg_statistic_ext_data change by upstream PG 6cbfb784c3c91146148a76d50cda6f69ae6a79fb similarly to the above resolution. - Adapt to reltuples = -1 change by upstream PG 3d351d916b20534f973eda760cde17d96545d4c4 similar to the above resolution. - Update an expected output for pg_database ncolstats from 13 to 16 since pg_database now has more columns. - explain.c: - yb_fix_indexpr_mutator: YB master de8c392 adds this function. Change varnoold to varnosyn in accordance with upstream PG 9ce77d75c5ab094637cc4a446296dc3be6e3c221. - ybccmds.c: - YBCDropColumn: YB master 19ab966 adds a raw_parser call. Upstream PG 844fe9f159a948377907a63d0ef3fb16dc51ce50 adds a new argument for raw_parser. Pass the appropriate argument RAW_PARSE_DEFAULT. - yb_alter_table_rewrite.out: - ALTER TABLE test_drop_pk_column_part DROP COLUMN id: YB master 19ab966 adds this test. Change the error message according to upstream PG a0555ddab9b672a04681ce0d9f6c94104c01b15f. - nodeYbBitmapTablescan.c: - YbBitmapTableNext: YB master b8fedf6 adds CreateTemplateTupleDesc and ExecInitScanTupleSlot calls. The signatures of those functions changes between PG 11.2 and 15.2. Follow a similar transformation as in nodeYbSeqscan.c YbSeqNext. - procarray.c: - GetSnapshotDataReuse: YB master edbd06b sets yb_read_time_point_handle in GetSnapshotData. Upstream PG 623a9ba79bbdd11c5eccb30b8bd5c446130e521c adds an additional GetSnapshotDataReuse alongside GetSnapshotData. Set yb_read_time_point_handle there as well. - yb_query_diagnostics.c: - BuiltinTrancheNames: YB master 47f40bb adds LWLockRegisterTranche call. Upstream PG 29c3e2dd5a6aeaf1a23d7d83d665501e2dcc6955 replaces that with a BuiltinTrancheNames array. Translate to lwlock.c BuiltinTrancheNames. - DumpToFile: YB master 47f40bb adds FileWrite function call. Upstream PG c24dcd0cfd949bdf245814c4c2b3df828ee7db36 adds extra offset parameter. Pass 0. - lwlock.c: - BuiltinTrancheNames: (see yb_query_diagnostics.c BuiltinTrancheNames) - hba.c: - get_hba_options: YB master 18bb9b8 adds if (hba->yb_jwt_jwks_url) block to this function. The function moves to hbafuncs.c by upstream PG d4781d8873f8c3fc8b0957cc03ce91627576cf36. - hbafuncs.c: (see hba.c) - wait_event.h: - WaitEventActivity: YB master 47f40bb adds YB_WAIT_EVENT_QUERY_DIAGNOSTICS_MAIN to WaitEventActivity in pgstat.h. Upstream PG a333476b925134f6185037eaff3424c07a9f466f moves this code to wait_event.h. - wait_event.c: - pgstat_get_wait_activity: (similar to wait_event.h) - yb_index_schedule: - YB master 56d2d2d removes yb_index_serial_schedule and yb_index_serial2_schedule and makes yb_index_schedule and yb_pg_index_schedule. YB pg15 9690419 adds yb_index_including test to yb_index_serial_schedule. Move that test to yb_index_schedule. - ipci.c - CreateSharedMemoryAndSemaphores: upstream PG 0bd305ee1d427ef29f5fa4fa20567e3b3f5ff792 moves code into CalculateShmemSize. YB 47f40bb adds code there. Move that code to the new location. Note also upstream PG 4f2400cb3f10aa79f99fba680c198237da28dd38 modifying an adjacent line. - backend/utils/misc/MakeFile: - OBJS: YB master 47f40bb adds yb_query_diagnostics.o to the list. YB pg15 changes the style. - snapmgr.c: - RestoreSnapshot: adjacent conflict between YB master edbd06b defining yb_read_time_point_handle and upstream PG 623a9ba79bbdd11c5eccb30b8bd5c446130e521c defining snapXactCompletionCount. - dbcommands.c: - createdb: upstream PG aa01051418f10afbdfa781b8dc109615ca785ff9 creates if (OidIsValid(dboid)) and else case, moving existing code into the else case. YB master dee7691 adds "CREATE DATABASE using templates other than template0 and template1" code in the same place. Order the YB master added code before the if-else. Secondly, the same YB master commit modifies a YBCCreateDatabase function call. There are two such calls here, one for each branch of the if-else (for history, follow YB pg15 07f3c7d commit message). Modify each call. - snapshot.h: - struct YbReadTimePointHandle: adjacent conflict between upstream PG 63746189b23815415cacc715fdc4f6b991f1a5e7 removing SnapshotSatisfiesFunc code and YB master edbd06b adding YbReadTimePointHandle code. - struct SnapshotData: adjacent conflict between upstream PG 623a9ba79bbdd11c5eccb30b8bd5c446130e521c adding snapXactCompletionCount field and YB master edbd06b adding yb_read_time_point_handle field. - pg_dml.cc: - BindColumn: adjacent lines conflict between YB pg15 0df60d3 changing code in function and YB master 47da28c changing function signature style. - has_aggregate_targets: adjacent lines conflict between YB pg15 5e6edbf removing has_system_targets and YB master 47da28c removing both has_aggregate_targets and has_system_targets. - pg_dml.h: - has_secondary_index_with_doc_op: adjacent lines conflict betewen YB pg15 5e6edbf removing has_system_targets and YB master 47da28c adding [[nodiscard]] to multiple functions (including has_system_targets) and filling in definitions for them. - pg_sample.cc: - PgSample::InitRandomState: YB pg15 055f04e splits rand_state to rand_state_s0 and rand_state_s1, and YB master 47da28c makes other changes to the function. - PrepareSamplingState: YB pg15 055f04e splits rand_state to rand_state_s0 and rand_state_s1, and YB master 47da28c moves this code into class PgSamplePicker. - pg_sample.h: - PrepareSamplingState: YB pg15 055f04e splits rand_state to rand_state_s0 and rand_state_s1, and YB master 47da28c deletes the code since pg_sample.cc now has the function defined in the class. - pg15_tests/passing_tests.tsv: - TestPgRegressWaitQueues: YB master 3494424 causes this test to crash, so remove it from the list to investigate later. Test Plan: On Almalinux 8: #!/usr/bin/env bash set -eu ./yb_build.sh fastdebug --gcc11 TZ=UTC pg15_tests/run_tests.sh Skip running test plan tests out of laziness. Jenkins: rebase: pg15 Reviewers: fizaa, tfoucher Reviewed By: fizaa Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D36729
ddhodge
added a commit
that referenced
this issue
Jul 30, 2024
…23065) * initial commit for logical replication docs * title changes * changes to view table * fixed line break * fixed line break * added content for delete and update * added more content * replaced hyperlink todos with reminders * added snapshot metrics * added more content * added more config properties to docs * added more config properties to docs * added more config properties to docs * replaced postgresql instances with yugabytedb * added properties * added complete properties * changed postgresql to yugabytedb * added example for all record types * fixed highlighting of table header * added type representations * added type representations * full content in now; * full content in now; * changed postgres references appropriately * added a missing keyword * changed name * self review comments * self review comments * added section for logical replication * added section for logical replication * modified content for monitor page * added content for monitoring * rebased to master; * CDC logical replication overview (#3) Co-authored-by: Vaibhav Kushwaha <[email protected]> * advanced-topic (#5) Co-authored-by: Vaibhav Kushwaha <[email protected]> * removed references to incremental and ad-hoc snapshots * replaced index page with an empty one * addressed review comments * added getting started section * added section for get started * self review comments * self review comments * group review comments * added hstore and domain type docs * Advance configurations for CDC using logical replication (#2) * Fix overview section (#7) * Monitor section (#4) Co-authored-by: Vaibhav Kushwaha <[email protected]> * Initial Snapshot content (#6) * Add getting started (#1) * Fix for broken note (#9) * Fix the issue yaml parsing Summary: Fixes the issue yaml parsing. We changed the formatting for yaml list. This diff fixes the usage for the same. Test Plan: Prepared alma9 node using ynp. Verified universe creation. Reviewers: vbansal, asharma Reviewed By: asharma Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36711 * [PLAT-14534]Add regex match for GCP Instance template Summary: Added regex match for gcp instance template. Regex taken from gcp documentation [[https://cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates | here]]. Test Plan: Tested manually that validation fails with invalid characters. Reviewers: #yba-api-review!, svarshney Reviewed By: svarshney Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36543 * update diagram (#23245) * [/PLAT-14708] Fix JSON field name in TaskInfo query Summary: This was missed when task params were moved out from details field. Test Plan: Trivial - existing tests should succeed. Reviewers: vbansal, cwang Reviewed By: vbansal Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36705 * [#23173] DocDB: Allow large bytes to be passed to RateLimiter Summary: RateLimiter has a debug assert that you cannot `Request` more than `GetSingleBurstBytes`. In release mode we do not perform this check and any call gets stuck forever. This change allows large bytes to be requested on RateLimiter. It does so by breaking requests larger than `GetSingleBurstBytes` into multiple smaller requests. This change is a temporary fix to allow xCluster to operate without any issues. RocksDB RateLimiter has multiple enhancements over the years that would help avoid this and more starvation issues. Ex: facebook/rocksdb@cb2476a. We should consider pulling in those changes. Fixes #23173 Jira: DB-12112 Test Plan: RateLimiterTest.LargeRequests Reviewers: slingam Reviewed By: slingam Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D36703 * [#23179] CDCSDK: Support data types with dynamically alloted oids in CDC Summary: This diff adds support for data types with dynamically alloted oids in CDC (for ex: hstore, enum array, etc). Such types contain invalid pg_type_oid for the corresponding columns in docdb schema. In the current implemtation, in `ybc_pggate`, while decoding the cdc records we look at the `type_map_` to obtain YBCPgTypeEntity, which is then used for decoding. However the `type_map_` does not contain any entries for the data types with dynamically alloted oids. As a result, this causes segmentation fault. To prevent such crashes, CDC prevents addition of tables with such columns to the stream. This diff removes the filtering logic and adds the tables to the stream even if it has such a type column. A function pointer will now be passed to `YBCPgGetCDCConsistentChanges`, which takes attribute number and the table_oid and returns the appropriate type entity by querying the `pg_type` catalog table. While decoding if a column is encountered with invalid pg_type_oid then, the passed function is invoked and type entity is obtained for decoding. **Upgrade/Rollback safety:** This diff adds a field `optional int32 attr_num` to DatumMessagePB. These changes are protected by the autoflag `ysql_yb_enable_replication_slot_consumption` which already exists but has not yet been released. Jira: DB-12118 Test Plan: Jenkins: urgent All the existing cdc tests ./yb_build.sh --java-test 'org.yb.pgsql.TestPgReplicationSlot#replicationConnectionConsumptionAllDataTypesWithYbOutput' Reviewers: skumar, stiwary, asrinivasan, dmitry Reviewed By: stiwary, dmitry Subscribers: steve.varnau, skarri, yql, ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36689 * [PLAT-14710] Do not return apiToken in response to getSessionInfo Summary: **Context** The GET /session_info YBA API returns: { "authToken": "…", "apiToken": "….", "apiTokenVersion": "….", "customerUUID": "uuid1", "userUUID": "useruuid1" } The apiToken and apiTokenVersion is supposed to be the last generated token that is valid. We had the following sequence of changes to this API. https://yugabyte.atlassian.net/browse/PLAT-8028 - Do not store YBA token in YBA. After the above fix, YBA does not store the apiToken anymore. So it cannot return it as part of the /session_info. The change for this ticket returned the hashed apiToken instead. https://yugabyte.atlassian.net/browse/PLAT-14672 - getSessionInfo should generate and return api key in response Since the hashed apiToken value is not useful to any client, and it broke YBM create cluster (https://yugabyte.atlassian.net/browse/CLOUDGA-22117), the first change for this ticket returned a new apiToken instead. Note that GET /session_info is meant to get customer and user information for the currently authenticated session. This is useful for automation starting off an authenticated session from an existing/cached API token. It is not necessary for the /session_info API to return the authToken and apiToken. The client already has one of authToken or apiToken with which it invoked /session_info API. In fact generating a new apiToken whenever /session_info is called will invalidate the previous apiToken which would not be expected by the client. There is a different API /api_token to regenerate the apiToken explicitly. **Fix in this change** So the right behaviour is for /session_info to stop sending the apiToken in the response. In fact, the current behaviour of generating a new apiToken everytime will break a client (for example node-agent usage of /session_info here (https://github.com/yugabyte/yugabyte-db/blob/4ca56cfe27d1cae64e0e61a1bde22406e003ec04/managed/node-agent/app/server/handler.go#L19). **Client impact of not returning apiToken in response of /session_info** This should not impact any normal client that was using /session_info only to get the user uuid and customer uuid. However, there might be a few clients (like YBM for example) that invoked /session_info to get the last generated apiToken from YBA. Unfortunately, this was a mis-use of this API. YBA generates the apiToken in response to a few entry point APIs like /register, /api_login and /api_token. The apiToken is long lived. YBA could choose to expire these apiTokens after a fixed amount of (long) time, but for now there is no expiration. The clients are expected to store the apiToken at their end and use the token to reestablish a session with YBA whenever needed. After establishinig a new session, clients would call GET /session_info to get the user uuid and customer uuid. This is getting fixed in YBM with https://yugabyte.atlassian.net/browse/CLOUDGA-22117. So this PLAT change should be taken up by YBM only after CLOUDGA-22117 is fixed. Test Plan: * Manually verified that session_info does not return authToken * Shubham verified that node-agent works with this fix. Thanks Shubham! Reviewers: svarshney, dkumar, tbedi, #yba-api-review! Reviewed By: svarshney Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D36712 * [docs] updates to CVE table status column (#23225) * updates to status column * review comment * format --------- Co-authored-by: Dwight Hodge <[email protected]> * [docs] Fix load balance keyword in drivers page (#23253) [docs] Fix `load_balance` -> `load-balance` in jdbc driver [docs] Fix `load_balance` -> `loadBalance` in nodejs driver * fixed compilation * fix link, format * format, links * links, format * format * format * minor edit * best practice (#8) * moved sections * moved pages * added key concepts page * added link to getting started * Dynamic table doc changes (#11) * icons * added box for lead link * revert ybclient change * revert accidental change * revert accidental change * revert accidental change * fix link block for getting started page * format * minor edit * links, format * format * links * format * remove reminder references * Modified output plugin docs (#12) * Naming edits * format * review comments * diagram * review comment * fix links * format * format * link * review comments * copy to stable * link --------- Co-authored-by: siddharth2411 <[email protected]> Co-authored-by: Shubham <[email protected]> Co-authored-by: asharma-yb <[email protected]> Co-authored-by: Dwight Hodge <[email protected]> Co-authored-by: Naorem Khogendro Singh <[email protected]> Co-authored-by: Hari Krishna Sunder <[email protected]> Co-authored-by: Sumukh-Phalgaonkar <[email protected]> Co-authored-by: Subramanian Neelakantan <[email protected]> Co-authored-by: Aishwarya Chakravarthy <[email protected]> Co-authored-by: Dwight Hodge <[email protected]> Co-authored-by: ddorian <[email protected]> Co-authored-by: Sumukh-Phalgaonkar <[email protected]>
|
Observing this issue on |
|
Fix is in 2024.1.2 |
|
We are seeing this again in |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-12112
Description
Observed this on:
2.23.0.0-b574In colocated database after setting up replication and performing following copy commands data is not replicated.
The GetChanges RPCs were timing out
Logs in Jira ticket
Steps
Issue Type
kind/bug
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: