Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] Build foreign key list when preloading relcache #23686

Closed
1 task done
kai-franz opened this issue Aug 27, 2024 · 0 comments
Closed
1 task done

[YSQL] Build foreign key list when preloading relcache #23686

kai-franz opened this issue Aug 27, 2024 · 0 comments
Assignees
@kai-franz
Labels
area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@kai-franz
Copy link
Contributor

kai-franz commented Aug 27, 2024 •
edited by jira bot
Loading

Jira Link: DB-12593

Description

Currently, when we preload the relcache, we leave the rd_fkeylist empty, letting it load on-demand. This means that on any given connection, the first time a query references a table with foreign key constraints, we have to load them from yb-master.

We should instead preload the foreign key cache as part of our relcache preloading process (which already loads default attributes, check constraints, etc.)

Issue Type

kind/enhancement

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@kai-franz kai-franz added area/ysql Yugabyte SQL (YSQL) status/awaiting-triage Issue awaiting triage labels Aug 27, 2024
@kai-franz kai-franz self-assigned this Aug 27, 2024
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue labels Aug 27, 2024
@sushantrmishra sushantrmishra removed the status/awaiting-triage Issue awaiting triage label Sep 11, 2024
kai-franz added a commit that referenced this issue Sep 27, 2024
@kai-franz
[#23686] YSQL: Build relcache foreign key list from YB catcache
3e93354 
Summary:
Today, when preloading is enabled (either `ysql_catalog_preload_additional_tables` or `ysql_catalog_preload_additional_table_list` is set), we preload the relcache, meaning that we load the relcache entry for every relation in one shot. When we do relcache preloading, we currently don't load foreign key lists—instead, we loading them on-demand. This means on a given connection, the first time each table with a foreign key is referenced, that connection has to fetch the foreign key list from the master leader.

With this revision, we make two changes:
(1) Modify `RelationGetFKeyList` so that it reads foreign keys from the Yugabyte-only `pg_constraint` catcache `YBCONSTRAINTRELIDTYPIDNAME`.  Each time `RelationGetFKeyList` is invoked, it will do a `SearchCatCacheList` on this cache with the partial key `conrelid`.
(2) Whenever we preload the `pg_constraint` cache, we also preload the list cache for the partial key `conrelid`. This has the effect that any calls to `RelationGetFKeyList` after preloading will be able to read from the cached list and will not have to go to master. (A lookup for a table created on a different connection after preloading will still need to go to master).

Both of these changes are controlled by a new GUC `yb_enable_fkey_catcache` (+tserver gflag wrapper) that is on by default.
Jira: DB-12593

Test Plan:
```
./yb_build.sh --cxx-test pg_catalog_perf-test --gtest_filter "*ForeignKeyRelcachePreloadTest*"
```

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D37595
fizaaluthra pushed a commit that referenced this issue Sep 27, 2024
@yugabyte-ci @fizaaluthra
[BACKPORT pg15-cherrypicks] all: Bulk port from master - 111
fbbcd7f 
Summary:
 12b2c40 [#23999] DocDB: Big shared memory segments
 b1e6329 [PLAT-15279] Add gzip compression to core dumps from DB.
 06472d5 [#24050] docdb: Fix re-packing rows after alter table add column with default value
 9009d11 [#23837] YSQL: Temporarily disable some tests with Connection Manager enabled
 11acca7 [#23325][#23326] yugabyted: Support for adding new databases for xCluster replication (Phase 2)
 96703da [PLAT-15465][PLAT-15466] Minor fixes in YNP
 c5aca3b [PLAT-14924][PLAT-12829][PLAT-15446] - ui bugs and improvements
 6e82692 [#23770] [#23797] YSQL: Stabilise some test failures with Connection Manager enabled
 b50bd1b [PLAT-15279] Adjusting the core pattern to create the cores with the core_ prefix for collect cores to catch it
 f692a60 [PLAT-14045] UBI-8 images don't have hostname
 d6a19da [PLAT-15377] Adding a global uncaught exception handler to yugaware
 acbb1ba [PLAT-15225] Verify there is no running master on nodes selected for master replacement
 Excluded: 3e93354 [#23686] YSQL: Build relcache foreign key list from YB catcache

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: tfoucher, fizaa, telgersma

Differential Revision: https://phorge.dev.yugabyte.com/D38503
kai-franz added a commit that referenced this issue Sep 30, 2024
@kai-franz
[BACKPORT 2024.2][#23686] YSQL: Build relcache foreign key list from …
c3609d4 
…YB catcache

Summary:
Original commit: 3e93354 / D37595
Today, when preloading is enabled (either `ysql_catalog_preload_additional_tables` or `ysql_catalog_preload_additional_table_list` is set), we preload the relcache, meaning that we load the relcache entry for every relation in one shot. When we do relcache preloading, we currently don't load foreign key lists—instead, we loading them on-demand. This means on a given connection, the first time each table with a foreign key is referenced, that connection has to fetch the foreign key list from the master leader.

With this revision, we make two changes:
(1) Modify `RelationGetFKeyList` so that it reads foreign keys from the Yugabyte-only `pg_constraint` catcache `YBCONSTRAINTRELIDTYPIDNAME`.  Each time `RelationGetFKeyList` is invoked, it will do a `SearchCatCacheList` on this cache with the partial key `conrelid`.
(2) Whenever we preload the `pg_constraint` cache, we also preload the list cache for the partial key `conrelid`. This has the effect that any calls to `RelationGetFKeyList` after preloading will be able to read from the cached list and will not have to go to master. (A lookup for a table created on a different connection after preloading will still need to go to master).

Both of these changes are controlled by a new GUC `yb_enable_fkey_catcache` (+tserver gflag wrapper) that is on by default.
Jira: DB-12593

Test Plan:
```
./yb_build.sh --cxx-test pg_catalog_perf-test --gtest_filter "*ForeignKeyRelcachePreloadTest*"
```

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D38518
kai-franz added a commit that referenced this issue Sep 30, 2024
@kai-franz
[BACKPORT 2024.1][#23686] YSQL: Build relcache foreign key list from …
5e9270a 
…YB catcache

Summary:
Original commit: 3e93354 / D37595
Today, when preloading is enabled (either `ysql_catalog_preload_additional_tables` or `ysql_catalog_preload_additional_table_list` is set), we preload the relcache, meaning that we load the relcache entry for every relation in one shot. When we do relcache preloading, we currently don't load foreign key lists—instead, we loading them on-demand. This means on a given connection, the first time each table with a foreign key is referenced, that connection has to fetch the foreign key list from the master leader.

With this revision, we make two changes:
(1) Modify `RelationGetFKeyList` so that it reads foreign keys from the Yugabyte-only `pg_constraint` catcache `YBCONSTRAINTRELIDTYPIDNAME`.  Each time `RelationGetFKeyList` is invoked, it will do a `SearchCatCacheList` on this cache with the partial key `conrelid`.
(2) Whenever we preload the `pg_constraint` cache, we also preload the list cache for the partial key `conrelid`. This has the effect that any calls to `RelationGetFKeyList` after preloading will be able to read from the cached list and will not have to go to master. (A lookup for a table created on a different connection after preloading will still need to go to master).

Both of these changes are controlled by a new GUC `yb_enable_fkey_catcache` (+tserver gflag wrapper) that is on by default.
Jira: DB-12593

Test Plan:
```
./yb_build.sh --cxx-test pg_catalog_perf-test --gtest_filter "*ForeignKeyRelcachePreloadTest*"
```

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D38519
kai-franz added a commit that referenced this issue Oct 1, 2024
@kai-franz
[BACKPORT 2.20][#23686] YSQL: Build relcache foreign key list from YB…
63c32a7 
… catcache

Summary:
Original commit: 3e93354 / D37595
Today, when preloading is enabled (either `ysql_catalog_preload_additional_tables` or `ysql_catalog_preload_additional_table_list` is set), we preload the relcache, meaning that we load the relcache entry for every relation in one shot. When we do relcache preloading, we currently don't load foreign key lists—instead, we loading them on-demand. This means on a given connection, the first time each table with a foreign key is referenced, that connection has to fetch the foreign key list from the master leader.

With this revision, we make two changes:
(1) Modify `RelationGetFKeyList` so that it reads foreign keys from the Yugabyte-only `pg_constraint` catcache `YBCONSTRAINTRELIDTYPIDNAME`.  Each time `RelationGetFKeyList` is invoked, it will do a `SearchCatCacheList` on this cache with the partial key `conrelid`.
(2) Whenever we preload the `pg_constraint` cache, we also preload the list cache for the partial key `conrelid`. This has the effect that any calls to `RelationGetFKeyList` after preloading will be able to read from the cached list and will not have to go to master. (A lookup for a table created on a different connection after preloading will still need to go to master).

Both of these changes are controlled by a new GUC `yb_enable_fkey_catcache` (+tserver gflag wrapper) that is on by default.
Jira: DB-12593

Test Plan:
```
./yb_build.sh --cxx-test pg_catalog_perf-test --gtest_filter "*ForeignKeyRelcachePreloadTest*"
```

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D38520
kai-franz added a commit that referenced this issue Oct 1, 2024
@kai-franz
[BACKPORT pg15-cherrypicks][#23686] YSQL: Build relcache foreign key …
f797f49 
…list from YB catcache

Summary:
relcache.c:

  - Includes:
    - YB master 3e93354 includes  "utils/catcache.h"
    - YB pg15 55782d5 adds includes for partcache.h and relcache.h
    - Keep both sets of includes
  - RelationGetFKeyList
    - YB master 3e93354 moves some code that uses heap_open/heap_close into ELSE blocks
    - YB pg15 initial merge 55782d5 touches the same pieces of code and replaces heap_open/heap_close with table_open/table_close
    - Resolve by doing both: switching to table_open/table_close and moving into ELSE blocks
guc.h:

  - GUC declarations:
    - YB pg15 55782d5 adds PGDLLIMPORT to GUC declarations.
    - YB master 3e93354 adds new GUC yb_enable_fkey_catcache
    - Resolve similar to YB pg15 initial merge 5627af5 by keeping new GUC with PGDLLIMPORT and moving yb_enable_fkey_catcache to where the other YB declarations are.
pg_catalog_perf-test.cc:
  - ForeignKeyRelcachePreloadTest:
    - pg15 makes changes to query planning, causing additional catalog lookup RPCs. Adjust the expected # of catalog RPCs to account for this.

Original commit: 3e93354 / D37595
Today, when preloading is enabled (either `ysql_catalog_preload_additional_tables` or `ysql_catalog_preload_additional_table_list` is set), we preload the relcache, meaning that we load the relcache entry for every relation in one shot. When we do relcache preloading, we currently don't load foreign key lists—instead, we loading them on-demand. This means on a given connection, the first time each table with a foreign key is referenced, that connection has to fetch the foreign key list from the master leader.

With this revision, we make two changes:
(1) Modify `RelationGetFKeyList` so that it reads foreign keys from the Yugabyte-only `pg_constraint` catcache `YBCONSTRAINTRELIDTYPIDNAME`.  Each time `RelationGetFKeyList` is invoked, it will do a `SearchCatCacheList` on this cache with the partial key `conrelid`.
(2) Whenever we preload the `pg_constraint` cache, we also preload the list cache for the partial key `conrelid`. This has the effect that any calls to `RelationGetFKeyList` after preloading will be able to read from the cached list and will not have to go to master. (A lookup for a table created on a different connection after preloading will still need to go to master).

Both of these changes are controlled by a new GUC `yb_enable_fkey_catcache` (+tserver gflag wrapper) that is on by default.
Jira: DB-12593

Test Plan:
```
./yb_build.sh --cxx-test pg_catalog_perf-test --gtest_filter "*ForeignKeyRelcachePreloadTest*"
```

Reviewers: tfoucher, fizaa, telgersma

Reviewed By: fizaa

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D38511
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kai-franz kai-franz

Labels
area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kai-franz @sushantrmishra @yugabyte-ci