Add dump/restore support for Hypercore TAM #7356
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erimatnor
force-pushed
the
hyperstore-pgdump
branch
from
669a90e
to
d7fedcb
Compare
erimatnor
changed the title
erimatnor
force-pushed
the
hyperstore-pgdump
branch
3 times, most recently
from
098490f
to
ab64ed9
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7356 +/- ##
==========================================
+ Coverage 80.06% 82.10% +2.03%
==========================================
Files 190 230 +40
Lines 37181 43128 +5947
Branches 9450 10835 +1385
==========================================
+ Hits 29770 35409 +5639
- Misses 2997 3401 +404
+ Partials 4414 4318 -96 ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
erimatnor
force-pushed
the
hyperstore-pgdump
branch
from
ab64ed9
to
304334f
Compare
Replace the scankey flag used to skip compressed data when starting a Hypercore scan with a function that sets this option on the scan descriptor. Internally, use the scan flags instead of scankey flags to convey this setting. Overloading scankey flags was not ideal since this is supposed to be per-column settings and not overall scan settings. Note that it is possible to set the scan flags when calling the TAM's beginscan callback, but the table_beginscan() wrapper does not expose flags and instead there's a separate function for each flag settings. Hypercore could define its own beginscan function to do the same, but this is left for the future.
erimatnor
force-pushed
the
hyperstore-pgdump
branch
from
304334f
to
80fcb9f
Compare
erimatnor
force-pushed
the
hyperstore-pgdump
branch
from
80fcb9f
to
e4ed218
Compare
Add support for dumping and restoring hypertables that have chunks that use the Hypercore TAM. Dumping a Hypercore table requires special consideration because its data is internally stored in two separate relations: one for compressed data and one for non-compressed data. The TAM returns data from both relations, but they may be dumped as separate tables. This risks dumping the compressed data twice: once via the TAM and once via the compressed table in compressed format. The `pg_dump` tool uses `COPY TO` to create dumps of each table, and, to avoid data duplication when used on Hypercore tables, this change introduces a GUC that allows selecting one of these two behaviors: 1. A `COPY TO` on a Hypercore table returns all data via the TAM, including data stored in the compressed relation. A `COPY TO` on the internal compressed relation returns no data. 2. A `COPY TO` on a Hypercore returns only non-compressed data, while a `COPY TO` on the compressed relation returns compressed data. A `SELECT` still returns all the data as normal. The second approach is the default because it is consistent with compression when Hypercore TAM is not used. It will produce a `pg_dump` archive that includes data in compressed form (if data was compressed when dumped). Conversely, option (1) will produce an archive that looks identical to a dump from an non-compressed table. There are pros and cons of each dump format. A non-compressed archive is a platform-agnostic logical dump that can be restored to any platform and architecture, while a compressed archive includes data that is compressed in a platform-dependent way and needs to be restored to a compatible system. A test is added that tests both these settings and corresponding dumping and restoring.
PG17 introduced a new Bump allocator which for per-tuple memory contexts. Bump doesn't support pfree(), which caused an error when detoasting compressed data on the per-tuple memory context since the detoasting needs to scan some catalog tables. Make sure Hypercore TAM always detoasts on a temporary memory to fix this issue and add a test for it.
erimatnor
force-pushed
the
hyperstore-pgdump
branch
from
e4ed218
to
7e2d5fd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for dumping and restoring hypertables that have chunks that use the Hypercore TAM.
Dumping a Hypercore table requires special consideration because its data is internally stored in two separate relations: one for compressed data and one for non-compressed data. The TAM returns data from both relations, but they may be dumped as separate tables. This risks dumping the compressed data twice: once via the TAM and once via the compressed table in compressed format.
The
pg_dumptool usesCOPY TOto create dumps of each table, and, to avoid data duplication when used on Hypercore tables, this change introduces a GUC that allows selecting one of these two behaviors:A
COPY TOon a Hypercore table returns all data via the TAM, including data stored in the compressed relation. ACOPY TOon the internal compressed relation returns no data.A
COPY TOon a Hypercore returns only non-compressed data, while aCOPY TOon the compressed relation returns compressed data. ASELECTstill returns all the data as normal.The second approach is the default because it is consistent with compression when Hypercore TAM is not used. It will produce a
pg_dumparchive that includes data in compressed form (if data was compressed when dumped). Conversely, option (1) will produce an archive that looks identical to a dump from an non-compressed table.There are pros and cons of each dump format. A non-compressed archive is a platform-agnostic logical dump that can be restored to any platform and architecture, while a compressed archive includes data that is compressed in a platform-dependent way and needs to be restored to a compatible system.
A test is added that tests both these settings and corresponding dumping and restoring.
Disable-check: force-changelog-file