[1229] Do not write RemoveFile.numRecords to the delta log checkpoint
#1230
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scottsand-db
changed the title
RemoveFile.numRecords to the delta log checkpoint
scottsand-db
changed the title
RemoveFile.numRecords to the delta log checkpointRemoveFile.numRecords to the delta log checkpoint
… checkpoint by default
scottsand-db
changed the title
RemoveFile.numRecords to the delta log checkpointRemoveFile.numRecords to the delta log checkpoint
zsxwing
approved these changes
Jul 1, 2022
|
Shouldnt this be in the 2.0 as well? |
vkorukanti
pushed a commit
to vkorukanti/delta
that referenced
this pull request
Jul 14, 2022
(Cherry-pick of delta-io#1230) This PR avoids persisting NumRecord of RemoveFile action to checkpoint by removing this attribute from the constructor of RemoveFile object. Resolves delta-io#1229. This PR ensures that `RemoveFile.numRecords` field is not written out to the delta checkpoint. We write out a checkpoint, and read it back as parquet, and ensure that its schema does not contain `numRecords`. No. Closes delta-io#1230. GitOrigin-RevId: 518e46c0622cca4277729e9e6e7ebb08452619f3
mmengarelli
pushed a commit
to mmengarelli/delta.io
that referenced
this pull request
Jul 26, 2022
This PR avoids persisting NumRecord of RemoveFile action to checkpoint by removing this attribute from the constructor of RemoveFile object. Resolves delta-io#1229. This PR ensures that `RemoveFile.numRecords` field is not written out to the delta checkpoint. We write out a checkpoint, and read it back as parquet, and ensure that its schema does not contain `numRecords`. No. Closes delta-io#1230. GitOrigin-RevId: 518e46c0622cca4277729e9e6e7ebb08452619f3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Resolves #1229.
This PR ensures that
RemoveFile.numRecordsfield is not written out to the delta checkpoint. We do this by removingnumRecordsfrom theRemoveFileconstructor.How was this patch tested?
We write out a checkpoint, and read it back as parquet, and ensure that its schema does not contain
numRecords.Does this PR introduce any user-facing changes?
No.