feat: extend configuration handling #1206
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roeap
requested review from
houqp,
xianwill,
wjones127,
fvaleye,
rtyler and
mosyp
as code owners
This was referenced Mar 3, 2023
|
I am leaning towards us making the breaking change now and move on. Then come back to add the compatibility layer if someone complains. |
houqp
approved these changes
Mar 6, 2023
roeap
added a commit
that referenced
this pull request
Mar 7, 2023
# Description Another PR on the road to #632 - ~~keeping it a draft, as it is based on #1206~~ While the `commitInfo` action is defined as completely optional, spark and delta-rs write at the very least interesting, but often also quite helpful information into the commit info. To make it easier to work with and centralize some conventions, we introduce a `CommitInfo` struct, that exposes some of the fields at the top level. Additionally we harmonize a bit between spark and delta-rs conventions. # Related Issue(s) part of #632 # Documentation <!--- Share links to useful documentation ---> --------- Co-authored-by: Will Jones <[email protected]>
chitralverma
pushed a commit
to chitralverma/delta-rs
that referenced
this pull request
Mar 17, 2023
# Description This PR refactors and extends the table configuration handling. The approach is analogous to how we and object_store handle configuration via storage properties. The idea is to provide a somewhat typed layer over the untyped configuration keys. There was one surprising thing along the way. From what I can tell, we may have been omitting the `delta.` prefix on the config keys we parse. So this would definitely be breaking behaviour, since we no longer recognize keys we were parsing before. We can in principle handle aliases for keys quite easily, but I was not sure what the desired behaviour is. cc @rtyler @xianwill - This change would probably affect `kafka-delta-ingest`, so especially interested in your opinions! # Related Issue(s) part of delta-io#632 # Documentation <!--- Share links to useful documentation --->
chitralverma
pushed a commit
to chitralverma/delta-rs
that referenced
this pull request
Mar 17, 2023
# Description Another PR on the road to delta-io#632 - ~~keeping it a draft, as it is based on delta-io#1206~~ While the `commitInfo` action is defined as completely optional, spark and delta-rs write at the very least interesting, but often also quite helpful information into the commit info. To make it easier to work with and centralize some conventions, we introduce a `CommitInfo` struct, that exposes some of the fields at the top level. Additionally we harmonize a bit between spark and delta-rs conventions. # Related Issue(s) part of delta-io#632 # Documentation <!--- Share links to useful documentation ---> --------- Co-authored-by: Will Jones <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR refactors and extends the table configuration handling. The approach is analogous to how we and object_store handle configuration via storage properties. The idea is to provide a somewhat typed layer over the untyped configuration keys.
There was one surprising thing along the way. From what I can tell, we may have been omitting the
delta.prefix on the config keys we parse. So this would definitely be breaking behaviour, since we no longer recognize keys we were parsing before. We can in principle handle aliases for keys quite easily, but I was not sure what the desired behaviour is.cc @rtyler @xianwill - This change would probably affect
kafka-delta-ingest, so especially interested in your opinions!Related Issue(s)
part of #632
Documentation