Always use an auto-generated doc values as a back-up for Avro doc-related metadata retrieval. #377
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #377 +/- ##
============================================
+ Coverage 91.47% 91.90% +0.42%
- Complexity 243 258 +15
============================================
Files 26 27 +1
Lines 927 963 +36
Branches 67 71 +4
============================================
+ Hits 848 885 +37
+ Misses 52 50 -2
- Partials 27 28 +1 |
rulle-io
changed the title
rulle-io
changed the title
rulle-io
changed the title
rulle-io
changed the title
rulle-io
force-pushed
the
supplied_schema_validation
branch
from
d232488 to
bcd97b7
Compare
|
Hi. Sorry it took me a while to get here, as I am putting little time on this project... Is this PR still relevant? It has some conflicts with the just merged #380 . If so, can you elaborate a bit further on the need for these changes? Specifically: what do we mean by "more fault-tolerant"? And what problem does "less dependant on a user-supplied schema" solves? |
added 2 commits
March 12, 2022 11:18
…en supplied schema (expected data format) and an actual data format, returned by a SQL query. Reorganize some code to make locations more logical. Always use generated Avro schema. Optional user provided schema used for `doc` fields retrieval.
rulle-io
force-pushed
the
supplied_schema_validation
branch
from
bcd97b7 to
12989f1
Compare
|
Updated the description. |
rulle-io
changed the title
rulle-io
changed the title
added 4 commits
January 19, 2023 00:45
Add more tests and updte docs.
…n by a SQL result).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is meant to be a solution for issue #579 .
Also make a schema generation process less dependant on a user-provided schema and more fault-tolerant.
Current implementation
dbeam always generates an doc-related properties for a Avro schema based on input parameters and
ResultSetvalue.Optionally a user can provide a custom "handwritten" schema.
A user-provided schema is only used for Avro
docvalues.Thus fields' names, types and type length are taken from an auto-generated schema.
Drawback(s)
One of drawbacks of this behaviour is that when a new field appears in a DB table and as consequence in a source SQL ResultSet (e.g.
SELECT *is used), and a user-provided scheam doesn't contain this field, the process will throw an error.Solution
dbeam's auto-generated schema is always used as a back-up, if a new a user-provided schema doesn't contain the field in question.
Additional use-case
An unplanned positive side-effect can be that one can use a a user-provided schema as a dictionary of descriptions (
docs) for various fields, so one schema file can be used for muliple tables. We are going to use this side-effect.Checklist for PR author(s)
mvn com.coveo:fmt-maven-plugin:format org.codehaus.mojo:license-maven-plugin:update-file-header)