Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: better parsing of glue tables; enable directory file structure #89

Merged
merged 2 commits into from
Nov 5, 2020

Conversation

rsyi
Copy link
Owner

@rsyi rsyi commented Nov 5, 2020

Closes #86.

With this:

  • Glue table names are parsed more effectively (rather than schema_table or catalog_schema_table, we write catalog.schema.table by parsing the provided glue name and returning the properly-formatted payload).
  • Use the database/instance name as a separate directory, by using the formatting "connection_name/database/catalog.schema.table". If name isn't given in the connections.yaml file, you'll instead get: "database/catalog.schema.table". If exclusively using Glue, this reduces the complexity of the file structure nicely.

I don't believe anyone other than @rubenssoto is using this extractor (at the moment), but if you are, please open an issue and I'd be happy to provide a migration script! Breaking semantic versioning here, b/c this change is coming out so soon after the release of the feature.

@rsyi rsyi added this to the 🔗 Better integrations milestone Nov 5, 2020
@codecov-io
Copy link

Codecov Report

Merging #89 into master will increase coverage by 0.44%.
The diff coverage is 78.57%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #89      +/-   ##
==========================================
+ Coverage   67.56%   68.01%   +0.44%     
==========================================
  Files          30       31       +1     
  Lines        1520     1588      +68     
==========================================
+ Hits         1027     1080      +53     
- Misses        493      508      +15     
Impacted Files Coverage Δ
pipelines/whale/loader/whale_loader.py 58.11% <66.66%> (-0.15%) ⬇️
pipelines/whale/extractor/glue_extractor.py 78.78% <78.78%> (ø)
pipelines/whale/utils/extractor_wrappers.py 76.66% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d618ab...961699b. Read the comment docs.

@rsyi rsyi merged commit 38a9d93 into master Nov 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Directory Organization
2 participants