Exposure Parsing #38
Exposure Parsing #38
Conversation
…ersion of dbt-metabase having improved yaml parsing. large formatting update to conform to black.
…special or semantic until ready to deprecate
…e properly used if found in metabase api response.
…synchronization, and formatting
… without depends on/test_metadata dont throw
…xpected input format to be defined in readme. otherwise automatic resolution of target field using relation test will prepend target run schema which should be fine in 95% of use cases. cases outside that can use manifest.json parsing or set fk_ref in yml.
…d ensured support for schema agnostic fk targets (schema resolved from manifest.json)
…sync will now only fail hard if timeout is explicit, otherwise default behaviour if --sync is true is to attempt sync for 30 seconds and proceed with aligning what can be aligned successfully. more formatting and a few comments for clarity of intent. also added option to pass custom cert bundle to verify.
…ranslated some args to store action based.
…usly with seemless function alongside primary artifact parser (manifest.json).
…f regex to permit catching last arg of either ref or source always being the target table. if pointing to an alias, we are collecting aliases during yml parsing to be passed to metabased client and translated to metabase table names as needed. this functionality should be unnoticed by the user but provide more resiliency as well as more user friendly outcome whilst still being very specific in our logging.
…the getter calls will just return none
…se with prev version.
…on path strings allowing relative paths for --dbt_path or --dbt_manifest_path simplified
…ntees us `schema.table` format. This allows us to guarantee the incorrectly formatted ref (which should be `schema.table`) is originating from yml. Log the warning and infer correct schema for our users using target schema which covers the 90% use case.
…ing as empty list in function call
…nit__ to nonemptystr class, cleaned some logging calls to use lazy interpolation
|
Both Everything is pulled from Metabase and checked against the subset of dbt models for table (model) name matches from dbt <-> Metabase. test_table is a model in your manifest/dbt project? |
Yes, I changed the name from the real one to the more generic |
|
@gouline |
|
All requested changes complete. Plus a little cherry in ensuring our config objects input args match our cli flag args verbatim for additional ease of use between the two methods of utilizing the package. |
|
All polished up. I think its as good as can be now (and super readable). Thanks for the review and suggestions. |
Description
Reposting because some branch renaming fudged up the PR. This is still ready for review.
Parsing exposures from Metabase automatically in relation to a dbt project. This function is huge for understanding exposures within your data model. Close the loop between model composition and tangible BI exposures of those models creating an extremely valuable company asset in the dbt docs. In the nature of this project, the aim is we can use almost the same invocation parameters users set up in CI / CD for
dbt-metabase exportas they can withdbt-metabase exposures. This lets you run the invocation in production to essentially keep you docs in full sync with BI.Type of change
How Has This Been Tested?
Open to ideas but ultimately I think making a mock api from metabase running on top of a database that has been seeded and ran through with the dbt jaffle shop project would be most robust. Just a lot of work.
Test Configuration:
Checklist:
References #22