feat(ingestion/powerbi): support multiple tables as upstream in native SQL parsing #8592
Conversation
github-actions
bot
added
the
ingestion
|
|
||
| parsed_result: Optional["SqlParsingResult"] = None | ||
| try: | ||
| schema_resolver = ( |
There was a problem hiding this comment.
some of this logic looks like it was copy-pasted from elsewhere
can we instead extract it into a common helper method?
There was a problem hiding this comment.
I added a function create_lineage_sql_parsed_result in sqlglot_lineage.py
| @@ -352,7 +460,11 @@ def resolve_to_data_platform_table_list(self) -> List[DataPlatformTable]: | |||
| continue | |||
|
|
|||
| table_full_name_creator: AbstractDataPlatformTableCreator = ( | |||
| supported_resolver.get_table_full_name_creator()() | |||
| supported_resolver.get_table_full_name_creator()( | |||
| ctx=ctx, | |||
There was a problem hiding this comment.
eventually, will it be possible to get rid of this AbstractDataPlatformTableCreator / move some of that logic into the sqlglot_lineage parser itself?
Overall I found it pretty tricky to follow this code and understand exactly what it was doing
There was a problem hiding this comment.
AbstractDataPlatformTableCreator is an abstraction over M-Query parsing and different classes are extending it to support special construct of M-Query, like RedShift, Snowflake, Postgres, MSSQL and some of them has native query inside M-Query code. In current code SNOWFLAKE and AMAZON_REDSHIFT supports M-Query way to create transformation as well as custom sql to create transformation inside M-Query. Both are using AbstractDataPlatformTableCreator.parse_custom_sql to generate lineage if during M-Query parsing they found custom query inside M-Query
There was a problem hiding this comment.
Please add some comments to this file about how it is structured at a high level and why it was done that way
…ithub.com:siddiquebagwan-gslab/datahub-fork into master+ing-195-sqlglot-lineage-for-tll-in-powerbi
| @@ -825,3 +825,43 @@ def sqlglot_lineage( | |||
| table_error=e, | |||
| ), | |||
| ) | |||
|
|
|||
|
|
|||
| def create_lineage_sql_parsed_result( | |||
There was a problem hiding this comment.
please also refactor tableau to use this method
| @@ -352,7 +460,11 @@ def resolve_to_data_platform_table_list(self) -> List[DataPlatformTable]: | |||
| continue | |||
|
|
|||
| table_full_name_creator: AbstractDataPlatformTableCreator = ( | |||
| supported_resolver.get_table_full_name_creator()() | |||
| supported_resolver.get_table_full_name_creator()( | |||
| ctx=ctx, | |||
There was a problem hiding this comment.
Please add some comments to this file about how it is structured at a high level and why it was done that way
hsheth2
added
the
merge-pending-ci
|
Failure not related to ingestion code so merging |
No description provided.