feat(ingest/teradata): Teradata source #8977
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ingestion
…ort Teradata properly
|
|
hsheth2
reviewed
Oct 12, 2023
| @@ -482,6 +482,7 @@ def _column_level_lineage( # noqa: C901 | |||
| # Our snowflake source lowercases column identifiers, so we are forced | |||
| # to do fuzzy (case-insensitive) resolution instead of exact resolution. | |||
| "snowflake", | |||
| "teradata", | |||
There was a problem hiding this comment.
let's add a comment around this
hsheth2
approved these changes
Oct 12, 2023
asikowitz
requested changes
Oct 12, 2023
|
|
||
| If you want to run profiling, you need to grant select permission on all the tables you want to profile. | ||
|
|
||
| 3. If linege or usage extraction is enabled, please, check if query logging is enabled and it is set to size which |
There was a problem hiding this comment.
Suggested change
| 3. If linege or usage extraction is enabled, please, check if query logging is enabled and it is set to size which | |
| 3. If lineage or usage extraction is enabled, please, check if query logging is enabled and it is set to size which |
| type: teradata | ||
| config: | ||
| host_port: "myteradatainstance.teradata.com:1025" | ||
| #platform_instance: "myteradatainstance" |
| password: mypassword | ||
| #database_pattern: | ||
| # allow: | ||
| # - "demo_user" |
There was a problem hiding this comment.
Maybe an example that has less to do with users, like my_database? Idk, just trying to make these docs as easy to follow
Comment on lines
+90
to
+95
| use_schema_resolver: bool = Field( | ||
| default=True, | ||
| description="Read SchemaMetadata aspects from DataHub to aid in SQL parsing. Turn off only for testing.", | ||
| hidden_from_docs=True, | ||
| ) | ||
|
|
There was a problem hiding this comment.
Let's remove this, don't think we need to handle the false case. Think this lets us remove the local self.urns too and always use schema_resolver.urns
| LINEAGE_QUERY: str = """SELECT ProcID, UserName as "user", StartTime AT TIME ZONE 'GMT' as "timestamp", DefaultDatabase as default_database, QueryText as query | ||
| FROM "DBC".DBQLogTbl | ||
| where ErrorCode = 0 | ||
| and QueryText like 'create table demo_user.test_lineage%' |
Comment on lines
+144
to
+168
| if self.graph: | ||
| if self.config.use_schema_resolver: | ||
| self.schema_resolver = ( | ||
| self.graph.initialize_schema_resolver_from_datahub( | ||
| platform=self.platform, | ||
| platform_instance=self.config.platform_instance, | ||
| env=self.config.env, | ||
| ) | ||
| ) | ||
| self.urns = self.schema_resolver.get_urns() | ||
| else: | ||
| self.schema_resolver = self.graph._make_schema_resolver( | ||
| platform=self.platform, | ||
| platform_instance=self.config.platform_instance, | ||
| env=self.config.env, | ||
| ) | ||
| self.urns = None | ||
| else: | ||
| self.schema_resolver = SchemaResolver( | ||
| platform=self.platform, | ||
| platform_instance=self.config.platform_instance, | ||
| graph=None, | ||
| env=self.config.env, | ||
| ) | ||
| self.urns = None |
There was a problem hiding this comment.
We shouldn't have to do this sort of logic to start. I think for now we should just always ingest schema metadata, pass that to the schema resolver, and never pass a graph into the schema resolver
maggiehays
added
the
hacktoberfest-accepted
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist