fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect #10898
Conversation
WalkthroughThe updates across various modules significantly enhance the metadata ingestion system by introducing an Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Inspector
participant SchemaRetriever
User->>SchemaRetriever: Request schema fields
SchemaRetriever->>Inspector: Inspect column types
Inspector->>SchemaRetriever: Return inspected data
SchemaRetriever-->>User: Return schema fields
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
github-actions
bot
added
ingestion
| @@ -641,7 +642,7 @@ def _get_direct_raw_col_upstreams( | |||
|
|
|||
| # Parse the column name out of the node name. | |||
| # Sqlglot calls .sql(), so we have to do the inverse. | |||
| normalized_col = sqlglot.parse_one(node.name).this.name | |||
| normalized_col = sqlglot.parse_one(node.name, dialect=dialect).this.name | |||
There was a problem hiding this comment.
can we add a test case that would have failed before but works with this change?
There was a problem hiding this comment.
I tried to reproduce the issue we talked about in context of the SAP HANA view parsing and I think it wasn't caused by parse_one method: As the identifiers are qualified using the optimize method - and more important with the identify parameter set to true - the name of the node should always be correct, including the capitalization.
For the call to the sql method in the to_node method the dialect is also not provided (https://github.com/tobymao/sqlglot/blob/5df3f5292488df6a8e21abf3b49086c823797e78/sqlglot/lineage.py#L234 and https://github.com/tobymao/sqlglot/blob/5df3f5292488df6a8e21abf3b49086c823797e78/sqlglot/lineage.py#L285), therefore the capitalization should also not be changed by the sql method. Not sure if you meant this when you mentioned the sql method in Slack, but I think you are right - everything is correct.
I have also run some performance tests (using timeit) regarding the dialect instance which would implicitly be created by parse_one and there is basically no difference with and without the dialect instance.
edit: I have removed that part from the PR. :-)
|
FYI also seeing some errors in the tests e.g. Might make sense to make a helper method like |
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
b5fbc10
to
9a7f61a
Compare
|
I don't think we should fallback to the repr function in case of an error: The data types which are returned by the reflection methods of SQLAlchemy are "produced" by the corresponding dialect and in general it should be possible to compile these data types by the dialect itself (otherwise the dialect would not produce these data types, right?). The NullType is a special case, which can explicitly not be compiled and will result in a CompileError: "NullType will result in a CompileError if the compiler is asked to render the type itself [...]" (see here: https://docs.sqlalchemy.org/en/20/core/type_api.html#sqlalchemy.types.NullType) I have added a utility function to sqlalchemy_type_converter.py which will return the __visit_name__ of the NullType (which is "null") in case NullType is suppied...I think this is better than having "NullType()" as the native data type. |
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
9a7f61a
to
17b655d
Compare
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
17b655d
to
f5eaa01
Compare
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
f5eaa01
to
4b1f9ba
Compare
| if isinstance(column_type, types.NullType): | ||
| return types.NullType.__visit_name__ | ||
|
|
||
| return column_type.compile(dialect=inspector.dialect) |
There was a problem hiding this comment.
I still think we should have a try catch around this - purely to ensure that we don't fail broadly if an underlying dialect throws an exception in .compile
There was a problem hiding this comment.
I have added a try/except, which will use the visit_name as a fallback and in case the data type is not visitable (which the data type should be, but who knows...) the repr of the data type. I would not expect to ever need this fallback, but to make sure that we are not failing because of the native data type it's probably better to have it.
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
3 times, most recently
from
401014f
to
9eec1d1
Compare
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
9eec1d1
to
1942fbb
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (21)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py (3 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py (7 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py (2 hunks)
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py (4 hunks)
- metadata-ingestion/tests/integration/hana/docker-compose.yml (1 hunks)
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json (53 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json (9 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json (9 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/test_oracle.py (1 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_mces_golden.json (18 hunks)
Files skipped from review due to trivial changes (3)
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json
Additional comments not posted (106)
metadata-ingestion/tests/integration/hana/docker-compose.yml (1)
9-9: Verify the necessity of all the removed port mappings.Reducing the port mappings can limit the functionality and accessibility of services interacting with the
testhanacontainer. Ensure that the removed ports are not required for any critical interactions.metadata-ingestion/tests/integration/oracle/test_oracle.py (1)
27-27: Verify the necessity and correctness of the lambda function.The lambda function added to the
processmethod enforces the return of the string 'NUMBER'. Ensure that this change is necessary for the tests and correctly implemented.metadata-ingestion/src/datahub/ingestion/source/sql/hive.py (2)
172-172: Verify the correct integration and utilization of theinspectorparameter.The new
inspectorparameter is introduced to theget_schema_fields_for_columnfunction. Ensure that it is correctly integrated and utilized throughout the function.
178-181: Verify the correct handling of theinspectorparameter by the superclass method.The call to the superclass method is updated to include the new
inspectorparameter. Ensure that the superclass method correctly handles this parameter and that the change does not introduce any issues.metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py (4)
8-9: LGTM! New imports are necessary.The new imports for
InspectorandVisitableare required for the added functionality.
181-181: LGTM! Method signature update is necessary.The addition of the
inspectorparameter enhances the function's capability to handle column types more robustly.
222-225: LGTM! Appropriate usage of theinspectorparameter.The
inspectorparameter is used correctly to get the native data type in the fallback description.
251-267: LGTM! Method signature update and new logic are necessary.The addition of the
inspectorparameter and the handling ofNullTypeimprove the method's robustness. The try/except block ensures graceful handling of compilation errors.metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json (9)
150-150: LGTM! Standardized representation ofINTEGER.The
nativeDataTypehas been updated from"INTEGER()"to"INTEGER", aligning it with a more conventional format.
162-162: LGTM! Simplified representation ofVARCHAR(50).The
nativeDataTypehas been updated from"VARCHAR(length=50)"to"VARCHAR(50)", enhancing clarity and consistency.
174-174: LGTM! Simplified representation ofVARCHAR(50).The
nativeDataTypehas been updated from"VARCHAR(length=50)"to"VARCHAR(50)", enhancing clarity and consistency.
186-186: LGTM! Simplified representation ofVARCHAR(50).The
nativeDataTypehas been updated from"VARCHAR(length=50)"to"VARCHAR(50)", enhancing clarity and consistency.
198-198: LGTM! Simplified representation ofVARCHAR(50).The
nativeDataTypehas been updated from"VARCHAR(length=50)"to"VARCHAR(50)", enhancing clarity and consistency.
210-210: LGTM! Standardized representation ofFLOAT.The
nativeDataTypehas been updated from"FLOAT()"to"FLOAT", aligning it with a more conventional format.
326-326: LGTM! Standardized representation ofINTEGER.The
nativeDataTypehas been updated from"INTEGER()"to"INTEGER", aligning it with a more conventional format.
338-338: LGTM! Simplified representation ofVARCHAR(50).The
nativeDataTypehas been updated from"VARCHAR(length=50)"to"VARCHAR(50)", enhancing clarity and consistency.
350-350: LGTM! Standardized representation ofINTEGER.The
nativeDataTypehas been updated from"INTEGER()"to"INTEGER", aligning it with a more conventional format.metadata-ingestion/src/datahub/ingestion/source/sql/trino.py (1)
390-399: LGTM! Method signature update and internal call update are necessary.The addition of the
inspectorparameter enhances the method's capability to handle column schemas more robustly. The internal call to the superclass method ensures that all necessary arguments are passed correctly.metadata-ingestion/src/datahub/ingestion/source/sql/athena.py (1)
502-502: Ensure that all function calls toget_schema_fields_for_columnmatch the new signature.The code changes are approved.
However, ensure that all function calls to
get_schema_fields_for_columnmatch the new signature.Verification successful
Ensure that all function calls to
get_schema_fields_for_columnmatch the new signature.The code changes are approved. The verification confirms that the new function signature, including the
inspectorparameter, is consistently used across the codebase.
metadata-ingestion/src/datahub/ingestion/source/sql/athena.pymetadata-ingestion/src/datahub/ingestion/source/sql/hive.pymetadata-ingestion/src/datahub/ingestion/source/sql/sql_common.pymetadata-ingestion/src/datahub/ingestion/source/sql/trino.pymetadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.pyScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'get_schema_fields_for_column'Length of output: 83
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --glob '*.py' -A 5 $'get_schema_fields_for_column'Length of output: 5961
metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json (9)
455-455: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromVARCHAR(length=500)toVARCHAR(500).
467-467: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromVARCHAR(length=200)toVARCHAR(200).
479-479: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromBIGINT()toBIGINT.
491-491: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromTEXT()toTEXT.
503-503: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromTEXT()toTEXT.
515-515: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromTIMESTAMP()toTIMESTAMP WITHOUT TIME ZONE.
527-527: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromVARCHAR(length=255)toVARCHAR(255).
539-539: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromVARCHAR(length=255)toVARCHAR(255).
551-551: LGTM! The change aligns with standard SQL syntax.The
nativeDataTypehas been correctly modified fromJSON(astext_type=Text())toJSON.metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py (2)
472-477: Ensure that all method calls to_process_projectionsmatch the new signature.The code changes are approved.
However, ensure that all method calls to
_process_projectionsmatch the new signature.Verification successful
All method calls to
_process_projectionsmatch the new signature.The code changes are approved.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all method calls to `_process_projections` match the new signature. # Test: Search for the method usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'_process_projections'Length of output: 75
Script:
#!/bin/bash # Description: Verify all method calls to `_process_projections` match the new signature. # Test: Search for the method usage. Expect: Only occurrences of the new signature. rg --type py -A 5 $'_process_projections'Length of output: 1279
681-681: Ensure that all method calls to_process_modelsmatch the new signature.The code changes are approved.
However, ensure that all method calls to
_process_modelsmatch the new signature.Verification successful
All method calls to
_process_modelsmatch the new signature.The code changes are correctly implemented and verified.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all method calls to `_process_models` match the new signature. # Test: Search for the method usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'_process_models'Length of output: 70
Script:
#!/bin/bash # Description: Verify all method calls to `_process_models` match the new signature. # Test: Search for the method usage. Expect: Only occurrences of the new signature. rg --type py -A 5 $'_process_models'Length of output: 1164
metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json (2)
261-261: Verify the impact of changingnativeDataTypefrom"NUMBER(asdecimal=False)"to"NUMBER".Ensure that this change aligns with the expected data type representation across the system and does not introduce any inconsistencies or errors.
Also applies to: 383-383, 508-508, 757-757, 879-879, 1004-1004
Line range hint
1-1008:
Verify the impact of removing multiple JSON objects.Ensure that the removal of these entities does not affect the data ingestion process and that they are no longer relevant or have been replaced by a different mechanism.
metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py (4)
524-524: LGTM! But verify the usage of theinspectorparameter in the function body.The addition of the
inspectorparameter to the function signature is approved.Ensure that the
inspectorparameter is utilized correctly within the function body.
524-524: LGTM! But verify the usage of theinspectorparameter in the function body.The addition of the
inspectorparameter to the function signature is approved.Ensure that the
inspectorparameter is utilized correctly within the function body.
757-759: LGTM! But verify the usage of theinspectorparameter in the function body.The addition of the
inspectorparameter to the function signature is approved.Ensure that the
inspectorparameter is utilized correctly within the function body.
882-882: LGTM! But verify the usage of theinspectorparameter in the function body.The addition of the
inspectorparameter to the function signature is approved.Ensure that the
inspectorparameter is utilized correctly within the function body.metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json (5)
261-261: Verify the impact of simplifyingnativeDataType.The change from
"NUMBER(asdecimal=False)"to"NUMBER"improves readability and consistency. Ensure that this change does not affect downstream processing or interpretation of numeric data types.
383-383: Verify the impact of simplifyingnativeDataType.The change from
"NUMBER(asdecimal=False)"to"NUMBER"improves readability and consistency. Ensure that this change does not affect downstream processing or interpretation of numeric data types.
508-508: Verify the impact of simplifyingnativeDataType.The change from
"NUMBER(asdecimal=False)"to"NUMBER"improves readability and consistency. Ensure that this change does not affect downstream processing or interpretation of numeric data types.
757-757: Verify the impact of simplifyingnativeDataType.The change from
"NUMBER(asdecimal=False)"to"NUMBER"improves readability and consistency. Ensure that this change does not affect downstream processing or interpretation of numeric data types.
879-879: Verify the impact of simplifyingnativeDataType.The change from
"NUMBER(asdecimal=False)"to"NUMBER"improves readability and consistency. Ensure that this change does not affect downstream processing or interpretation of numeric data types.metadata-ingestion/tests/integration/trino/trino_mces_golden.json (6)
259-259: Approved: Simplified data type representation.The change from
INTEGER()toINTEGERaligns with standard SQL data type definitions and simplifies the representation.
271-271: Approved: Simplified data type representation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type definitions and simplifies the representation.
283-283: Approved: Simplified data type representation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type definitions and simplifies the representation.
295-295: Approved: Simplified data type representation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type definitions and simplifies the representation.
307-307: Approved: Simplified data type representation.The change from
JSON()toJSONaligns with standard SQL data type definitions and simplifies the representation.
531-531: Approved: Simplified data type representation.The change from
DATE()toDATEaligns with standard SQL data type definitions and simplifies the representation.metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py (6)
123-125: Approved: Necessary import for native data type handling.The import statement for
get_native_data_type_for_sqlalchemy_typeis necessary for the changes made to handle native data types using SQLAlchemy's inspector.
794-794: Approved: Enhanced schema field retrieval.The addition of the
inspectorparameter to theget_schema_fieldsmethod call allows for improved schema field retrieval and type handling using SQLAlchemy's inspector.
975-975: Approved: Enhanced schema field retrieval.The addition of the
inspectorparameter to theget_schema_fieldsmethod signature enhances the method's ability to retrieve and handle schema fields using SQLAlchemy's inspector.
988-988: Approved: Enhanced schema field retrieval.The addition of the
inspectorparameter to theget_schema_fields_for_columnmethod call allows for improved schema field retrieval and type handling using SQLAlchemy's inspector.
1000-1000: Approved: Enhanced schema field retrieval.The addition of the
inspectorparameter to theget_schema_fields_for_columnmethod signature enhances the method's ability to retrieve and handle schema fields using SQLAlchemy's inspector.
1014-1019: Approved: Improved native data type handling.The updated logic for determining the
nativeDataTypeof a column by using theget_native_data_type_for_sqlalchemy_typefunction ensures that the native data type is derived correctly based on the SQLAlchemy type system, enhancing type safety and correctness.metadata-ingestion/tests/integration/hana/hana_mces_golden.json (25)
8-15: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
customPropertiesenhances readability and usability.
29-30: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
statusenhances readability and usability.
45-47: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
dataPlatformInstanceenhances readability and usability.
61-64: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
subTypesenhances readability and usability.
79-80: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
browsePathsV2enhances readability and usability.
95-102: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
customPropertiesenhances readability and usability.
117-118: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
statusenhances readability and usability.
133-135: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
dataPlatformInstanceenhances readability and usability.
149-151: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
subTypesenhances readability and usability.
167-168: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
containerenhances readability and usability.
183-189: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
browsePathsV2enhances readability and usability.
204-205: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
containerenhances readability and usability.
Line range hint
238-259:
Good transition to structured JSON objects and enhanced profiling capabilities.The change from a string-based representation to a structured JSON object for
SchemaMetadataenhances readability and usability. The detailed statistical data for each field improves the profiling capabilities of the datasets.
341-343: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
subTypesenhances readability and usability.
359-361: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
domainsenhances readability and usability.
377-387: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
browsePathsV2enhances readability and usability.
402-403: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
containerenhances readability and usability.
Line range hint
436-457:
Good transition to structured JSON objects and enhanced profiling capabilities.The change from a string-based representation to a structured JSON object for
SchemaMetadataenhances readability and usability. The detailed statistical data for each field improves the profiling capabilities of the datasets.
539-541: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
subTypesenhances readability and usability.
557-559: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
domainsenhances readability and usability.
575-585: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
browsePathsV2enhances readability and usability.
600-601: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
containerenhances readability and usability.
Line range hint
634-655:
Good transition to structured JSON objects and enhanced profiling capabilities.The change from a string-based representation to a structured JSON object for
SchemaMetadataenhances readability and usability. The detailed statistical data for each field improves the profiling capabilities of the datasets.
725-727: Good transition to structured JSON objects.The change from a string-based representation to a structured JSON object for
subTypesenhances readability and usability.
743-745: **Goodmetadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json (12)
150-150: Simplified data type representation.The change from
INTEGER()toINTEGERremoves unnecessary parentheses, simplifying the data type representation.
162-162: Simplified data type representation.The change from
DATE()toDATEremoves unnecessary parentheses, simplifying the data type representation.
174-174: More concise VARCHAR length definition.The change from
VARCHAR(length=14)toVARCHAR(14)uses a more concise form for defining the length of the VARCHAR type.
186-186: More concise VARCHAR length definition.The change from
VARCHAR(length=16)toVARCHAR(16)uses a more concise form for defining the length of the VARCHAR type.
198-198: Enhanced consistency in ENUM formatting.The change from
ENUM('M', 'F')toENUM('M','F')removes spaces, enhancing consistency in formatting.
221-221: Simplified data type representation.The change from
DATE()toDATEremoves unnecessary parentheses, simplifying the data type representation.
337-337: Simplified data type representation.The change from
INTEGER()toINTEGERremoves unnecessary parentheses, simplifying the data type representation.
349-349: Simplified data type representation.The change from
INTEGER()toINTEGERremoves unnecessary parentheses, simplifying the data type representation.
361-361: Simplified data type representation.The change from
DATE()toDATEremoves unnecessary parentheses, simplifying the data type representation.
373-373: Simplified data type representation.The change from
DATE()toDATEremoves unnecessary parentheses, simplifying the data type representation.
2459-2459: Enhanced consistency in SET formatting.The change from
SET('a', 'b', 'c', 'd')toSET('a','b','c','d')removes spaces, enhancing consistency in formatting.
2575-2575: More concise VARCHAR length definition.The change from
VARCHAR(length=50)toVARCHAR(50)uses a more concise form for defining the length of the VARCHAR type.metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json (6)
234-234: Verify the updatedtransient_lastddltimevalue.Ensure that the new timestamp
1722106707is correct and consistent with the expected format and context.
270-270: Verify the updatednativeDataTypevalue.Ensure that the new data type
INTEGERis correct and consistent with the expected format and context.
474-474: Verify the updatedtransient_lastddltimevalue.Ensure that the new timestamp
1722106711is correct and consistent with the expected format and context.
508-508: Verify the updatednativeDataTypevalue.Ensure that the new data type
VARCHARis correct and consistent with the expected format and context.
756-756: Verify the updatedtransient_lastddltimevalue.Ensure that the new timestamp
1722106709is correct and consistent with the expected format and context.
790-790: Verify the updatednativeDataTypevalue.Ensure that the new data type
VARCHARis correct and consistent with the expected format and context.metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json (10)
247-247: Verify the consistency oftransient_lastddltimevalues.Ensure that the updated timestamp value for
transient_lastddltimeis consistent with other entries and follows the correct format.
283-283: LGTM! Ensure consistency across all entries.The change from
INTEGER()toINTEGERstandardizes the data type representation. Verify that similar changes are applied consistently across other entries.
508-508: Verify the consistency oftransient_lastddltimevalues.Ensure that the updated timestamp value for
transient_lastddltimeis consistent with other entries and follows the correct format.
542-542: LGTM! Ensure consistency across all entries.The change from
VARCHAR()toVARCHARstandardizes the data type representation. Verify that similar changes are applied consistently across other entries.
767-767: Verify the consistency oftransient_lastddltimevalues.Ensure that the updated timestamp value for
transient_lastddltimeis consistent with other entries and follows the correct format.
801-801: LGTM! Ensure consistency across all entries.The change from
VARCHAR()toVARCHARstandardizes the data type representation. Verify that similar changes are applied consistently across other entries.
994-994: Verify the consistency oftransient_lastddltimevalues.Ensure that the updated timestamp value for
transient_lastddltimeis consistent with other entries and follows the correct format.
1028-1028: LGTM! Ensure consistency across all entries.The change from
VARCHAR()toVARCHARstandardizes the data type representation. Verify that similar changes are applied consistently across other entries.
1265-1265: Verify the consistency oftransient_lastddltimevalues.Ensure that the updated timestamp value for
transient_lastddltimeis consistent with other entries and follows the correct format.
1299-1299: LGTM! Ensure consistency across all entries.The change from
VARCHAR()toVARCHARstandardizes the data type representation. Verify that similar changes are applied consistently across other entries.
| @@ -2,15 +2,11 @@ | |||
| version: '3.4' | |||
| services: | |||
| testhana: | |||
| image: "store/saplabs/hanaexpress:2.00.054.00.20210603.1" | |||
| image: "saplabs/hanaexpress:latest" | |||
There was a problem hiding this comment.
Consider pinning the Docker image to a specific version.
Using the latest version of the Docker image can introduce variability and potential instability due to untested changes. Pinning to a specific version ensures stability and reproducibility.
- image: "saplabs/hanaexpress:latest"
+ image: "saplabs/hanaexpress:2.00.054.00.20210603.1"Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| image: "saplabs/hanaexpress:latest" | |
| image: "saplabs/hanaexpress:2.00.054.00.20210603.1" |
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
1942fbb
to
dd3f27b
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (21)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py (3 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py (7 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py (2 hunks)
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py (4 hunks)
- metadata-ingestion/tests/integration/hana/docker-compose.yml (1 hunks)
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json (53 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json (9 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json (9 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/test_oracle.py (1 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_mces_golden.json (18 hunks)
Files skipped from review due to trivial changes (6)
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json
Files skipped from review as they are similar to previous changes (14)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py
- metadata-ingestion/tests/integration/hana/docker-compose.yml
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json
- metadata-ingestion/tests/integration/oracle/test_oracle.py
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json
Additional comments not posted (11)
metadata-ingestion/tests/integration/trino/trino_mces_golden.json (11)
259-259: LGTM! SimplifiednativeDataTyperepresentation.The change from
INTEGER()toINTEGERaligns with standard SQL data type representations and improves clarity.
271-271: LGTM! SimplifiednativeDataTyperepresentation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type representations and improves clarity.
283-283: LGTM! SimplifiednativeDataTyperepresentation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type representations and improves clarity.
295-295: LGTM! SimplifiednativeDataTyperepresentation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type representations and improves clarity.
307-307: LGTM! SimplifiednativeDataTyperepresentation.The change from
JSON()toJSONaligns with standard SQL data type representations and improves clarity.
507-507: LGTM! SimplifiednativeDataTyperepresentation.The change from
INTEGER()toINTEGERaligns with standard SQL data type representations and improves clarity.
519-519: LGTM! SimplifiednativeDataTyperepresentation.The change from
INTEGER()toINTEGERaligns with standard SQL data type representations and improves clarity.
531-531: LGTM! SimplifiednativeDataTyperepresentation.The change from
DATE()toDATEaligns with standard SQL data type representations and improves clarity.
543-543: LGTM! SimplifiednativeDataTyperepresentation.The change from
DATE()toDATEaligns with standard SQL data type representations and improves clarity.
726-726: LGTM! SimplifiednativeDataTyperepresentation.The change from
INTEGER()toINTEGERaligns with standard SQL data type representations and improves clarity.
738-738: LGTM! SimplifiednativeDataTyperepresentation.The change from
VARCHAR(length=50)toVARCHAR(50)aligns with standard SQL data type representations and improves clarity.
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
dd3f27b
to
1a07c6d
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (22)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py (3 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py (7 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py (2 hunks)
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py (4 hunks)
- metadata-ingestion/tests/integration/hana/docker-compose.yml (1 hunks)
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json (53 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json (9 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json (9 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/test_oracle.py (1 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_mces_golden.json (18 hunks)
- metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.py (7 hunks)
Files skipped from review due to trivial changes (3)
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json
Files skipped from review as they are similar to previous changes (17)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py
- metadata-ingestion/tests/integration/hana/docker-compose.yml
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json
- metadata-ingestion/tests/integration/oracle/test_oracle.py
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json
- metadata-ingestion/tests/integration/trino/trino_mces_golden.json
Additional comments not posted (10)
metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.py (5)
48-50: LGTM! But verify the function usage in the codebase.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correct.However, ensure that all function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.Verification successful
All function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correctly reflected in the function calls throughout the codebase.
metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.pymetadata-ingestion/src/datahub/ingestion/source/sql/athena.pymetadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.pyScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 94
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type py -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 6596
86-87: LGTM! But verify the function usage in the codebase.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correct.However, ensure that all function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.Verification successful
Verified: All calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correct and consistent throughout the codebase.
metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.pymetadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.pymetadata-ingestion/src/datahub/ingestion/source/sql/athena.pyScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 94
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type py -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 6596
66-68: LGTM! But verify the function usage in the codebase.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correct.However, ensure that all function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.Verification successful
Function usage verified!
All calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature with theinspectorparameter.
metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.pymetadata-ingestion/src/datahub/ingestion/source/sql/athena.pyScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 94
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type py -A 5 'get_schema_fields_for_sqlalchemy_column'Length of output: 6596
111-113: LGTM! But verify the function usage in the codebase.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correct.However, ensure that all function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.Verification successful
Verified: All function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.The addition of the
inspectorparameter is correctly reflected across the codebase.
metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.pymetadata-ingestion/src/datahub/ingestion/source/sql/athena.pyScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 94
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg -t py -A 5 'get_schema_fields_for_sqlalchemy_column'Length of output: 6592
22-24: LGTM! But verify the function usage in the codebase.The addition of the
inspectorparameter and its instantiation as aMagicMockwith aDefaultDialectis correct.However, ensure that all function calls to
get_schema_fields_for_sqlalchemy_columnmatch the new signature.Verification successful
Function usage verified!
The function
get_schema_fields_for_sqlalchemy_columnis correctly called with theinspectorparameter in the codebase.
- Confirmed occurrences in test cases and other parts of the codebase.
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Test: Search for the function usage. Expect: Only occurrences of the new signature. rg --type python -A 5 $'get_schema_fields_for_sqlalchemy_column'Length of output: 94
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Find all Python files and search for the function usage. Expect: Only occurrences of the new signature. fd -e py -x rg 'get_schema_fields_for_sqlalchemy_column' {}Length of output: 620
Script:
#!/bin/bash # Description: Verify all function calls to `get_schema_fields_for_sqlalchemy_column` match the new signature. # Find all Python files and search for the function usage with more context lines. Expect: Only occurrences of the new signature. fd -e py -x rg -A 10 -B 2 'get_schema_fields_for_sqlalchemy_column' {}Length of output: 4341
metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json (5)
261-261: Simplification of nativeDataType is correct.The
nativeDataTypefield is correctly simplified from"NUMBER(asdecimal=False)"to"NUMBER".
383-383: Simplification of nativeDataType is correct.The
nativeDataTypefield is correctly simplified from"NUMBER(asdecimal=False)"to"NUMBER".
508-508: Simplification of nativeDataType is correct.The
nativeDataTypefield is correctly simplified from"NUMBER(asdecimal=False)"to"NUMBER".
757-757: Simplification of nativeDataType is correct.The
nativeDataTypefield is correctly simplified from"NUMBER(asdecimal=False)"to"NUMBER".
879-879: Simplification of nativeDataType is correct.The
nativeDataTypefield is correctly simplified from"NUMBER(asdecimal=False)"to"NUMBER".
…s by compiling data type using dialect
Masterchen09
force-pushed
the
fix-sqlalchemy-data-types
branch
from
1a07c6d
to
0e254e3
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (26)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py (3 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py (7 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py (2 hunks)
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py (4 hunks)
- metadata-ingestion/tests/integration/hana/docker-compose.yml (1 hunks)
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json (49 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json (9 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json (10 hunks)
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json (9 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json (7 hunks)
- metadata-ingestion/tests/integration/oracle/test_oracle.py (1 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json (11 hunks)
- metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_no_db_to_file.json (24 hunks)
- metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_no_db_with_filter.json (14 hunks)
- metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_to_file.json (14 hunks)
- metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_with_lower_case_urn.json (14 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json (26 hunks)
- metadata-ingestion/tests/integration/trino/trino_mces_golden.json (18 hunks)
- metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.py (7 hunks)
Files skipped from review due to trivial changes (3)
- metadata-ingestion/tests/integration/mysql/mysql_mces_no_db_golden.json
- metadata-ingestion/tests/integration/postgres/postgres_all_db_mces_with_db_golden.json
- metadata-ingestion/tests/integration/postgres/postgres_mces_with_db_golden.json
Files skipped from review as they are similar to previous changes (18)
- metadata-ingestion/src/datahub/ingestion/source/sql/athena.py
- metadata-ingestion/src/datahub/ingestion/source/sql/hive.py
- metadata-ingestion/src/datahub/ingestion/source/sql/hive_metastore.py
- metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py
- metadata-ingestion/src/datahub/ingestion/source/sql/trino.py
- metadata-ingestion/src/datahub/ingestion/source/sql/vertica.py
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py
- metadata-ingestion/tests/integration/hana/docker-compose.yml
- metadata-ingestion/tests/integration/hana/hana_mces_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_mces_with_db_golden.json
- metadata-ingestion/tests/integration/mysql/mysql_table_level_only.json
- metadata-ingestion/tests/integration/mysql/mysql_table_row_count_estimate_only.json
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_database.json
- metadata-ingestion/tests/integration/oracle/golden_test_ingest_with_out_database.json
- metadata-ingestion/tests/integration/oracle/test_oracle.py
- metadata-ingestion/tests/integration/trino/trino_hive_instance_mces_golden.json
- metadata-ingestion/tests/integration/trino/trino_hive_mces_golden.json
- metadata-ingestion/tests/unit/utilities/test_sqlalchemy_type_converter.py
Additional comments not posted (44)
metadata-ingestion/tests/integration/trino/trino_mces_golden.json (11)
259-259: Simplified data type representation.The
nativeDataTypefor theidfield has been correctly updated to "INTEGER".
271-271: Simplified data type representation.The
nativeDataTypefor thenamefield has been correctly updated to "VARCHAR(50)".
283-283: Simplified data type representation.The
nativeDataTypefor theauthorfield has been correctly updated to "VARCHAR(50)".
295-295: Simplified data type representation.The
nativeDataTypefor thepublisherfield has been correctly updated to "VARCHAR(50)".
307-307: Simplified data type representation.The
nativeDataTypefor thetagsfield has been correctly updated to "JSON".
507-507: Simplified data type representation.The
nativeDataTypefor thebook_idfield has been correctly updated to "INTEGER".
519-519: Simplified data type representation.The
nativeDataTypefor themember_idfield has been correctly updated to "INTEGER".
531-531: Simplified data type representation.The
nativeDataTypefor theissue_datefield has been correctly updated to "DATE".
543-543: Simplified data type representation.The
nativeDataTypefor thereturn_datefield has been correctly updated to "DATE".
726-726: Simplified data type representation.The
nativeDataTypefor theidfield has been correctly updated to "INTEGER".
738-738: Simplified data type representation.The
nativeDataTypefor thenamefield has been correctly updated to "VARCHAR(50)".metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_no_db_with_filter.json (9)
115-119: Verify the correctness of job metadata fields.Ensure that the new values for
job_id,job_name,description,date_created, anddate_modifiedare accurate and consistent with the rest of the metadata.
1307-1307: Approved: Data type representation change.The
nativeDataTypefor theIDfield has been updated toINTEGER, which aligns with SQL Server best practices.
1319-1319: Approved: Data type representation change.The
nativeDataTypefor theProductNamefield has been updated toNVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS, which aligns with SQL Server best practices.
1549-1549: Approved: Data type representation change.The
nativeDataTypefor theIDfield has been updated toINTEGER, which aligns with SQL Server best practices.
1561-1561: Approved: Data type representation change.The
nativeDataTypefor theItemNamefield has been updated toNVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS, which aligns with SQL Server best practices.
1681-1681: Approved: Data type representation change.The
nativeDataTypefor theIDfield has been updated toINTEGER, which aligns with SQL Server best practices.
1694-1694: Approved: Data type representation change.The
nativeDataTypefor theLastNamefield has been updated toVARCHAR(255) COLLATE SQL_Latin1_General_CP1_CI_AS, which aligns with SQL Server best practices.
1706-1706: Approved: Data type representation change.The
nativeDataTypefor theFirstNamefield has been updated toVARCHAR(255) COLLATE SQL_Latin1_General_CP1_CI_AS, which aligns with SQL Server best practices.
1718-1718: Approved: Data type representation change.The
nativeDataTypefor theAgefield has been updated toINTEGER, which aligns with SQL Server best practices.metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_to_file.json (8)
115-115: Update confirmed:job_idfield.The
job_idfield has been updated to"c6fb6778-14f1-4516-bb41-e5eaa97a687b", reflecting a new job execution context.
118-119: Update confirmed:date_createdanddate_modifiedfields.The
date_createdfield has been updated to"2024-07-27 23:58:29.780000"and thedate_modifiedfield has been updated to"2024-07-27 23:58:29.943000", reflecting new timestamps for job execution.
1307-1307: Update confirmed:nativeDataTypeforIDfield.The
nativeDataTypefor theIDfield has been updated to"INTEGER", aligning with SQL Server's expected data type specifications.
1319-1319: Update confirmed:nativeDataTypeforProductNamefield.The
nativeDataTypefor theProductNamefield has been updated to"NVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS", aligning with SQL Server's expected data type specifications.
1549-1549: Update confirmed:nativeDataTypeforIDfield inItemstable.The
nativeDataTypefor theIDfield in theItemstable has been updated to"INTEGER", aligning with SQL Server's expected data type specifications.
1561-1561: Update confirmed:nativeDataTypeforItemNamefield.The
nativeDataTypefor theItemNamefield has been updated to"NVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS", aligning with SQL Server's expected data type specifications.
1681-1681: Update confirmed:nativeDataTypeforIDfield inPersonstable.The
nativeDataTypefor theIDfield in thePersonstable has been updated to"INTEGER", aligning with SQL Server's expected data type specifications.
1694-1694: Update confirmed:nativeDataTypeforLastNameandFirstNamefields.The
nativeDataTypefor theLastNameandFirstNamefields has been updated to"VARCHAR(255) COLLATE SQL_Latin1_General_CP1_CI_AS", aligning with SQL Server's expected data type specifications.Also applies to: 1706-1706
metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_with_lower_case_urn.json (8)
115-115: Change approved: Updatedjob_id.The
job_idhas been updated to a new UUID. This change is straightforward and does not introduce any issues.
118-118: Change approved: Updateddate_created.The
date_createdtimestamp has been updated to a more recent date. This change is straightforward and does not introduce any issues.
119-119: Change approved: Updateddate_modified.The
date_modifiedtimestamp has been updated to a more recent date. This change is straightforward and does not introduce any issues.
1307-1307: Change approved: UpdatednativeDataTypeforIDfield.The
nativeDataTypefor theIDfield has been updated fromINTEGER()toINTEGER. This change improves clarity and compliance with SQL Server data type conventions.
1319-1319: Change approved: UpdatednativeDataTypeforProductNamefield.The
nativeDataTypefor theProductNamefield has been updated fromNVARCHAR()toNVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS. This change improves clarity and compliance with SQL Server data type conventions.
1561-1561: Change approved: UpdatednativeDataTypeforItemNamefield.The
nativeDataTypefor theItemNamefield has been updated fromNVARCHAR()toNVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS. This change improves clarity and compliance with SQL Server data type conventions.
1694-1694: Change approved: UpdatednativeDataTypeforLastNameandFirstNamefields.The
nativeDataTypefor theLastNameandFirstNamefields has been updated fromVARCHAR(length=255, collation='SQL_Latin1_General_CP1_CI_AS')toVARCHAR(255) COLLATE SQL_Latin1_General_CP1_CI_AS. This change improves clarity and compliance with SQL Server data type conventions.Also applies to: 1706-1706
1838-1838: Change approved: UpdatednativeDataTypeforSomeIdandTempIDfields.The
nativeDataTypefor theSomeIdfield has been updated fromUNIQUEIDENTIFIER()toUNIQUEIDENTIFIER, and for theTempIDfield fromINTEGER()toINTEGER. This change improves clarity and compliance with SQL Server data type conventions.Also applies to: 1850-1850
metadata-ingestion/tests/integration/sql_server/golden_files/golden_mces_mssql_no_db_to_file.json (8)
115-115: LGTM!The
job_idhas been updated to a new UUID, indicating a new job instance or process.
118-118: LGTM!The
date_createdhas been updated to a more recent date, indicating a refresh of the metadata.
119-119: LGTM!The
date_modifiedhas been updated to a more recent date, indicating a refresh of the metadata.
1307-1307: LGTM!The
nativeDataTypefor theIDfield has been simplified fromINTEGER()toINTEGER, aligning with standard SQL syntax.
1319-1319: LGTM!The
nativeDataTypefor theProductNamefield has been updated toNVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS, aligning with standard SQL syntax and specifying collation.
1549-1549: LGTM!The
nativeDataTypefor theIDfield has been simplified fromINTEGER()toINTEGER, aligning with standard SQL syntax.
1561-1561: LGTM!The
nativeDataTypefor theItemNamefield has been updated toNVARCHAR(max) COLLATE SQL_Latin1_General_CP1_CI_AS, aligning with standard SQL syntax and specifying collation.
1681-1681: LGTM!The
nativeDataTypefor theIDfield has been simplified fromINTEGER()toINTEGER, aligning with standard SQL syntax.
metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py
Outdated
Show resolved
Hide resolved
hsheth2
added
the
merge-pending-ci
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py (4 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/utilities/sqlalchemy_type_converter.py
hsheth2
changed the title
* feat(forms) Handle deleting forms references when hard deleting forms (datahub-project#10820) * refactor(ui): Misc improvements to the setup ingestion flow (ingest uplift 1/2) (datahub-project#10764) Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> * fix(ingestion/airflow-plugin): pipeline tasks discoverable in search (datahub-project#10819) * feat(ingest/transformer): tags to terms transformer (datahub-project#10758) Co-authored-by: Aseem Bansal <[email protected]> * fix(ingestion/unity-catalog): fixed issue with profiling with GE turned on (datahub-project#10752) Co-authored-by: Aseem Bansal <[email protected]> * feat(forms) Add java SDK for form entity PATCH + CRUD examples (datahub-project#10822) * feat(SDK) Add java SDK for structuredProperty entity PATCH + CRUD examples (datahub-project#10823) * feat(SDK) Add StructuredPropertyPatchBuilder in python sdk and provide sample CRUD files (datahub-project#10824) * feat(forms) Add CRUD endpoints to GraphQL for Form entities (datahub-project#10825) * add flag for includeSoftDeleted in scroll entities API (datahub-project#10831) * feat(deprecation) Return actor entity with deprecation aspect (datahub-project#10832) * feat(structuredProperties) Add CRUD graphql APIs for structured property entities (datahub-project#10826) * add scroll parameters to openapi v3 spec (datahub-project#10833) * fix(ingest): correct profile_day_of_week implementation (datahub-project#10818) * feat(ingest/glue): allow ingestion of empty databases from Glue (datahub-project#10666) Co-authored-by: Harshal Sheth <[email protected]> * feat(cli): add more details to get cli (datahub-project#10815) * fix(ingestion/glue): ensure date formatting works on all platforms for aws glue (datahub-project#10836) * fix(ingestion): fix datajob patcher (datahub-project#10827) * fix(smoke-test): add suffix in temp file creation (datahub-project#10841) * feat(ingest/glue): add helper method to permit user or group ownership (datahub-project#10784) * feat(): Show data platform instances in policy modal if they are set on the policy (datahub-project#10645) Co-authored-by: Hendrik Richert <[email protected]> * docs(patch): add patch documentation for how implementation works (datahub-project#10010) Co-authored-by: John Joyce <[email protected]> * fix(jar): add missing custom-plugin-jar task (datahub-project#10847) * fix(): also check exceptions/stack trace when filtering log messages (datahub-project#10391) Co-authored-by: John Joyce <[email protected]> * docs(): Update posts.md (datahub-project#9893) Co-authored-by: Hyejin Yoon <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * chore(ingest): update acryl-datahub-classify version (datahub-project#10844) * refactor(ingest): Refactor structured logging to support infos, warnings, and failures structured reporting to UI (datahub-project#10828) Co-authored-by: John Joyce <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * fix(restli): log aspect-not-found as a warning rather than as an error (datahub-project#10834) * fix(ingest/nifi): remove duplicate upstream jobs (datahub-project#10849) * fix(smoke-test): test access to create/revoke personal access tokens (datahub-project#10848) * fix(smoke-test): missing test for move domain (datahub-project#10837) * ci: update usernames to not considered for community (datahub-project#10851) * env: change defaults for data contract visibility (datahub-project#10854) * fix(ingest/tableau): quote special characters in external URL (datahub-project#10842) * fix(smoke-test): fix flakiness of auto complete test * ci(ingest): pin dask dependency for feast (datahub-project#10865) * fix(ingestion/lookml): liquid template resolution and view-to-view cll (datahub-project#10542) * feat(ingest/audit): add client id and version in system metadata props (datahub-project#10829) * chore(ingest): Mypy 1.10.1 pin (datahub-project#10867) * docs: use acryl-datahub-actions as expected python package to install (datahub-project#10852) * docs: add new js snippet (datahub-project#10846) * refactor(ingestion): remove company domain for security reason (datahub-project#10839) * fix(ingestion/spark): Platform instance and column level lineage fix (datahub-project#10843) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(ingestion/tableau): optionally ingest multiple sites and create site containers (datahub-project#10498) Co-authored-by: Yanik Häni <[email protected]> * fix(ingestion/looker): Add sqlglot dependency and remove unused sqlparser (datahub-project#10874) * fix(manage-tokens): fix manage access token policy (datahub-project#10853) * Batch get entity endpoints (datahub-project#10880) * feat(system): support conditional write semantics (datahub-project#10868) * fix(build): upgrade vercel builds to Node 20.x (datahub-project#10890) * feat(ingest/lookml): shallow clone repos (datahub-project#10888) * fix(ingest/looker): add missing dependency (datahub-project#10876) * fix(ingest): only populate audit stamps where accurate (datahub-project#10604) * fix(ingest/dbt): always encode tag urns (datahub-project#10799) * fix(ingest/redshift): handle multiline alter table commands (datahub-project#10727) * fix(ingestion/looker): column name missing in explore (datahub-project#10892) * fix(lineage) Fix lineage source/dest filtering with explored per hop limit (datahub-project#10879) * feat(conditional-writes): misc updates and fixes (datahub-project#10901) * feat(ci): update outdated action (datahub-project#10899) * feat(rest-emitter): adding async flag to rest emitter (datahub-project#10902) Co-authored-by: Gabe Lyons <[email protected]> * feat(ingest): add snowflake-queries source (datahub-project#10835) * fix(ingest): improve `auto_materialize_referenced_tags_terms` error handling (datahub-project#10906) * docs: add new company to adoption list (datahub-project#10909) * refactor(redshift): Improve redshift error handling with new structured reporting system (datahub-project#10870) Co-authored-by: John Joyce <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * feat(ui) Finalize support for all entity types on forms (datahub-project#10915) * Index ExecutionRequestResults status field (datahub-project#10811) * feat(ingest): grafana connector (datahub-project#10891) Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> * fix(gms) Add Form entity type to EntityTypeMapper (datahub-project#10916) * feat(dataset): add support for external url in Dataset (datahub-project#10877) * docs(saas-overview) added missing features to observe section (datahub-project#10913) Co-authored-by: John Joyce <[email protected]> * fix(ingest/spark): Fixing Micrometer warning (datahub-project#10882) * fix(structured properties): allow application of structured properties without schema file (datahub-project#10918) * fix(data-contracts-web) handle other schedule types (datahub-project#10919) * fix(ingestion/tableau): human-readable message for PERMISSIONS_MODE_SWITCHED error (datahub-project#10866) Co-authored-by: Harshal Sheth <[email protected]> * Add feature flag for view defintions (datahub-project#10914) Co-authored-by: Ethan Cartwright <[email protected]> * feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction (datahub-project#10884) * fix(airflow): add error handling around render_template() (datahub-project#10907) * feat(ingestion/sqlglot): add optional `default_dialect` parameter to sqlglot lineage (datahub-project#10830) * feat(mcp-mutator): new mcp mutator plugin (datahub-project#10904) * fix(ingest/bigquery): changes helper function to decode unicode scape sequences (datahub-project#10845) * feat(ingest/postgres): fetch table sizes for profile (datahub-project#10864) * feat(ingest/abs): Adding azure blob storage ingestion source (datahub-project#10813) * fix(ingest/redshift): reduce severity of SQL parsing issues (datahub-project#10924) * fix(build): fix lint fix web react (datahub-project#10896) * fix(ingest/bigquery): handle quota exceeded for project.list requests (datahub-project#10912) * feat(ingest): report extractor failures more loudly (datahub-project#10908) * feat(ingest/snowflake): integrate snowflake-queries into main source (datahub-project#10905) * fix(ingest): fix docs build (datahub-project#10926) * fix(ingest/snowflake): fix test connection (datahub-project#10927) * fix(ingest/lookml): add view load failures to cache (datahub-project#10923) * docs(slack) overhauled setup instructions and screenshots (datahub-project#10922) Co-authored-by: John Joyce <[email protected]> * fix(airflow): Add comma parsing of owners to DataJobs (datahub-project#10903) * fix(entityservice): fix merging sideeffects (datahub-project#10937) * feat(ingest): Support System Ingestion Sources, Show and hide system ingestion sources with Command-S (datahub-project#10938) Co-authored-by: John Joyce <[email protected]> * chore() Set a default lineage filtering end time on backend when a start time is present (datahub-project#10925) Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> * Added relationships APIs to V3. Added these generic APIs to V3 swagger doc. (datahub-project#10939) * docs: add learning center to docs (datahub-project#10921) * doc: Update hubspot form id (datahub-project#10943) * chore(airflow): add python 3.11 w/ Airflow 2.9 to CI (datahub-project#10941) * fix(ingest/Glue): column upstream lineage between S3 and Glue (datahub-project#10895) * fix(ingest/abs): split abs utils into multiple files (datahub-project#10945) * doc(ingest/looker): fix doc for sql parsing documentation (datahub-project#10883) Co-authored-by: Harshal Sheth <[email protected]> * fix(ingest/bigquery): Adding missing BigQuery types (datahub-project#10950) * fix(ingest/setup): feast and abs source setup (datahub-project#10951) * fix(connections) Harden adding /gms to connections in backend (datahub-project#10942) * feat(siblings) Add flag to prevent combining siblings in the UI (datahub-project#10952) * fix(docs): make graphql doc gen more automated (datahub-project#10953) * feat(ingest/athena): Add option for Athena partitioned profiling (datahub-project#10723) * fix(spark-lineage): default timeout for future responses (datahub-project#10947) * feat(datajob/flow): add environment filter using info aspects (datahub-project#10814) * fix(ui/ingest): correct privilege used to show tab (datahub-project#10483) Co-authored-by: Kunal-kankriya <[email protected]> * feat(ingest/looker): include dashboard urns in browse v2 (datahub-project#10955) * add a structured type to batchGet in OpenAPI V3 spec (datahub-project#10956) * fix(ui): scroll on the domain sidebar to show all domains (datahub-project#10966) * fix(ingest/sagemaker): resolve incorrect variable assignment for SageMaker API call (datahub-project#10965) * fix(airflow/build): Pinning mypy (datahub-project#10972) * Fixed a bug where the OpenAPI V3 spec was incorrect. The bug was introduced in datahub-project#10939. (datahub-project#10974) * fix(ingest/test): Fix for mssql integration tests (datahub-project#10978) * fix(entity-service) exist check correctly extracts status (datahub-project#10973) * fix(structuredProps) casing bug in StructuredPropertiesValidator (datahub-project#10982) * bugfix: use anyOf instead of allOf when creating references in openapi v3 spec (datahub-project#10986) * fix(ui): Remove ant less imports (datahub-project#10988) * feat(ingest/graph): Add get_results_by_filter to DataHubGraph (datahub-project#10987) * feat(ingest/cli): init does not actually support environment variables (datahub-project#10989) * fix(ingest/graph): Update get_results_by_filter graphql query (datahub-project#10991) * feat(ingest/spark): Promote beta plugin (datahub-project#10881) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(ingest): support domains in meta -> "datahub" section (datahub-project#10967) * feat(ingest): add `check server-config` command (datahub-project#10990) * feat(cli): Make consistent use of DataHubGraphClientConfig (datahub-project#10466) Deprecates get_url_and_token() in favor of a more complete option: load_graph_config() that returns a full DatahubClientConfig. This change was then propagated across previous usages of get_url_and_token so that connections to DataHub server from the client respect the full breadth of configuration specified by DatahubClientConfig. I.e: You can now specify disable_ssl_verification: true in your ~/.datahubenv file so that all cli functions to the server work when ssl certification is disabled. Fixes datahub-project#9705 * fix(ingest/s3): Fixing container creation when there is no folder in path (datahub-project#10993) * fix(ingest/looker): support platform instance for dashboards & charts (datahub-project#10771) * feat(ingest/bigquery): improve handling of information schema in sql parser (datahub-project#10985) * feat(ingest): improve `ingest deploy` command (datahub-project#10944) * fix(backend): allow excluding soft-deleted entities in relationship-queries; exclude soft-deleted members of groups (datahub-project#10920) - allow excluding soft-deleted entities in relationship-queries - exclude soft-deleted members of groups * fix(ingest/looker): downgrade missing chart type log level (datahub-project#10996) * doc(acryl-cloud): release docs for 0.3.4.x (datahub-project#10984) Co-authored-by: John Joyce <[email protected]> Co-authored-by: RyanHolstien <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Pedro Silva <[email protected]> * fix(protobuf/build): Fix protobuf check jar script (datahub-project#11006) * fix(ui/ingest): Support invalid cron jobs (datahub-project#10998) * fix(ingest): fix graph config loading (datahub-project#11002) Co-authored-by: Pedro Silva <[email protected]> * feat(docs): Document __DATAHUB_TO_FILE_ directive (datahub-project#10968) Co-authored-by: Harshal Sheth <[email protected]> * fix(graphql/upsertIngestionSource): Validate cron schedule; parse error in CLI (datahub-project#11011) * feat(ece): support custom ownership type urns in ECE generation (datahub-project#10999) * feat(assertion-v2): changed Validation tab to Quality and created new Governance tab (datahub-project#10935) * fix(ingestion/glue): Add support for missing config options for profiling in Glue (datahub-project#10858) * feat(propagation): Add models for schema field docs, tags, terms (datahub-project#2959) (datahub-project#11016) Co-authored-by: Chris Collins <[email protected]> * docs: standardize terminology to DataHub Cloud (datahub-project#11003) * fix(ingestion/transformer): replace the externalUrl container (datahub-project#11013) * docs(slack) troubleshoot docs (datahub-project#11014) * feat(propagation): Add graphql API (datahub-project#11030) Co-authored-by: Chris Collins <[email protected]> * feat(propagation): Add models for Action feature settings (datahub-project#11029) * docs(custom properties): Remove duplicate from sidebar (datahub-project#11033) * feat(models): Introducing Dataset Partitions Aspect (datahub-project#10997) Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> * feat(propagation): Add Documentation Propagation Settings (datahub-project#11038) * fix(models): chart schema fields mapping, add dataHubAction entity, t… (datahub-project#11040) * fix(ci): smoke test lint failures (datahub-project#11044) * docs: fix learning center color scheme & typo (datahub-project#11043) * feat: add cloud main page (datahub-project#11017) Co-authored-by: Jay <[email protected]> * feat(restore-indices): add additional step to also clear system metadata service (datahub-project#10662) Co-authored-by: John Joyce <[email protected]> * docs: fix typo (datahub-project#11046) * fix(lint): apply spotless (datahub-project#11050) * docs(airflow): example query to get datajobs for a dataflow (datahub-project#11034) * feat(cli): Add run-id option to put sub-command (datahub-project#11023) Adds an option to assign run-id to a given put command execution. This is useful when transformers do not exist for a given ingestion payload, we can follow up with custom metadata and assign it to an ingestion pipeline. * fix(ingest): improve sql error reporting calls (datahub-project#11025) * fix(airflow): fix CI setup (datahub-project#11031) * feat(ingest/dbt): add experimental `prefer_sql_parser_lineage` flag (datahub-project#11039) * fix(ingestion/lookml): enable stack-trace in lookml logs (datahub-project#10971) * (chore): Linting fix (datahub-project#11015) * chore(ci): update deprecated github actions (datahub-project#10977) * Fix ALB configuration example (datahub-project#10981) * chore(ingestion-base): bump base image packages (datahub-project#11053) * feat(cli): Trim report of dataHubExecutionRequestResult to max GMS size (datahub-project#11051) * fix(ingestion/lookml): emit dummy sql condition for lookml custom condition tag (datahub-project#11008) Co-authored-by: Harshal Sheth <[email protected]> * fix(ingestion/powerbi): fix issue with broken report lineage (datahub-project#10910) * feat(ingest/tableau): add retry on timeout (datahub-project#10995) * change generate kafka connect properties from env (datahub-project#10545) Co-authored-by: david-leifker <[email protected]> * fix(ingest): fix oracle cronjob ingestion (datahub-project#11001) Co-authored-by: david-leifker <[email protected]> * chore(ci): revert update deprecated github actions (datahub-project#10977) (datahub-project#11062) * feat(ingest/dbt-cloud): update metadata_endpoint inference (datahub-project#11041) * build: Reduce size of datahub-frontend-react image by 50-ish% (datahub-project#10878) Co-authored-by: david-leifker <[email protected]> * fix(ci): Fix lint issue in datahub_ingestion_run_summary_provider.py (datahub-project#11063) * docs(ingest): update developing-a-transformer.md (datahub-project#11019) * feat(search-test): update search tests from datahub-project#10408 (datahub-project#11056) * feat(cli): add aspects parameter to DataHubGraph.get_entity_semityped (datahub-project#11009) Co-authored-by: Harshal Sheth <[email protected]> * docs(airflow): update min version for plugin v2 (datahub-project#11065) * doc(ingestion/tableau): doc update for derived permission (datahub-project#11054) Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Harshal Sheth <[email protected]> * fix(py): remove dep on types-pkg_resources (datahub-project#11076) * feat(ingest/mode): add option to exclude restricted (datahub-project#11081) * fix(ingest): set lastObserved in sdk when unset (datahub-project#11071) * doc(ingest): Update capabilities (datahub-project#11072) * chore(vulnerability): Log Injection (datahub-project#11090) * chore(vulnerability): Information exposure through a stack trace (datahub-project#11091) * chore(vulnerability): Comparison of narrow type with wide type in loop condition (datahub-project#11089) * chore(vulnerability): Insertion of sensitive information into log files (datahub-project#11088) * chore(vulnerability): Risky Cryptographic Algorithm (datahub-project#11059) * chore(vulnerability): Overly permissive regex range (datahub-project#11061) Co-authored-by: Harshal Sheth <[email protected]> * fix: update customer data (datahub-project#11075) * fix(models): fixing the datasetPartition models (datahub-project#11085) Co-authored-by: John Joyce <[email protected]> * fix(ui): Adding view, forms GraphQL query, remove showing a fallback error message on unhandled GraphQL error (datahub-project#11084) Co-authored-by: John Joyce <[email protected]> * feat(docs-site): hiding learn more from cloud page (datahub-project#11097) * fix(docs): Add correct usage of orFilters in search API docs (datahub-project#11082) Co-authored-by: Jay <[email protected]> * fix(ingest/mode): Regexp in mode name matcher didn't allow underscore (datahub-project#11098) * docs: Refactor customer stories section (datahub-project#10869) Co-authored-by: Jeff Merrick <[email protected]> * fix(release): fix full/slim suffix on tag (datahub-project#11087) * feat(config): support alternate hashing algorithm for doc id (datahub-project#10423) Co-authored-by: david-leifker <[email protected]> Co-authored-by: John Joyce <[email protected]> * fix(emitter): fix typo in get method of java kafka emitter (datahub-project#11007) * fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect (datahub-project#10898) Co-authored-by: Harshal Sheth <[email protected]> * chore: Update contributors list in PR labeler (datahub-project#11105) * feat(ingest): tweak stale entity removal messaging (datahub-project#11064) * fix(ingestion): enforce lastObserved timestamps in SystemMetadata (datahub-project#11104) * fix(ingest/powerbi): fix broken lineage between chart and dataset (datahub-project#11080) * feat(ingest/lookml): CLL support for sql set in sql_table_name attribute of lookml view (datahub-project#11069) * docs: update graphql docs on forms & structured properties (datahub-project#11100) * test(search): search openAPI v3 test (datahub-project#11049) * fix(ingest/tableau): prevent empty site content urls (datahub-project#11057) Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(entity-client): implement client batch interface (datahub-project#11106) * fix(snowflake): avoid reporting warnings/info for sys tables (datahub-project#11114) * fix(ingest): downgrade column type mapping warning to info (datahub-project#11115) * feat(api): add AuditStamp to the V3 API entity/aspect response (datahub-project#11118) * fix(ingest/redshift): replace r'\n' with '\n' to avoid token error redshift serverless… (datahub-project#11111) * fix(entiy-client): handle null entityUrn case for restli (datahub-project#11122) * fix(sql-parser): prevent bad urns from alter table lineage (datahub-project#11092) * fix(ingest/bigquery): use small batch size if use_tables_list_query_v2 is set (datahub-project#11121) * fix(graphql): add missing entities to EntityTypeMapper and EntityTypeUrnMapper (datahub-project#10366) * feat(ui): Changes to allow editable dataset name (datahub-project#10608) Co-authored-by: Jay Kadambi <[email protected]> * fix: remove saxo (datahub-project#11127) * feat(mcl-processor): Update mcl processor hooks (datahub-project#11134) * fix(openapi): fix openapi v2 endpoints & v3 documentation update * Revert "fix(openapi): fix openapi v2 endpoints & v3 documentation update" This reverts commit 573c1cb. * docs(policies): updates to policies documentation (datahub-project#11073) * fix(openapi): fix openapi v2 and v3 docs update (datahub-project#11139) * feat(auth): grant type and acr values custom oidc parameters support (datahub-project#11116) * fix(mutator): mutator hook fixes (datahub-project#11140) * feat(search): support sorting on multiple fields (datahub-project#10775) * feat(ingest): various logging improvements (datahub-project#11126) * fix(ingestion/lookml): fix for sql parsing error (datahub-project#11079) Co-authored-by: Harshal Sheth <[email protected]> * feat(docs-site) cloud page spacing and content polishes (datahub-project#11141) * feat(ui) Enable editing structured props on fields (datahub-project#11042) * feat(tests): add md5 and last computed to testResult model (datahub-project#11117) * test(openapi): openapi regression smoke tests (datahub-project#11143) * fix(airflow): fix tox tests + update docs (datahub-project#11125) * docs: add chime to adoption stories (datahub-project#11142) * fix(ingest/databricks): Updating code to work with Databricks sdk 0.30 (datahub-project#11158) * fix(kafka-setup): add missing script to image (datahub-project#11190) * fix(config): fix hash algo config (datahub-project#11191) * test(smoke-test): updates to smoke-tests (datahub-project#11152) * fix(elasticsearch): refactor idHashAlgo setting (datahub-project#11193) * chore(kafka): kafka version bump (datahub-project#11211) * readd UsageStatsWorkUnit * fix merge problems * change logo --------- Co-authored-by: Chris Collins <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: dushayntAW <[email protected]> Co-authored-by: sagar-salvi-apptware <[email protected]> Co-authored-by: Aseem Bansal <[email protected]> Co-authored-by: Kevin Chun <[email protected]> Co-authored-by: jordanjeremy <[email protected]> Co-authored-by: skrydal <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> Co-authored-by: david-leifker <[email protected]> Co-authored-by: sid-acryl <[email protected]> Co-authored-by: Julien Jehannet <[email protected]> Co-authored-by: Hendrik Richert <[email protected]> Co-authored-by: Hendrik Richert <[email protected]> Co-authored-by: RyanHolstien <[email protected]> Co-authored-by: Felix Lüdin <[email protected]> Co-authored-by: Pirry <[email protected]> Co-authored-by: Hyejin Yoon <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: cburroughs <[email protected]> Co-authored-by: ksrinath <[email protected]> Co-authored-by: Mayuri Nehate <[email protected]> Co-authored-by: Kunal-kankriya <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: ipolding-cais <[email protected]> Co-authored-by: Tamas Nemeth <[email protected]> Co-authored-by: Shubham Jagtap <[email protected]> Co-authored-by: haeniya <[email protected]> Co-authored-by: Yanik Häni <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: 808OVADOZE <[email protected]> Co-authored-by: noggi <[email protected]> Co-authored-by: Nicholas Pena <[email protected]> Co-authored-by: Jay <[email protected]> Co-authored-by: ethan-cartwright <[email protected]> Co-authored-by: Ethan Cartwright <[email protected]> Co-authored-by: Nadav Gross <[email protected]> Co-authored-by: Patrick Franco Braz <[email protected]> Co-authored-by: pie1nthesky <[email protected]> Co-authored-by: Joel Pinto Mata (KPN-DSH-DEX team) <[email protected]> Co-authored-by: Ellie O'Neil <[email protected]> Co-authored-by: Ajoy Majumdar <[email protected]> Co-authored-by: deepgarg-visa <[email protected]> Co-authored-by: Tristan Heisler <[email protected]> Co-authored-by: Andrew Sikowitz <[email protected]> Co-authored-by: Davi Arnaut <[email protected]> Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: amit-apptware <[email protected]> Co-authored-by: Sam Black <[email protected]> Co-authored-by: Raj Tekal <[email protected]> Co-authored-by: Steffen Grohsschmiedt <[email protected]> Co-authored-by: jaegwon.seo <[email protected]> Co-authored-by: Renan F. Lima <[email protected]> Co-authored-by: Matt Exchange <[email protected]> Co-authored-by: Jonny Dixon <[email protected]> Co-authored-by: Pedro Silva <[email protected]> Co-authored-by: Pinaki Bhattacharjee <[email protected]> Co-authored-by: Jeff Merrick <[email protected]> Co-authored-by: skrydal <[email protected]> Co-authored-by: AndreasHegerNuritas <[email protected]> Co-authored-by: jayasimhankv <[email protected]> Co-authored-by: Jay Kadambi <[email protected]> Co-authored-by: David Leifker <[email protected]>
@hsheth2 As discussed yesterday - here is the (draft) PR for you to check the issue with the native data types in the SQLAlchemy sources. 😊
Checklist
Summary by CodeRabbit
New Features
Inspectorparameter.Bug Fixes
nativeDataTypefields across various integrations by removing unnecessary parentheses and attributes.Documentation
Chores