Add migration script to remove size limits from namespaces, dataset n… #1925
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ames, and source connection urls Signed-off-by: Michael Collado <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #1925 +/- ##
=========================================
Coverage 77.90% 77.90%
Complexity 937 937
=========================================
Files 193 193
Lines 5218 5218
Branches 418 418
=========================================
Hits 4065 4065
Misses 706 706
Partials 447 447 📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
fm100
approved these changes
Mar 29, 2022
Comment on lines
+1
to
+10
| ALTER TABLE dataset_versions ALTER COLUMN namespace_name TYPE VARCHAR; | ||
| ALTER TABLE dataset_versions ALTER COLUMN dataset_name TYPE VARCHAR; | ||
| ALTER TABLE datasets ALTER COLUMN name TYPE VARCHAR; | ||
| ALTER TABLE datasets ALTER COLUMN physical_name TYPE VARCHAR; | ||
| ALTER TABLE datasets ALTER COLUMN source_name TYPE VARCHAR; | ||
| ALTER TABLE namespaces ALTER COLUMN name TYPE VARCHAR; | ||
| ALTER TABLE runs ALTER COLUMN external_id TYPE VARCHAR; | ||
| ALTER TABLE sources ALTER COLUMN name TYPE VARCHAR; | ||
| ALTER TABLE sources ALTER COLUMN connection_url TYPE VARCHAR; | ||
| ALTER TABLE tags ALTER COLUMN name TYPE VARCHAR; |
There was a problem hiding this comment.
So these are to change from VARCHAR(256) to VARCHAR, right?
This was referenced Mar 31, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Several columns in the database are restricted to 255 characters, perhaps without strong justification. Several of these cause problems, including namespaces, in #1758, as well as the
connection_urlwhen storing some Snowflake tables (the connection URLs contain a lot of extra parameters which aren't always scrubbed before sending the OpenLineage events). If there isn't a strong reason for restricting these columns, the default restriction on VARCHARs will work just as well with no performance penalty.Closes: #1758
Solution
Remove the size restriction on various VARCHAR types
Checklist
CHANGELOG.mdwith details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sqldatabase schema migration according to Flyway's naming convention (if relevant)