Validate SqoopHook connection string and disable extra options from public hook methods #33039
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
approved these changes
Aug 2, 2023
|
Pending tests passing :) |
pankajkoti
force-pushed
the
validate-sqoop-connection
branch
from
dee9865
to
d0508ef
Compare
potiuk
approved these changes
Aug 3, 2023
eladkal
reviewed
Aug 3, 2023
…ook methods Check that the connection string constructed using the connection's `host`, `port` and `schema` does not contain query params as it is not intended. Additionally, also disable the `extra_import_options` and `extra_export_options` arguments accepted directly by the hook methods but accept it as a param via the hook constructor when initialising the hook or by passing it in hook_params when initialising the hook from operators.
…nd not accepted by databases
pankajkoti
force-pushed
the
validate-sqoop-connection
branch
from
98bfad8
to
7988831
Compare
|
Test constraints check seems to be timing out repeatedly after 80 minutes. |
Yes. Seems there is a backtracking of |
|
BTW. I just realised that PR's like this should not cause the "--upgrade-to-newer-dependencies` (this is what triggers the backtracking issue) fix to CI infrastructure is coming |
|
After this one is merged : #33082 and after your rebase, the constraint issue should go away even if we do not fix it in main. |
|
Merging, regardless. |
44 tasks
ephraimbuddy
pushed a commit
that referenced
this pull request
Aug 28, 2023
…ublic hook methods (#33039) * Validate SqoopHook connection string and disable extra options from hook methods Check that the connection string constructed using the connection's `host`, `port` and `schema` does not contain query params as it is not intended. Additionally, also disable the `extra_import_options` and `extra_export_options` arguments accepted directly by the hook methods but accept it as a param via the hook constructor when initialising the hook or by passing it in hook_params when initialising the hook from operators. * Propogate missing hook param changes to the operator * Remove test for invalid port with query param as ports are integers and not accepted by databases * Add 4.0.0 to provider.yaml (cherry picked from commit 59f5f58)
97 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Check that the connection string constructed using the connection's
host,portandschemadoes not contain query params as it isnot intended. Additionally, also disable the
extra_import_optionsand
extra_export_optionsarguments accepted directly by the hookmethods but accept it as a param via the hook constructor when
initialising the hook or by passing it in
hook_paramswhen initialisingthe hook from operators.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.