-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ssh tunnel for postgres normalization #5818
Conversation
/test connector=bases/base-normalization
|
/test connector=bases/base-normalization
|
…/airbytehq/airbyte into george/new-pg-ssh-norm # Conflicts: # airbyte-integrations/bases/base-normalization/normalization/transform_config/transform.py
/test connector=bases/base-normalization
|
e63a5a3
to
7df041f
Compare
@cgardens good shout. You may have done this elsewhere but normalisation needs to be published first and then the postgres tests should work 👍 |
What
Adds support for ssh tunnelling during normalization (& custom dbt transforms if non-Kube) for postgres destination.
Custom dbt transforms don't work on Kube as per this issue.
The core additions in this PR will enable other destinations to use ssh for normalization too provided ssh support/tests have been added as in Charles' PR. There is just a small update required per destination in
airbyte-integrations/bases/base-normalization/normalization/transform_config/transform.py
.How
Added a bash script
sshtunneling.sh
that can read ssh tunnel options from the config file and if ssh tunneling is on, open an ssh tunnel using standard ssh procedure (and subsequently close it). Utilising a control-socket so we can target close the specific ssh session.For key authorization, the key gets written to a tmp file and used in standard ssh command.
For password authorization, I've used sshpass to allow passing password non-interactively.
Notes
I'm aware that the implementation in airbyte-workers for custom dbt transforms is less than ideal. This is mainly due to the fact it runs on a user provided docker image rather than one we've defined. Specifically:
jq
(andsshpass
if using password auth) are installed on their provided docker image, albeit with a janky attempt to install these if they aren't present.This can definitely be improved (TODO: add link to issue for improving this here)
New normalization will need to be published before the postgres tests can pass (I think)
Recommended reading order
Pre-merge Checklist