Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalization: Support SSH tunneling for postgres destination #5294

Closed
sherifnada opened this issue Aug 10, 2021 · 1 comment
Closed

Normalization: Support SSH tunneling for postgres destination #5294

sherifnada opened this issue Aug 10, 2021 · 1 comment

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Aug 10, 2021

Tell us about the problem you're trying to solve

Adding support for SSH tunnels for any database destination will require adding support in two places:

  1. in the database connector itself
  2. in normalization

this issue is split from #312 to capture the fact that it needs to be added to normalization.

Describe the solution you’d like

Today DBT is invoked from this line in this bash script. The simplest way to add ssh tunneling functionality to normalization is by adding a function/block of code after configuredbt and before dbt run which initializes the SSH tunnel and removes it afterwards (potentially using trap).

I don't love this implementation mainly because I don't love the idea of making an already non trivial bash script even less trivial. However, invoking DBT directly from Python is not recommended. So our options are either:

  1. run a python script which starts the SSH tunnel and forks another process to run dbt run
  2. keep using the bash script

If we go with option 2 the bash addition we'd add would probably:

  1. read the config.json (bound to the $CONFIG_FILE var) input into the connector and detects if SSH configuration is present. The currently proposed JsonSchema for configuring SSH tunnel functionality can be found here in the spec.json file.
  2. Depending on the kind of SSH configuration (ssh key vs. username/password based) run the right ssh command. See the SSH Tunnel section in this RFC for an example of the command we'd need to run to get an ssh tunnel using private keys.
@sherifnada
Copy link
Contributor Author

Once this ticket is done we should create an equivalent one for MySQL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment