Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the https:// optional #165

Merged
merged 1 commit into from
May 6, 2021
Merged

Conversation

Fokko
Copy link
Contributor

@Fokko Fokko commented May 4, 2021

resolves #153

Description

This allows you to both do:

default:
  target: dev
  outputs:
    dev:
...
      host: adb-8170469668123456.8.azuredatabricks.net

and:

default:
  target: dev
  outputs:
    dev:
...
      host: https://adb-8170469668123456.8.azuredatabricks.net

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

@cla-bot cla-bot bot added the cla:yes label May 4, 2021
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this one @Fokko! I'm all about it

@@ -321,7 +321,7 @@ def open(cls, connection):
'cluster', 'organization'])

conn_url = cls.SPARK_CONNECTION_URL.format(
host=creds.host,
host=creds.host if creds.host.startswith('https://') else 'https://' + creds.host,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like flake8 is giving you a hard time for this line being too long:

dbt/adapters/spark/connections.py:324:80: E501 line too long (106 > 79 characters)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dang! Thanks for pointing it out. Just pushed a fix

Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for this @Fokko!

@jtcohen6 jtcohen6 merged commit dff1b61 into dbt-labs:master May 6, 2021
@joebeeson
Copy link

Just ran into this issue after tracing down why I couldn't set HTTP vs HTTPS on a local testing installation. Is it appropriate to assume the connection is HTTPS? Particularly with no way to configure it otherwise.

@jtcohen6
Copy link
Contributor

@joebeeson That's a fair thought. I guess I'd hope that folks are connecting to their Spark clusters via HTTPS, but you never know.

Perhaps we could switch the logic added by this PR:

https://github.com/fishtown-analytics/dbt-spark/blob/dff1b613ddf87e4e72e8a47475bcfd1d55796a5c/dbt/adapters/spark/connections.py#L323-L326

To instead be:

 if not host.startswith('http'):
       host = 'https://' + creds.host 

That way, https:// is presumed by default, but if you want to explicitly specify http://your.host.name in profiles.yml, you could.

Is that a switch you'd be interested in contributing?

@Fokko Fokko deleted the fd-make-https-optional branch May 18, 2023 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatically remove https:// in front of the host
3 participants