Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing idle issue and improving debug logging #648

Merged
merged 6 commits into from
May 1, 2024
Merged

Conversation

benc-db
Copy link
Collaborator

@benc-db benc-db commented Apr 30, 2024

Hopefully resolves #647

Description

Customers with high concurrency on SQL Warehouses have been hitting an issue where they see 'Remote disconnected' as the error. It appears somewhere in the stack, connections that have been idle for 180+s hit this issue when we attempt to reuse them. This PR has the following:

  • Move default time for considering a connection stale to 60s (most of the benefit of reusing connections comes from having many quick operations anyway, since the overhead of creating and closing connections is sub-second).
  • Significant refactoring of logging to improve clarity and consistency in debug logs
  • I noticed that BearerAuth was invoking the header factory on create rather than call, which I believe is what was causing Authentication Issue: Polling Python Job Status - Access Token Cached on Creation #647

Checklist

I should probably add tests for the logging, but I will hold that for when I bring this code into 2.0.0 branch (to reduce the significant pain that will already be).

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

@benc-db benc-db merged commit cdf5803 into main May 1, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Authentication Issue: Polling Python Job Status - Access Token Cached on Creation
1 participant