-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python model specific session handling to prevent using invalid sessions #547
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Raymond Cypher <[email protected]>
Signed-off-by: Raymond Cypher <[email protected]>
rcypher-databricks
requested review from
andrefurlan-db,
susodapop and
benc-db
as code owners
January 10, 2024 21:03
Overriding close() in connection manager to catch/absorb errors on closing a session. For example closing a session that has timed out on the back end. Signed-off-by: Raymond Cypher <[email protected]>
rcypher-databricks
temporarily deployed
to
azure-prod
January 10, 2024 22:32
— with
GitHub Actions
Inactive
rcypher-databricks
temporarily deployed
to
azure-prod
January 10, 2024 22:33
— with
GitHub Actions
Inactive
rcypher-databricks
temporarily deployed
to
azure-prod
January 10, 2024 22:33
— with
GitHub Actions
Inactive
benc-db
previously approved these changes
Jan 10, 2024
Fixed up type annotations and removed a duplicate function definition. Signed-off-by: Raymond Cypher <[email protected]>
rcypher-databricks
changed the title
Stale sessions
Python model specific session handling to prevent using invalid sessions
Jan 11, 2024
rcypher-databricks
temporarily deployed
to
azure-prod
January 12, 2024 00:17
— with
GitHub Actions
Inactive
rcypher-databricks
had a problem deploying
to
azure-prod
January 12, 2024 00:17
— with
GitHub Actions
Failure
rcypher-databricks
temporarily deployed
to
azure-prod
January 12, 2024 00:17
— with
GitHub Actions
Inactive
Signed-off-by: Raymond Cypher <[email protected]>
rcypher-databricks
temporarily deployed
to
azure-prod
January 12, 2024 00:23
— with
GitHub Actions
Inactive
rcypher-databricks
temporarily deployed
to
azure-prod
January 12, 2024 00:23
— with
GitHub Actions
Inactive
rcypher-databricks
temporarily deployed
to
azure-prod
January 12, 2024 00:23
— with
GitHub Actions
Inactive
benc-db
reviewed
Jan 12, 2024
benc-db
approved these changes
Jan 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves:
ES-985588: dbt-databricks data models timeout
Description
A long running python model can leave a connection/session idle for long enough that it times out and is closed.
Subsequent use of the connection results in an 'Invalid Sessionhandle' error.
Cause
For non-python data models the model is executed using sql statements. So even if the sql statement takes a long time to run, the connection is in active use and doesn't timeout on the server. For a python model a connection is acquired, but the model is run using rpc calls. Thus, if the python model takes a long time to run, the session can timeout on the back end.
The code to cleanup idle connections was not catching this because when a connection is released, after running a model, the last used time is set. So even though running the python model didn't use the connection as far as the cleanup code was concerned it had been in active use.
There are two cases where the invalid connection would be used.
Fix
Updated the DatabricksDBTConnection class to track the language of the model being executed. If the language is python we do not update the last used time when the connection is released. This is to reflect the fact that the python model isn't necessarily using the connection.
This change allows the idle cleanup code to correctly cleanup the idle connection.
Checklist
CHANGELOG.md
and added information about my change to the "dbt-databricks next" section.