Long sessions #517
Merged
Long sessions #517
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added USE_LONG_SESSIONS flag. Extended Connection with DatabricksDBTConnection. Added properties acquire_release_count and last_used_time. Updated DatabricksConnectionManager.set_connection_name to create instance of DatabricksDBTConnection and update new properties. Override ConnectionManager.release to not close the session and to update the new connection properties. Signed-off-by: Raymond Cypher <[email protected]>
Added new connection pool (threads_compute_connections) to DatabricksConnectionManager. It is a map of thread ID to map of compute name to DatabricksDBTConnection. Updated DatabricksConnectionManager.set_connection_name() to look for existing connections in the new pool and then update the existing thread_connections connection pool. Overrode cleanup_all in DatabricksConnectionManager to fire events based on the connection acquire/release count, rather than the connection state. Signed-off-by: Raymond Cypher <[email protected]>
Added _start_using and _stop_using to DatabricksDBTConnection. These handle logging, last_used_time, and acquire_release_count. Moved long session specific code from DatabricksConnectionManager.set_connection_name to its own method. Refactoring and renaming of the long session code. Signed-off-by: Raymond Cypher <[email protected]>
Signed-off-by: Raymond Cypher <[email protected]>
Signed-off-by: Raymond Cypher <[email protected]>
Signed-off-by: Raymond Cypher <[email protected]>
rcypher-databricks
requested review from
andrefurlan-db,
susodapop and
benc-db
as code owners
benc-db
requested changes
Nov 28, 2023
Use ConnectionState constants instead of string literals. Use assert instead of throwing an exception if in unexpected code path when USE_LONG_SESSIONS=False Added long session test with warehouse per model Signed-off-by: Raymond Cypher <[email protected]>
Added default max idle time of 600 seconds. Added code to read user specified max idle time from profile or from alternate compute definitions. Updated DatabricksDBTConnection with connect_max_idle property and helper functions to determine if the connection should be cleaned up. In DatabricksConnectionManager added _cleanup_idle_connections(). This is called whenever a connection is acquired for use. Added a new class method _open2() to DatabricksConnectionManager. This is used with USE_LONG_SESSIONS is true and uses the http_path property of DatabricksDBTConnection. Signed-off-by: Raymond Cypher <[email protected]>
benc-db
reviewed
Dec 6, 2023
benc-db
reviewed
Dec 6, 2023
benc-db
reviewed
Dec 6, 2023
benc-db
approved these changes
Dec 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #
Description
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.