You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have searched the existing issues, and I could not find an existing issue for this feature
I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
Describe the feature
#6073 describes how projects with 1000's of nodes can take ~15 minutes to dbt build even a single node.
The _already_known method call is a relatively expensive method call during DAG parsing. There are some cases where it is unnecessarily called, namely when in_degree != 0here.
Swapping the conditions of the if statement would short-circuit calling _already_known while keeping the logic the same. This is shown in 9f96087. (We know we can safely swap the order by commutativity of conjunction).
Describe alternatives you've considered
There's at least two alternatives:
Default approach of changing nothing
More complicated refactor
This proposal seems obviously superior to the first alternative while not preventing us from taking up the second alternative at a later date. So it's an easy win.
Who will this benefit?
This will (slightly) benefit teams with very large DAGs (thousands of nodes).
Are you interested in contributing this feature?
There's already a PR ready for this!
Anything else?
No response
The text was updated successfully, but these errors were encountered:
github-actionsbot
changed the title
[Feature] When possible, bypass an expensive method during parsing
[CT-1881] [Feature] When possible, bypass an expensive method during parsing
Jan 23, 2023
jtcohen6
changed the title
[CT-1881] [Feature] When possible, bypass an expensive method during parsing
[CT-1881] [Feature] When possible, bypass an expensive method during graph construction
Jan 23, 2023
Is this your first time submitting a feature request?
Describe the feature
#6073 describes how projects with 1000's of nodes can take ~15 minutes to
dbt build
even a single node.The
_already_known
method call is a relatively expensive method call during DAG parsing. There are some cases where it is unnecessarily called, namely whenin_degree != 0
here.Swapping the conditions of the if statement would short-circuit calling
_already_known
while keeping the logic the same. This is shown in 9f96087. (We know we can safely swap the order by commutativity of conjunction).Describe alternatives you've considered
There's at least two alternatives:
This proposal seems obviously superior to the first alternative while not preventing us from taking up the second alternative at a later date. So it's an easy win.
Who will this benefit?
This will (slightly) benefit teams with very large DAGs (thousands of nodes).
Are you interested in contributing this feature?
There's already a PR ready for this!
Anything else?
No response
The text was updated successfully, but these errors were encountered: