Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hotfix rsrs static query performance #58

Merged
merged 2 commits into from
Feb 15, 2023

Conversation

tkirschke
Copy link
Member

Error

  • In dbt Cloud, during an execution of "dbt build" or "dbt docs generate", the Job failed with the error message "This run exceeded your account's run memory limits."
  • Occured for Multi Source Entities in incremental runs

Cause of Error

  • The background code checks for each source if it was included before, to disable the high-water-mark in such a case for this newly added source
  • Here all the rows of each source are loaded into dbt and set as a variable
  • With high volumes of data, this variable grows very big and therefore the account's memory is exceeded

Fix

  • Instead of selecting all rows for each source, only a COUNT(*) is executed. This works fine, because later on the query results are only used to check if there are any rows, not to actually do something with the data.
  • This change is applied for all multi-source entities (Hubs, Links, NH-Links, Rec-Track Satellites), for all databases

Copy link
Contributor

@bschlottfeldt bschlottfeldt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good to me :)

@tkirschke tkirschke merged commit e458bca into main Feb 15, 2023
@tkirschke tkirschke deleted the hotfix-rsrs-static-query-performance branch February 15, 2023 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants