Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2478] Excessive Template Rendering During Parse of Internal Analytics #7449

Closed
peterallenwebb opened this issue Apr 24, 2023 · 0 comments · Fixed by #7451
Closed

[CT-2478] Excessive Template Rendering During Parse of Internal Analytics #7449

peterallenwebb opened this issue Apr 24, 2023 · 0 comments · Fixed by #7451
Assignees
Labels
enhancement New feature or request performance

Comments

@peterallenwebb
Copy link
Contributor

peterallenwebb commented Apr 24, 2023

Profiling dbt parse runs on our internal analytics project reveals repetitive rendering of simple string literals as templates. The problem is significant enough that patching it makes parsing a noticeable 12% faster. I'll open a PR shortly which mitigates this somewhat, but the list below should illuminate the nature of the problem. Each string we are rendering as a template in native mode is included, prefixed by the number of times it is rendered. Each time that is done, the template is parsed, converted to python, and then executed. The word 'table' is going through this relatively slow process 41 times!

   2 ad_reporting
   2 amazon_ads
   2 amazon_ads_source
   2 analyses
   2 analysis
   1 analytics_dev
   2 apple_search_ads
   2 apple_search_ads_source
   1 assets
   1 audit_helper
   1 calogica/dbt_date
   2 calogica/dbt_expectations
   3 cloud_users
   1 coalesce
   1 codegen
   2 data
   1 datamocktool
   1 dbt
   2 dbt-labs/audit_helper
   1 dbt-labs/codegen
   1 dbt-labs/dbt_external_tables
   6 dbt-labs/dbt_utils
   1 dbt-labs/metrics
   1 dbt-labs/snowplow
  17 dbt-labs/spark_utils
   1 dbt_datamocktool
   1 dbt_date
   1 dbt_expectations
   1 dbt_external_tables
   4 dbt_modules
   8 dbt_packages
   1 dbt_pwebb
   1 dbt_snow_mask
   1 dbt_snowflake
   1 dbt_utils
   1 dbtlabs_pwebb
   1 delete+insert
   1 dev
   2 diff_when_downstream
   1 docs
   1 entechlog/dbt_snow_mask
   5 ephemeral
   1 externalbrowser
   2 facebook_ads
   2 facebook_ads_source
   2 facebook_pages
   1 facebook_pages_source
   1 fishtown_internal_analytics
   1 fivetran/ad_reporting
   1 fivetran/amazon_ads
   1 fivetran/amazon_ads_source
   1 fivetran/apple_search_ads
   1 fivetran/apple_search_ads_source
   1 fivetran/facebook_ads
   1 fivetran/facebook_ads_source
   1 fivetran/facebook_pages
   1 fivetran/facebook_pages_source
  18 fivetran/fivetran_utils
   1 fivetran/google_ads
   1 fivetran/google_ads_source
   1 fivetran/hubspot
   1 fivetran/hubspot_source
   1 fivetran/instagram_business
   1 fivetran/instagram_business_source
   1 fivetran/linkedin
   1 fivetran/linkedin_pages
   1 fivetran/linkedin_pages_source
   1 fivetran/linkedin_source
   1 fivetran/microsoft_ads
   1 fivetran/microsoft_ads_source
   1 fivetran/pendo
   1 fivetran/pendo_source
   1 fivetran/pinterest
   1 fivetran/pinterest_source
   1 fivetran/reddit_ads
   1 fivetran/reddit_ads_source
   1 fivetran/snapchat_ads
   1 fivetran/snapchat_ads_source
   1 fivetran/social_media_reporting
   1 fivetran/tiktok_ads
   1 fivetran/tiktok_ads_source
   1 fivetran/twitter_ads
   1 fivetran/twitter_ads_source
   1 fivetran/twitter_organic
   1 fivetran/twitter_organic_source
   1 fivetran/zendesk_source
   1 fivetran_utils
   1 fqn
   2 garage-snowflake
   2 google_ads
   2 google_ads_source
   2 hubspot
   1 hubspot_source
   1 incremental
   2 instagram_business
   1 instagram_business_source
   1 int_ad_reporting
   1 int_pendo
   1 integration_tests
   1 kw27752
   1 linkedin
   1 linkedin_ads
   1 linkedin_ads_source
   2 linkedin_pages
   1 linkedin_pages_source
   1 linkedin_source
   6 logs
  14 macros
   1 metrics
   2 microsoft_ads
   2 microsoft_ads_source
   1 mjirv/dbt_datamocktool
   4 models
   2 pendo
   1 pendo_source
   2 pinterest
   2 pinterest_source
   2 reddit_ads
   2 reddit_ads_source
   2 salesforce
   2 seeds
   3 selector
   1 single_tenant_freshness_test
   2 snapchat_ads
   2 snapchat_ads_source
   3 snapshots
   1 snowflake
   1 snowplow
   2 social_media_reporting
   1 source:bizzabo+
   1 source:slack+
   1 source:stitch_youtube+
   1 source:twitter+
   1 spark_utils
   1 stg_facebook_pages
   1 stg_hubspot
   1 stg_instagram_business
   1 stg_linkedin_pages
   1 stg_pendo
   1 stg_snowflake__users
   1 stg_tiktok_ads
   1 stg_twitter_organic
   1 sync_all_columns
  41 table
  20 target
   1 test
   4 tests
   2 tiktok_ads
   1 tiktok_ads_source
   1 transformer
   1 transforming
   2 twitter_ads
   2 twitter_ads_source
   2 twitter_organic
   1 twitter_organic_source
   1 user
  23 view
@peterallenwebb peterallenwebb added enhancement New feature or request performance labels Apr 24, 2023
@peterallenwebb peterallenwebb self-assigned this Apr 24, 2023
@github-actions github-actions bot changed the title Excessive Template Rendering During Parse of Internal Analytics [CT-2478] Excessive Template Rendering During Parse of Internal Analytics Apr 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants