[CT-3281] [Feature] Add a --perf-info flag to write out perf_info.json for all commands that write a manifest

[will-sargent-dbtlabs]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Hey Core team 👋🏼
I was wondering if it would be possible to write out the perf_info.json results for more than just parse calls, maybe with a --perf-info flag.
If it wouldn't add meaningful overhead to something like a run or build command this info could be picked up by dbt cloud and logged to Datadog where we could scrape metrics from it.

I think that would be really valuable feedback, and would allow us to better see the impact of the great performance improvements you're working on!

Describe alternatives you've considered

Analysis via post processing in our internal analytics system.

The downside is that some internal parsing metrics are hidden, and the data is time-delayed by the ingestion / transformation cycle.

In addition, it also makes it impossible to prepare actual "metrics" inside datadog.

Who will this benefit?

dbt Cloud Solutions and Engineering teams.

Are you interested in contributing this feature?

no

Anything else?

Internal Slack Thread:
https://dbt-labs.slack.com/archives/C05FWBP9X1U/p1698332747811859

[dbeatty10]

Thanks for opening this @will-sargent-dbtlabs !

Currently

write_perf_info=False is the current default for all commands that include @manifest.

But for dbt parse, we override it to be write_perf_info=True here.

The net effect is that perf_info.json is written to the target-path during dbt parse, but not other commands.

Feature request

One way we could fulfill your feature request is to just switch it to always be write_perf_info=True for all relevant commands.

But assuming we want users to be able to opt-in instead, we could add a new perf_info global config that allows users to choose.

i.e. a new global config like this (from lowest precedence to highest):

• perf_info (user config)
• DBT_PERF_INFO (environment variable)
• --perf-info (CLI flag)

The CLI flag would be relevant to all of the following commands that currently have @requires.manifest:

• dbt build
• dbt docs generate
• dbt compile
• dbt show
• dbt list
• dbt run
• dbt retry
• dbt clone
• dbt run operation
• dbt seed
• dbt snapshot
• dbt source freshness
• dbt test

Alternatively, we could choose some other relevant subset.

Refinement

Labeling this as "Refinement" to make the following choices:

1. Make this always-on, or add a new global config?
2. If a global config, do we want to stick with the name perf_info? Or do we want to name it something else?
3. If adding a CLI flag, which commands to apply it to? All? Some subset?