Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2537] Include "compiled" node attributes in run results #7519

Closed
Tracked by #8316
jtcohen6 opened this issue May 5, 2023 · 1 comment · Fixed by #8492
Closed
Tracked by #8316

[CT-2537] Include "compiled" node attributes in run results #7519

jtcohen6 opened this issue May 5, 2023 · 1 comment · Fixed by #8492
Assignees
Labels
artifacts enhancement New feature or request

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented May 5, 2023

Proposed change

Let's include attributes that can vary at compile/runtime. Those attributes are:

  • compiled: bool = True (always False after parsing)
  • compiled_code (always empty after parsing) - this is most important
  • relation_name - this is set after parsing, but it can be updated for tests that have --store-failures enabled
  • depends_on.macros ([CT-500] Investigate differences in manifest.json output for 'compile' and 'run' commands #5079) - since additional macro dependencies are required/registered while a model is being materialized, versus just compiled (e.g. create_table_as)

(Here is where we set many of those attributes, during compilation)

The run_results.json produced by compile, docs generate, run/build/test/etc should include these fields (and only these fields) in addition to the node's unique_id. All other fields can be accessed from the manifest, and represent logical state at a point in time.

Background

  • Way back in v0.19, we removed the full node entry from run_results.json, and moved the compiled_code attribute into the manifest, so as to power dbt-docs from only manifest + catalog.
  • Starting in v1.4 (CT 1604 remove compiled classes #6384), we narrowed down the set of fields that differ between "uncompiled" and "compiled" nodes.

Rationale

  • The manifest represents "logical" state (parsed from project code + configs)
  • Run results + catalog represent "applied" state (as materialized by dbt, as the objects exist in the data warehouse)

Compiled code can be a form of "applied" state. If a model's SQL depends on the results of an introspective query, it can vary given different inputs from the data warehouse.

Acceptance Criteria

  • The run_results.json produced by compile, docs generate, run, build, and test should include the compiled, compiled_code, relation_name, and depends_on.macros fields (and only these fields) in addition to the node's unique_id.
  • The same information should also be included in the results returned by dbtRunner when is used to invoke dbt as a library.
@github-actions github-actions bot changed the title Include "compiled" node attributes in run results [CT-2537] Include "compiled" node attributes in run results May 5, 2023
@iknox-fa
Copy link
Contributor

@jtcohen6 per BLG: We're estimating this as written but have some concerns and would like to understand the use-case better-- Maybe a situation that would call for a new runtime artifact, compile_results.json?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
artifacts enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants