Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2744] Refactor: entry point for cross-project ref #7954

Closed
Tracked by #7372
jtcohen6 opened this issue Jun 26, 2023 · 0 comments · Fixed by #7955
Closed
Tracked by #7372

[CT-2744] Refactor: entry point for cross-project ref #7954

jtcohen6 opened this issue Jun 26, 2023 · 0 comments · Fixed by #7955
Assignees
Labels
multi_project tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality

Comments

@jtcohen6
Copy link
Contributor

After much spiking and iterating on how to handle "public models from an upstream project," we've landed on a really satisfying implementation (#7890): These are just models. We needn't add special-case logic (or duplicate boilerplate code) within dbt-core for handling these nodes: listing them, resolving references to them, (not) running them, etc.

This implementation gives us a much narrower entry point within dbt-core for the resolution of cross-project ref. In #6725 (reply in thread), I shared more context around our intent for this feature. It should be uniquely available for dbt Cloud (Enterprise), given the problem it's solving—collaboration across multiple teams, with dozens or hundreds of contributors. It is not a publicly committed interface within the dbt-core framework. And it will yield a functional outcome comparable to that of using packages, via a mechanism that's significantly better at scale.

In a separate discussion earlier this year (#7173), we shared our aspiration for a future in which dbt-core is a pluggable library: narrowly scoped to its core workflow, while also providing more flexible points for extensions. Now that the refactor above is complete—cleaner interfaces, more-precise entry point—we can experiment with putting all logic specific to cross-project ref within a separate plugin.

So, the scope of this issue is to finish consolidating the remnants of this logic within dbt-core; to provide ourselves with a single clean entry point; and to make a first attempt at implementing a plugin framework, on which we can keep iterating.

@jtcohen6 jtcohen6 added tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality multi_project labels Jun 26, 2023
@github-actions github-actions bot changed the title Refactor: entry point for cross-project ref [CT-2744] Refactor: entry point for cross-project ref Jun 26, 2023
@MichelleArk MichelleArk self-assigned this Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
multi_project tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants