Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2327] [Spike] Produce publication artifact and resolve cross-project refs #7227

Closed
Tracked by #7372 ...
MichelleArk opened this issue Mar 27, 2023 · 1 comment · Fixed by #7276
Closed
Tracked by #7372 ...

[CT-2327] [Spike] Produce publication artifact and resolve cross-project refs #7227

MichelleArk opened this issue Mar 27, 2023 · 1 comment · Fixed by #7276

Comments

@MichelleArk
Copy link
Contributor

Write a new 'publication artifact' as a result of dbt invocations. This artifact should contain the set of logical project state necessary for downstream projects to take dependencies (resolve refs) to public models in the producer project.

Minimally, the publication artifact would contain:

  • common artifact metadata (version, custom env vars, etc)
  • project name
  • quoting policy
  • the relation components (database, schema, alias) for each public model in the project
  • project-level dependencies (to prevent project-level cycles) or model-level dependencies (to prevent model-level cycles across projects)

Let's prove out that this publication artifact can be used to resolve two-variant refs, where the first variant is a project (not package) name. If the two-variant ref can be resolved with a package - prefer the package resolution to using the publication artifact.

@github-actions github-actions bot changed the title [Spike] Produce publication artifact and resolve cross-project refs [CT-2327] [Spike] Produce publication artifact and resolve cross-project refs Mar 27, 2023
@jtcohen6
Copy link
Contributor

jtcohen6 commented Apr 12, 2023

Paraphrasing from some conversations with @bossnunta

dbt list should include information about public models in upstream projects:

  • with a clear attribute saying that this is a node not in this project / an installed package (let's not call this external_nodes, since that's taken on a different meaning over the last ~year)
  • with the understanding that there will be significantly less information available about each of those nodes (only what's in the publication artifact), versus nodes defined in the project

Errors during cross-project ref

Will there be a way to tell from dbt-core when an error emerges from a bad "cross-project" reference, versus a bad intra-project reference?

When we're trying to resolve the ref, definitely!

If we can successfully resolve the ref, but then the user lacks permissions on the underlying database object... that would be trickier.

From the UX perspective, it could be compelling (P1) to tell users that their error comes from the fact that your current project does not have access to the referenced model in the upstream project. Then we can link you to a documentation page that tells you to:

  • if it's the same adapter type -- go set up permissions (Snowflake share, BigQuery project IAM, Redshift cross-db query, etc)
  • if it's a different adapter type -- we don't support this (yet)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants