Definitions is live in 1.1.7

[schrockn]

Hello everyone!

A couple weeks ago we posted discussion detailing our plans to move forward with a new API Definitions to replace Workspace and Repositories in our tools and UI.

As detailed there are real issues with these abstractions, especially for new users:

Workspaces don't do that much––they are a list of code locations––but occupy a lot of headspace in our UI and tools. Repositories can cause immense confusion as (1) they collide with the notion of GitHub repositories but have a different scope leading to confusing states where you can have multiple Dagster repositories within a single GitHub repository and (2) for most users they also did not provide that much value beyond the code location organizational unit, leaving them with the question of why the even exist in the first place.

Today with version 1.1.7 we are rolling this out. Our tutorials, quickstarts, templates, and documentation all "lead" with Definitions now. In addition, the concepts have been eliminated from our UI, leaving it simpler and easier to navigate.

API

A few notable changes to the API.

1. Rather than returning a list of heterogenous definitions out of a decorated function, Definitions is a vanilla object with typed, named arguments. This is better for discovery, documentation, and robustness.
2. Definitions takes a top-level resources argument, which is automatically applied to assets, rather than requiring the use of with_resources.
3. There can only be one Definitions object per code location, whereas before there could be multiple repositories in the code location.
4. The resources dictionary in Definitions can accept raw objects, not just resource definitions. Objects inheriting from IOManager are automatically coerced to IOManagerDefinition; all other objects are coerced to ResourceDefinition.

As an example, with these changes

@resource
def some_object():
   return SomeObject()
   
@repository
def a_repo():
   return [
        *with_resources(asset_requires_some, {"some_object" : some_object})
   ]

can instead be expressed with:

defs = Definitions(
    assets=[asset_requires_some],
    resources={"some_object": SomeObject()}
)

Tools

Tooling changes:

1. Dagit and CLI tools target a Python module and load a single instance of Definitions found in a symbol in that module. The symbol can be any name. (Note: this is a change from the RFC, which proposed requiring the symbol be named defs)]
2. Dagit and the Dagster CLI tools can now take multiple instances of the -m and -f. For each one of these flags it loads a separate code location.
3. All references to Workspace and Repository have been eliminated from the UI.
4. workspace.yaml is no longer the preferred solution for convenient command-line-free usage of dagit and other tools. Instead we just rely on a [tool.dagster] of pyproject.toml a Python standard (see PEP) design with extensibility points with these uses cases in mind.

FAQ for existing users

Do I have to migrate any code?

No. Our primary goal is to ease onboarding for net new users and to simplify our UI. While we think the new APIs are nicer there is no immediate imperative to migrate and we will support @repository and workspace.yaml for the foreseeable future.

However migration is straightforward for most users, and the resulting code is cleaner. See #11135 and #11072 for examples.

What will the UI look like if I have multiple repositories in a code location?

The UI has flattened the hierarchy of code location and repositories to just code locations. N repositories in a single code location will appear as N code locations in the UI. The name of that "code location" will be repo_name@location_name in the UI. The @ is not a valid character in our names, so if you see this in the context of a code location, you know that this "code location" is actually a named repository within a code location.

While somewhat counterintutive for users that use multiple repositories to organize code in the same code location, we think this is worth the tradeoff of a simpler UI and better mental model. This is a "one step backward to enable two steps forward" for those of you the rely on repositories in this fashion and we appreciate your patience and understanding. We plan on supporting other, more flexible ways of organizing definitions in the future.

I rely on having multiple repositories in the same code location for organizational purposes. What is your recommendation?

If you have a small number of repositories with many definitions inside of them, migrating to Definitions probably makes sense for you.

If you have a large number of repositories and do not want to create a large number of code locations, we don't recommend migrating to Definitions. Continue to use repositories in the near term.

I use workspace.yaml for my deployments. Is that going to change?

No. Workspace files will continue to be the way to load code locations in complex scenarios, such as connecting to code locations in gRPC servers in self-hosted OSS deployments. With this change we are moving away from their use in lightweight local development and utilizing pyproject.toml for that use case.

[markfickett]

Thanks for this writeup. I didn't see a link to the API docs, leaving it here for posterity: https://docs.dagster.io/_apidocs/definitions .

[mgierada]

The new Definitions API is a nice way to set up dagster and I am glad we have that now in the latest release. However, it is not clear how I can access or set up code location or repository names. Those are still required by dagster-cloudcli, right?

[schrockn]

In Dagster cloud the location name is set in your dagster_cloud.yaml file.

[mgierada]

Oh, thanks @schrockn! That makes sense! Just in case someone else stuck on the same issue of running dagster cloud jobs in CI/CD pipeline after switching to Definitions API, please make sure you are no longer passing repository_name in an action example_here or you set a default value like here.

[bendnorman]

How are people using Definitions to organize their code and data pipelines? Do you create a separate Definition for each data source? Do you create separate Definitions for different stages in data pipelines? Is it common for folks to organize different parts of pipelines into Definitions and then link them together using SourceAssets?

[dayakiran]

I don't see examples of loading multiple code locations in workspace.yaml. When I try to provide multiple locations under the load_from section, the dagit UI wouldn't come up. When I only have a single code location, it does come up. Do you have an example of workspace.yaml where multiple code locations can be defined?

[dayakiran]

Alternatively can we have multiple locations in pyproject.toml?

[schrockn]

@dayakiran here's an example workspace.yaml file:

load_from:
    - python_file: foo.py
    - python_file: bar.py

[schrockn]

@dayakiran re: pyproject.toml we have an issue tracking this: #11439

Feel free to chime in registering your request!

[dayakiran]

Can I use a combination of python package and file I tried something like the following : load_from : - python_package : demo - python_file : c1/test.py Each of these work individually but not as a combination. In the logs I see that dagit is trying to load the package first as a code server... then the code server is brought down. The UI doesnt come up.

…

On Wed, Jan 4, 2023, 6:28 AM Nicholas Schrock ***@***.***> wrote: @dayakiran <https://github.com/dayakiran> here's an example workspace.yaml file: load_from: - python_file: foo.py - python_file: bar.py — Reply to this email directly, view it on GitHub <#11167 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA77SWCUJ4LXAI2REIQGTWDWQVNHPANCNFSM6AAAAAATADDGWI> . You are receiving this because you were mentioned.Message ID: ***@***.***>