Selector Upgrade #7101

acurtis-evi · 2023-03-01T20:32:52Z

resolves #5009

Description

There are some really cool features in DBT such as the ability to choose models and their descendents, graph ordered builds, and much more. However, I found that some of the syntax seemed to be lacking (particularly around the selectors yml syntax).

The definition part of the selector yaml supports some model selection operators, but requires that the end user use a different approach to describe what can be done in short-hand on the command line. For example, on the command line, I can select the children of a model using

dbt ls --select some_model+

In the yaml,

- name: some_selector
  description:
    - some_model+

does not work and instead

- name: some_selector
  description:
    - method: fqn
       value: some_model
       children: true

must be used.

The yaml is very useful in defining routine selections. This inconsistency with the command line forces the end user to learn two approaches.

Furthermore, there is support from the command line to union and intersect using the --select operator. However, if one wishes to exclude, the separate --exclude operator is needed. The syntax is very straight forward for the command line approach, but again the yaml approach is different. I see value in the yaml approach. I just believe that the two should be compatible. In this PR, I propose that the --select can include NOT (all caps - but easy to change) as a way of embedding excluded models in a single string.

dbt ls --select 'selector:some_selector+1 NOT selector:some_selector'

In addition, dbt doesn't allow selectors to be expanded using the children/parent/ancestor operators. This PR addresses that and makes selectors just like any other method.

Syntax in the yaml is also supported

- definition: selector:some_other_selector+1 NOT some_model+2 some_other_model
    name: some_selector

Most of the code removes the selectors dictionary from being passed around and instead opts to pass the project to the graph/cli get_selected and get_graph_queue methods.

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

…,selector:other_selector

acurtis-evi · 2023-03-01T20:43:52Z

core/dbt/graph/selector.py

+            return self.get_selected(self.project.get_selector(spec.value))
+        else:
+            method = self.get_method(spec.method, spec.method_arguments)
+            return set(method.search(included_nodes, spec.value))



The self.project.get_selector is the only place that checks if selectors actually exist

acurtis-evi · 2023-03-01T20:44:17Z

core/dbt/config/selectors.py

-            sel_def = definition.get("value")
-            if sel_def not in selector_dict:
-                raise DbtSelectorsError(f"Existing selector definition for {sel_def} not found.")
-            return selector_dict[definition["value"]]["definition"]
        return definition


Only checked in graph/selector.py (or config/project.py)

acurtis-evi · 2023-03-01T20:45:06Z

core/dbt/config/selectors.py

-            definition = cls.parse_from_definition(
-                selector["definition"], selector_dict=deepcopy(selector_dict)
+            selector_dict[sel_name]["definition"] = cls.parse_from_definition(
+                selector["definition"]
            )


The selector_dict doesn't initially resolve the selectors. This is left until the end to allow for it to be more dynamic

acurtis-evi · 2023-03-01T20:46:25Z

core/dbt/graph/cli.py

@@ -33,28 +33,64 @@ def parse_union(
    # turn ['a b', 'c'] -> ['a', 'b', 'c']
    raw_specs = itertools.chain.from_iterable(r.split(" ") for r in components)
    union_components: List[SelectionSpec] = []
+    exclude_components: List[SelectionSpec] = []
+    in_exclude = False
+
    flags = get_flags()


Allow for --select 'some_expression --exclude some_other_expression', having it be one string allows for the yaml syntax to support union/intersection/exclude in a single string

acurtis-evi · 2023-03-01T20:46:58Z

core/dbt/graph/cli.py

-        sel_def = definition.get("value")
-        if sel_def not in result:
-            raise DbtValidationError(f"Existing selector definition for {sel_def} not found.")
-        return result[definition["value"]]["definition"]
    elif "method" in definition and "value" in definition:


moved to graph/selector.py / config/project.py

acurtis-evi · 2023-03-01T20:47:41Z

core/dbt/graph/selector_methods.py

@@ -45,6 +45,7 @@ class MethodName(StrEnum):
    Metric = "metric"
    Result = "result"
    SourceStatus = "source_status"
+    Selector = "selector"



The "selector" is added without a corresponding class

acurtis-evi · 2023-03-01T20:48:05Z

core/dbt/task/list.py

@@ -58,7 +58,7 @@ def __init__(self, args, config, manifest):
    def _iterate_selected_nodes(self):
        selector = self.get_node_selector()
        spec = self.get_selection_spec()
-        nodes = sorted(selector.get_selected(spec))
+        nodes = sorted(selector.get_selected(spec, self.config))


get_selected needs the project to call get_selector

acurtis-evi · 2023-03-01T20:48:27Z

core/dbt/task/runnable.py

@@ -125,7 +125,7 @@ def defer_to_manifest(self, adapter, selected_uids: AbstractSet[str]):
    def get_graph_queue(self) -> GraphQueue:
        selector = self.get_node_selector()
        spec = self.get_selection_spec()
-        return selector.get_graph_queue(spec)
+        return selector.get_graph_queue(spec, self.config)



get_graph_queue needs the project to call get_selector

acurtis-evi · 2023-03-02T18:09:32Z

The --exclude operator was meant to work like a NOT operator like ...

dbt ls --select 'selector:some_selector+1 NOT selector:some_selector,... something_else'

where everything to the right of NOT would be used to exclude from the left.

The reason to choose --exclude was that it was very unlikely that it would conflict with another model and break anything. Also, the cli could just append --exclude to the --select.

With the selector being added, it effectively allows for parenthetical support and very advanced expressions.

The exact syntax could easily change if it is desired.

acurtis-evi · 2023-03-03T16:38:22Z

@gshank and @stu-k - I believe this is ready from a functional perspective. I've never actually contributed anything to DBT, so not sure what else should be done

jtcohen6 · 2023-03-03T21:47:33Z

@acurtis-evi Very cool! :)

I definitely want to take a look at this, and play around with some of the proposed additions from a UX perspective, before moving into more-detailed code review with the dev team. (@dbeatty10 too, if you're able & interested!)

dbeatty10 · 2023-03-03T23:49:37Z

The --exclude operator was meant to work like a NOT operator like ...

dbt ls --select 'selector:some_selector+1 NOT selector:some_selector,... something_else'
where everything to the right of NOT would be used to exclude from the left.

Just to make sure I'm following, this is a set difference, right?

A quick brainstorm of ideas relating to the syntax:

NOT
not
exclude
minus
except
- (minus sign)
∖ (Set Minus Unicode U+2216)
\ (backslash)

acurtis-evi · 2023-03-04T00:11:39Z

It is a set difference. In this PR, the set difference part is isolated to one function parse_union in dbt/core/graph/cli.py.

The idea is that it would work like

dbt ls --select 'selector:some_selector+1' --exclude 'selector:some_selector,... something_else'

The other ideas in this PR are more powerful, particularly the ability to allow the selector yaml syntax to be identical to command line syntax and to be able to expand upon selectors themselves (selector:some_selector+1 for example)

acurtis-evi · 2023-03-04T00:12:40Z

I think EXCEPT might be a better name than NOT

acurtis-evi · 2023-03-07T18:53:13Z

Changed NOT => EXCEPT

Support using --selector + --select / --exclude at the same time. A new selector named arg_selector is introduced which is the combination of --select/--exclude and if --selector is specified, that remains the primary action.

dbt ls --selector arg_selector --select some_model

acurtis-evi · 2023-03-07T18:57:51Z

There is also support for the arg_selector in the selectors.yml. This PR moves the selectors.yml validation to runtime, allowing for selectors to be out of order and to refer to the arg_selector. It is possible to have cycles with this approach, but I tend to think that the benefit of the selection flexibility significantly outweighs the downside of it.

github-actions · 2024-01-20T01:47:07Z

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

mroy-seedbox · 2024-01-20T02:24:39Z

It would be nice to be able to run selectors minus one or two models.

Right now, the only way to do so is to define another selector, which is a cumbersome process in some environment (but would be very simple on the command line).

mroy-seedbox · 2024-11-13T01:49:14Z

Alternative (and perhaps simpler) implementation proposed here: #10992.

acurtis-evi added 2 commits March 1, 2023 15:16

Allow selectors to be in general syntax like selector:some_selector+1…

70d1922

…,selector:other_selector

Allow --exclude to be specified in the middle of the union

098c832

acurtis-evi requested a review from a team March 1, 2023 20:32

acurtis-evi requested a review from a team as a code owner March 1, 2023 20:32

acurtis-evi requested review from stu-k and gshank March 1, 2023 20:32

cla-bot bot added the cla:yes label Mar 1, 2023

acurtis-evi mentioned this pull request Mar 1, 2023

[CT-469] Combine --select/--exclude with --selector when used together? #5009

Open

1 task

acurtis-evi commented Mar 1, 2023

View reviewed changes

acurtis-evi added 3 commits March 1, 2023 16:14

Fixed a couple of issues during testing

7a0e92d

removed print statement

e267f07

Added changie

37fc04a

acurtis-evi requested a review from a team as a code owner March 1, 2023 21:26

Updated changie

ce1bd66

jtcohen6 added the Team:Execution label Mar 1, 2023

Change --exclude => all caps NOT

1325b81

Merge branch 'main' into selector

96eb532

acurtis-evi added 3 commits March 7, 2023 12:41

NOT => EXCLUDE

0fd780e

Support dbt run --selector arg_selector+ --select some_models

9273f86

EXCLUDE => EXCEPT

ec540e0

acurtis-evi mentioned this pull request Mar 8, 2023

1.4.next acurtis-evi/dbt-core#5

Draft

6 tasks

jtcohen6 self-assigned this Mar 22, 2023

jtcohen6 added the Refinement Maintainer input needed label Jun 25, 2023

jtcohen6 removed the Team:Execution label Jul 19, 2023

gshank removed request for gshank and stu-k July 19, 2023 14:37

github-actions bot added the stale Issues that have gone stale label Jan 20, 2024

github-actions bot removed the stale Issues that have gone stale label Jan 22, 2024

dbeatty10 added the community This PR is from a community member label Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Selector Upgrade #7101

Selector Upgrade #7101

acurtis-evi commented Mar 1, 2023 •

edited by dbeatty10

Loading

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi Mar 1, 2023

acurtis-evi commented Mar 2, 2023 •

edited

Loading

acurtis-evi commented Mar 3, 2023

jtcohen6 commented Mar 3, 2023

dbeatty10 commented Mar 3, 2023

acurtis-evi commented Mar 4, 2023

acurtis-evi commented Mar 4, 2023

acurtis-evi commented Mar 7, 2023

acurtis-evi commented Mar 7, 2023

github-actions bot commented Jan 20, 2024

mroy-seedbox commented Jan 20, 2024

mroy-seedbox commented Nov 13, 2024

Selector Upgrade #7101

Are you sure you want to change the base?

Selector Upgrade #7101

Conversation

acurtis-evi commented Mar 1, 2023 • edited by dbeatty10 Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acurtis-evi commented Mar 2, 2023 • edited Loading

acurtis-evi commented Mar 3, 2023

jtcohen6 commented Mar 3, 2023

dbeatty10 commented Mar 3, 2023

acurtis-evi commented Mar 4, 2023

acurtis-evi commented Mar 4, 2023

acurtis-evi commented Mar 7, 2023

acurtis-evi commented Mar 7, 2023

github-actions bot commented Jan 20, 2024

mroy-seedbox commented Jan 20, 2024

mroy-seedbox commented Nov 13, 2024

acurtis-evi commented Mar 1, 2023 •

edited by dbeatty10

Loading

acurtis-evi commented Mar 2, 2023 •

edited

Loading