Feature: Add option to remove the source_column_name on the union_rel…

…ations macro (#624)

* Changed the union macro to function with `none` as an argument

* Updated the README with the new functionality

* Added to the CHANGELOG

* Update README.md

* Added integration test for omitting source column on union_relations macro

* Update integration_tests/tests/generic/expect_table_columns_to_match_set.sql

CHANGELOG.md

@@ -8,6 +8,14 @@
 ## Contributors:
 --->
 
+# Unreleased
+
+## New features
+- New feature to omit the `source_column_name` column on the `union_relations` macro ([#331](https://github.com/dbt-labs/dbt-utils/issues/331), [#624](https://github.com/dbt-labs/dbt-utils/pull/624))
+
+## Contributors:
+- [@christineberger](https://github.com/christineberger) (#624)
+
 # dbt-utils v0.8.6
 
 ## New features

README.md

@@ -886,7 +886,7 @@ final query. Note the `include` and `exclude` arguments are mutually exclusive.
 * `column_override` (optional): A dictionary of explicit column type overrides,
 e.g. `{"some_field": "varchar(100)"}`.``
 * `source_column_name` (optional, `default="_dbt_source_relation"`): The name of
-the column that records the source of this row.
+the column that records the source of this row. Pass `None` to omit this column from the results.
 * `where` (optional): Filter conditions to include in the `where` clause.
 
 #### generate_series ([source](macros/sql/generate_series.sql))

integration_tests/models/sql/schema.yml

@@ -162,6 +162,11 @@ models:
       - name: favorite_number
         tests:
           - dbt_utils.not_constant
+
+  - name: test_union_no_source_column
+    tests:
+      - expect_table_columns_to_match_set:
+          column_list: ["id", "name", "favorite_color", "favorite_number"]
 
   - name: test_get_relations_by_pattern
     tests:

integration_tests/models/sql/test_union_no_source_column.sql

@@ -0,0 +1,6 @@
+{{ dbt_utils.union_relations([
+       ref('data_union_table_1'),
+       ref('data_union_table_2')
+   ],
+   source_column_name = none
+) }}

integration_tests/tests/generic/expect_table_columns_to_match_set.sql

@@ -0,0 +1,54 @@
+{#
+    This macro is copied and slightly edited from the dbt_expectations package.
+    At the time of this addition, dbt_expectations couldn't be added because
+    integration_tests is installing dbt_utils from local without a hard-coded 
+    path. dbt is not able to resolve duplicate dependencies of dbt_utils
+    due to this.
+#}
+
+{%- test expect_table_columns_to_match_set(model, column_list, transform="upper") -%}
+{%- if execute -%}
+    {%- set column_list = column_list | map(transform) | list -%}
+    
+    {# Replaces dbt_expectations._get_column_list() #} 
+    {%- set relation_column_names = adapter.get_columns_in_relation(model) 
+                                    | map(attribute="name") 
+                                    | map(transform) 
+                                    | list 
+    -%}
+
+    {# Replaces dbt_expectations._list_intersect() #}
+    {%- set matching_columns = [] -%}
+    {%- for itm in column_list -%}
+        {%- if itm in relation_column_names -%}
+            {%- do matching_columns.append(itm) -%}
+        {%- endif -%}
+    {%- endfor -%}
+
+    with relation_columns as (
+
+        {% for col_name in relation_column_names %}
+        select cast('{{ col_name }}' as {{ dbt_utils.type_string() }}) as relation_column
+        {% if not loop.last %}union all{% endif %}
+        {% endfor %}
+    ),
+    input_columns as (
+
+        {% for col_name in column_list %}
+        select cast('{{ col_name }}' as {{ dbt_utils.type_string() }}) as input_column
+        {% if not loop.last %}union all{% endif %}
+        {% endfor %}
+    )
+    select *
+    from
+        relation_columns r
+        full outer join
+        input_columns i on r.relation_column = i.input_column
+    where
+        -- catch any column in input list that is not in the list of table columns
+        -- or any table column that is not in the input list
+        r.relation_column is null or
+        i.input_column is null
+
+{%- endif -%}
+{%- endtest -%}

macros/sql/union.sql

@@ -85,7 +85,10 @@
         (
             select
 
+                {%- if source_column_name != none %}
                 cast({{ dbt_utils.string_literal(relation) }} as {{ dbt_utils.type_string() }}) as {{ source_column_name }},
+                {%- endif %}
+
                 {% for col_name in ordered_column_names -%}
 
                     {%- set col = column_superset[col_name] %}