[CT-2516] [Bug] Upgrade to 1.5.0 generates "TypeError: argument of type 'PosixPath' is not iterable" on model runs.

[christopherekfeldt]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

We just upgraded our poetry packages for dbt-core and dbt-bigquery to 1.5.0.

When running: poetry run dbt --version everything works fine. But when calling a model like:

poetry run dbt run -s model_name
poetry run dbt build -s model_name

We get a new error never seen before.

Expected Behavior

That the models get built as prior with the normal CLI output.

Steps To Reproduce

1. Update pyproject.toml with setting dbt-core & dbt-bigquery versions to "1.5.0"
2. run "poetry update"
3. run "poetry run dbt --version"
4. Verify that it outputs the correct versions
   
5. run "poetry run dbt run -s name_of_model"

Relevant log output

14:13:06  Running with dbt=1.5.0
14:13:49  Encountered an error:
argument of type 'PosixPath' is not iterable
14:13:49  Traceback (most recent call last):
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 86, in wrapper
    result, success = func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 71, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 142, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 215, in wrapper
    return func(*args, **kwargs)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/cli/requires.py", line 242, in wrapper
    manifest = ManifestLoader.get_full_manifest(
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 227, in get_full_manifest
    manifest = loader.load()
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 386, in load
    self.parse_project(
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 539, in parse_project
    hook_parser.parse_file(file_block)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/hooks.py", line 120, in parse_file
    self.parse_node(hook)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/base.py", line 394, in parse_node
    self.render_update(node, config)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/base.py", line 370, in render_update
    context = self.render_with_context(node, config)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/base.py", line 240, in render_with_context
    get_rendered(parsed_node.raw_code, context, parsed_node, capture_macros=True)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/clients/jinja.py", line 591, in get_rendered
    return render_template(template, ctx, node)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/clients/jinja.py", line 546, in render_template
    return template.render(ctx)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
TypeError: argument of type 'PosixPath' is not iterable

Environment

- OS: Ubuntu 20.04
- Python: 3.8.10
- dbt: 1.5.0

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

[jtcohen6]

@christopherekfeldt Thanks for opening! I'm going to label this as a potential regression for v1.5.

It's weird that this is happening during project parsing. Based on the error message and occurrence, I'd guess that you're accessing a flag that's being passed in, and trying to loop over it (?).

Educated guesses - any chance you're doing one of the following?

• Referencing the invocation_args_dict context variable somewhere in your project
• Have a package installed that's doing this, i.e. dbt_artifacts

Also - does your poetry package implicitly set any other flag values for dbt, via CLI flag or env var, such as --project-dir or --profiles-dir?

[christopherekfeldt]

No I don't think so. I searched it and found nothing in our project reflecting "invocation_args_dict".

These are the packages specified in our packages.yml:

packages:

• package: dbt-labs/dbt_utils
  version: 1.0.0

• package: dbt-labs/codegen
  version: 0.9.0

• package: elementary-data/elementary
  version: 0.7.5

• package: calogica/dbt_expectations
  version: 0.8.5

On the last part my answer is no, we only have one project dir and a common profile in our dbt root folder.
The only connection to dbt apart from installing the tools dbt-core and dbt-bigquery with Poetry is the tool sqlfluff-templater-dbt = "2.0.3" which we use for pre-commit together with sqlfluff. But can't see how that is correlated with the parsing of the project.

[joellabes]

Heads up that the elementary package uses the invocation_args_dict variable:

https://github.com/elementary-data/dbt-data-reliability/blob/0.7.5/macros/edr/dbt_artifacts/upload_dbt_invocation.sql#L55-L90

[christopherekfeldt]

Okay, commented out all packages besides dbt-utils and removed dbt_packages/ folder and rerun "poetry run dbt deps".

Getting the same output still when running dbt compile/run/build -s model_name

[jtcohen6]

Heads up that the elementary package uses the invocation_args_dict variable:

That felt like a promising lead! However:

• I tried installing the package into an empty project, and I was able to run everything without a problem.
• It sounds like @christopherekfeldt removed the package, and is still seeing the same error

On looking again at the stacktrace, I'm now seeing that the error seems to crop up during hook parsing:

  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/manifest.py", line 539, in parse_project
    hook_parser.parse_file(file_block)
  File "/mnt/c/git/bica-dbt/dbt/.venv/lib/python3.8/site-packages/dbt/parser/hooks.py", line 120, in parse_file
    self.parse_node(hook)

@christopherekfeldt Do you have any on-run-start or on-run-end hooks defined in dbt_project.yml? Do any of them make use of the flags variable?

[christopherekfeldt]

These are our hooks atm:

on-run-start:

• '{{ exceptions.raise_compiler_error("Cannot use .profiles as profiles dir locally.") if "/dbt/.profiles" in flags.PROFILES_DIR }}'
• '{{ exceptions.raise_compiler_error("Cannot use defer in target %s, run "export DBT_DEFER=false"." % target.name) if flags.DEFER_MODE and target.name in ["tcm", "ver", "prd"] }}'
• '{{ exceptions.raise_compiler_error("Cannot use defer to env %s in target %s, run "export DBT_DEFER=false"." % target.name) if flags.DEFER_MODE and target.name in ["prd_ci"] and env_var("DBT_DEFER_ENV") != "prd" }}'
• '{{ exceptions.raise_compiler_error("DBT_WRAPPER not set to true. Must use bin/dbt wrapper locally!") if env_var("DBT_WRAPPER", "false") != "true" and target.name in ["lab", "dev"] }}'
• '{{ exceptions.raise_compiler_error("Models with invalid intraday tags detected. Aborting run!") if not check_intraday_tags() }}'
• '{{ mock_sources() if var("mock_sources") }}'

on-run-end:

• '{% do adapter.drop_schema(api.Relation.create(database=target.database, schema=target.schema)) if target.name.endswith("_ci") %}'

[christopherekfeldt]

So all 3 first hooks could be the bad guy here? should the logic be switched out for something else?

[jtcohen6]

Looking at the first hook you shared:

- '{{ exceptions.raise_compiler_error("Cannot use .profiles as profiles dir locally.") if "/dbt/.profiles" in flags.PROFILES_DIR }}'

That <str> in flags.PROFILES_DIR will do it! Since the value of flags.PROFILES_DIR is actually a pathlib.Path, not a string.

You can fix this immediately by explicitly converting flags.PROFILES_DIR to a string, with the Jinja | string filter:

- '{{ exceptions.raise_compiler_error("Cannot use .profiles as profiles dir locally.") if "/dbt/.profiles" in flags.PROFILES_DIR | string }}'

We could also handle this within dbt-core, and restore previous behavior, by converting all path flags to strings before exposing them to the Jinja context:

dbt-core/core/dbt/flags.py

Lines 93 to 104 in 35f8ceb

	# This is used by core/dbt/context/base.py to return a flag object
	# in Jinja.
	def get_flag_obj():
	new_flags = Namespace()
	for key, val in get_flag_dict().items():
	setattr(new_flags, key.upper(), val)
	# The following 3 are CLI arguments only so they're not full-fledged flags,
	# but we put in flags for users.
	setattr(new_flags, "FULL_REFRESH", getattr(GLOBAL_FLAGS, "FULL_REFRESH", None))
	setattr(new_flags, "STORE_FAILURES", getattr(GLOBAL_FLAGS, "STORE_FAILURES", None))
	setattr(new_flags, "WHICH", getattr(GLOBAL_FLAGS, "WHICH", None))
	return new_flags

from pathlib import Path

def get_flag_obj():
    new_flags = Namespace()
    for key, val in get_flag_dict().items():
        if isinstance(val, Path):
            val = str(val)
        setattr(new_flags, key.upper(), val)

That change makes sense to me more generally, anyway.