[CT-2531] [Regression] v1.5.0 cannot run "dbt deps" in Dockerfile - "Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist."

[AndrewLane]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

If I am installing dbt v1.5.0 in a Dockerfile and then run dbt deps later, I get a strange failure:

Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist.

Expected Behavior

I should be able to use dbt deps in my Dockerfile as I have before.

Steps To Reproduce

With the following 4 files in a directory:

dbt_project.yml

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: "foo"
version: "1.0.0"
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: "foo"

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

packages-install-path: "dbt_packages" # directory where packages are installed
clean-targets: # directories to be removed by `dbt clean`
  - "dbt_packages"

Dockerfile-1.4.2

FROM python:3.8.15-slim-bullseye

RUN apt-get update && apt-get install -y --no-install-recommends git

WORKDIR /dbt
COPY packages.yml dbt_project.yml /dbt/

RUN pip install --no-cache-dir dbt-core==1.4.2

RUN dbt deps

Dockerfile-1.5.0

FROM python:3.8.15-slim-bullseye

RUN apt-get update && apt-get install -y --no-install-recommends git

WORKDIR /dbt
COPY packages.yml dbt_project.yml /dbt/

RUN pip install --no-cache-dir dbt-core==1.5.0

RUN dbt deps

packages.yml

packages:
  - git: "https://github.com/EqualExperts/dbt-unit-testing"
    revision: v0.2.6

If I build my v1.4.2 image, it works:

docker build . --tag 1.4.2-deleteme --file Dockerfile-1.4.2

However, a similar command with v1.5.0 fails.

docker build . --tag 1.5.0-deleteme --file Dockerfile-1.5.0

Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist.

Relevant log output

>docker build . --tag 1.5.0-deleteme --file Dockerfile-1.5.0 
[+] Building 2.9s (10/10) FINISHED
 => [internal] load build definition from Dockerfile-1.5.0                                                                                 0.0s 
 => => transferring dockerfile: 274B                                                                                                       0.0s 
 => [internal] load .dockerignore                                                                                                          0.0s 
 => => transferring context: 2B                                                                                                            0.0s 
 => [internal] load metadata for docker.io/library/python:3.8.15-slim-bullseye                                                             0.3s 
 => [internal] load build context                                                                                                          0.0s 
 => => transferring context: 68B                                                                                                           0.0s 
 => [1/6] FROM docker.io/library/python:3.8.15-slim-bullseye@sha256:b5abf044b0a94dbfc0463b9f7bd1e13010a096c64b98a441c7dabb5fb44b0308       0.0s 
 => CACHED [2/6] RUN apt-get update && apt-get install -y --no-install-recommends git                                                      0.0s 
 => CACHED [3/6] WORKDIR /dbt                                                                                                              0.0s 
 => CACHED [4/6] COPY packages.yml dbt_project.yml /dbt/                                                                                   0.0s 
 => CACHED [5/6] RUN pip install --no-cache-dir dbt-core==1.5.0                                                                            0.0s 
 => ERROR [6/6] RUN dbt deps                                                                                                               2.5s 
------
 > [6/6] RUN dbt deps:
#10 2.356 Usage: dbt deps [OPTIONS]
#10 2.356 Try 'dbt deps -h' for help.
#10 2.356
#10 2.356 Error: Invalid value for '--profiles-dir': Path '/root/.dbt' does not exist.
------
executor failed running [/bin/sh -c dbt deps]: exit code: 2

Environment

No response

Which database adapter are you using with dbt?

No response

Additional Context

No response

[dbeatty10]

Thanks for a nice a write-up @AndrewLane !

I haven't fully researched this, but starting in 1.5, the --profiles-dir path is being validated. When that parameter isn't supplied, the default is a subdirectory of the user's home.

Without digging deeper, I don't know why --profiles-dir is an option for dbt deps.

Workaround

In the meantime, here's a workaround you can use:

RUN dbt deps --profiles-dir .

Opting against RUN dbt deps || true as the recommended workaround because it would hide any other relevant failures during dbt deps.

[dbeatty10]

After a little more digging, I found where this validation is happening:

dbt-core/core/dbt/cli/params.py

Line 292 in fd73066

type=click.Path(exists=True),

There's at least four different ways we can choose to address this:

1. Leave all behavior as-is for 1.5 and add a note to the migration guide so folks know how to make updates
2. Use type=click.Path(exists=False) to ignore path validation
3. Give dbt deps it's own custom version of the --profiles-dir parameter that ignores path validation (similar to Option 2 above)
4. Remove @p.profiles_dir from dbt deps

@jtcohen6 I'd be inclined to try out Option 2 since it would preserve the behavior of <=1.4 ubiquitously across commands. Alternatively, Option 4 would be appropriate if dbt deps doesn't actually need this parameter. Is there an option that you prefer?

[girotobial]

I was going to add a new issue but I think it's just as relevant here. On a first time setup of dbt I am receiving the same error message when calling dbt init jaffle_shop as per the setup guide.

Usage: dbt init [OPTIONS] [PROJECT_NAME]
Try 'dbt init -h' for help.

Error: Invalid value for '--profiles-dir': Path '/home/girotobial/.dbt' does not exist.

[jtcohen6]

Quick thoughts:

• dbt deps does not need --profiles-dir, it does not load a profile at all
• dbt init does need --profiles-dir, but it's perfectly valid if it doesn't yet exist — the init command will write it!

I'm happy with Option 2: switch the parameter to type=click.Path(exists=False) to turn off this validation for all commands.