[BUG] service is required by service but is disabled

[nmguse-bighealth]

Description

v2.19.0 introduced a change that broke a docker compose run command that was working previously (v2.18.1)

The change, I believe, would have come from this PR: #10602

docker compose -p unique_name run --rm --no-deps test black --line-length 150 --check app/ tests/

Error: service postgres is required by test but is disabled. Can be enabled by profiles []

I believe this is roughly all that would be required to reproduce, but I haven't directly tested this as docker mac hasn't updated docker compose yet:

version: "3"
services:
  test:
    profiles:
      - test
    build:
      context: .
    depends_on:
      postgres

  postgres:
    image: postgres:14-alpine

Steps To Reproduce

No response

Compose Version

v2.19.0

Docker Environment

No response

Anything else?

No response

[allanlewis]

I'm working around this by adding these steps to my failing job:

    - name: Log Docker Compose version
      run: docker compose version
    - name: Ensure Docker CLI plugins directory exists
      run: mkdir -pv "${HOME}"/.docker/cli-plugins
    - name: Download working version of Docker Compose
      run: wget -nv -O "${HOME}"/.docker/cli-plugins/docker-compose "${DOCKER_COMPOSE_URL}"
      env:
        DOCKER_COMPOSE_URL: https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64
    - name: Make Docker Compose binary executable
      run: chmod -v +x "${HOME}"/.docker/cli-plugins/docker-compose
    - name: Assert that the version of Docker Compose is as expected
      run: set -o pipefail && docker compose version | tee /dev/stderr | grep -q 'v2\.18\.1$'

It seems to work, and the download is very fast, presumably because it's all within GitHub.

[jvalecillos]

I can confirm that the version v2.18.1 works while 2.19.0 shows the issue. Going back to the older version for CI as suggested by @allanlewis solves the issue.

[milas]

I'm not able to reproduce this with the example given. (Tweaked slightly to fix syntax errors and not use a local build, I'm launching an container that just does nothing.)

services:
  test:
    profiles:
      - test
    image: busybox
    init: true
    command: ["sleep", "infinity"]
    depends_on:
      - postgres

  postgres:
    image: postgres:14-alpine

If I run test:

$ docker compose run --rm test
[+] Creating 2/2
 ✔ Network dep_default       Created                                                                                                                  0.0s
 ✔ Container dep-postgres-1  Created                                                                                                                  0.0s
[+] Running 1/1
 ✔ Container dep-postgres-1  Started                                                                                                                  0.1s
^C%

With --no-deps:

❯ docker compose run --rm --no-deps test
[+] Creating 1/0
 ✔ Network dep_default  Created                                                                                                                       0.0s
^C%

It also works when I specify --profile=test.

Could someone who is hitting this issue give a different minimal repro?

[GentikSolm]

Also getting hit with this. Command that is failing is
docker compose run -e --rm -i=false --no-deps chessclub python3 -m black --check --exclude alembic/versions .
with related snippet

version: "3"
services:
...
  chessclub:
    restart: on-failure
    build:
      context: ./api
      target: dev
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    volumes:
        ....
    env_file:
      - .env
    healthcheck:
      test: curl --fail -s http://localhost:8080/api/status || exit 1
      interval: 1s
      timeout: 10s
      retries: 30

  db:
    image: postgres
    restart: on-failure:10
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    healthcheck:
      test: pg_isready -d ${POSTGRES_DB} -U ${POSTGRES_USER}
      interval: 3s
      timeout: 5s
      retries: 10
...

[luord]

It seems like it specifically fails if the service we want to run with no deps has to be built; this repo has a consistently failing minimal project: https://github.com/luord/dc-test

The only change is from image: busybox to build: . and a Dockerfile with only FROM busybox.

On that note, a workaround that seems to work for me that doesn't require reinstalling the previous docker version is building the image before docker run, i.e.:

docker compose build foo
docker compose run --no-deps foo

[nmguse-bighealth]

Was able to update my local workstation to the latest docker compose and confirm what luord said is the same impact I am seeing. If you use an image, the failure does not seem to occur. I however still encounter this error after running docker compose build commands...

Example:
docker-compose.yml

version: "3"
services:
  test:
    profiles:
      - test
    build:
      context: .
    depends_on:
      postgres:
        condition: service_healthy
  postgres:
    image: postgres:14-alpine
    environment:
      POSTGRES_PASSWORD: test
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

Dockerfile:

FROM python:3.9

CLI output:

 $ docker compose build test
[+] Building 0.8s (5/5) FINISHED
 => [test internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                            0.0s
 => [test internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 591B                                                                                                                                                       0.0s
 => [test internal] load metadata for docker.io/library/python:3.9                                                                                                                         0.7s
 => CACHED [test 1/1] FROM docker.io/library/python:3.9@sha256:ba10a2af9d6c3bd0d20c46ecbf866dabcbad4e6a3dd7b82e2dfb1a9b6d479d87                                                            0.0s
 => [test] exporting to image                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                                    0.0s
 => => writing image sha256:ff8f23605c4296a872810899ce308ad8fb9e77b5aa2c8c9e0d8f7e9510897d90                                                                                               0.0s
 => => naming to docker.io/library/test-test                                                                                                                                        0.0s
 $ docker compose -p unique_name run --rm --no-deps test black --line-length 150 --check app/ tests/
[+] Building 0.0s (0/0)
service postgres is required by test but is disabled. Can be enabled by profiles []

[accetto]

I've also got this problem after updating to Docker Desktop for Linux v4.21.1. It also includes Docker Compose v2.19.1 .

However, I'm getting the error message only with compose files that use 2 levels of extending. In my particular case I also use 2 separate YAML files.

For example, in this case I would get the error message shown bellow:

## in file 'common.yml'
services:
    service_A:
        profiles:
            - dummy
        ....
    service_B:
        profiles:
            - dummy
        extends: service_A
        depends_on: service_A
        ....

## in file 'concrete.yml'
services:
    service_C:
        profiles:
            - concrete
        extends: service_B
        depends_on: service_B
        ....

The error message in this case would be:

service service_A is required by service_C but is disabled. Can be enabled by profiles [dummy]

However, the following service would be built just fine:

## in file 'another.yml'
services:
    service_D:
    profiles:
        - another
    extends: service_A
    depends_on: service_A
    ....

Hopefully it helps by troubleshooting.

[sentientmachin3]

I'm having this issue on version 2.19.1. Here's my docker-compose.yaml (env vars and volumes are trimmed for simplicity):

version: "3.8"
name: "name"

services:
  pgadmin:
    container_name: pgadmin
    image: dpage/pgadmin4
    restart: always
    environment: ...
    ports:
      - 8080:80
    volumes: ...

  db:
    image: postgres:alpine
    ports:
      - 5432:5432
    environment: ...
    container_name: postgres
    volumes: ...

  api:
    build: ./api
    image: api_dev
    ports:
      - 3080:3080
    environment: ...
    container_name: api
    volumes: ...
    links:
      - db:postgres

Basically when I'm restarting the api service, I get the error message reported in the issue.

[mariaa144]

Yeah. This seems to be a bad bug. I couldn't restart my nextcloud instance so the app container could connect to db! After downgrading to 2.18.1 I no longer had the issue.

I was very confused for a few hours.

[AlexanderPershin]

I've encountered the same bug today after updating docker:

$ docker compose -f ./docker-compose-local.yml restart oauth
service verify is required by nginx but is disabled. Can be enabled by profiles []

$ docker compose version
Docker Compose version v2.19.1

Service can be restarted by vs code extension though

---

update
It seems that vs code uses docker restart <container_name> which I used istead of docker compose ... restart and it worked

[sentientmachin3]

I think the issue is still present on version 2.20.2.

[bdashrad]

I'm hitting this issue on 2.20.2 as well

[bdashrad]

 $ docker compose build
[+] Building 0.0s (0/0)
service migrations is required by member-server but is disabled. Can be enabled by profiles [ci]

A little more background. It appears sometime between 1.18.2 and now docker compose began following the profiles rule outlined here: https://docs.docker.com/compose/profiles/#auto-enabling-profiles-and-dependency-resolution Previously, a service with no profile would trigger a build on a dependent service even if that one had a profile.

Example file that causes an issue.

---
version: "3.8"
name: server

services:
  server:
    image: gcr.io/my-org-12345/server:latest
    build:
      context: .
    depends_on:
      - migrations

  test:
    image: gcr.io/my-org-12345/server:test
    build:
      context: .
      target: test
    depends_on:
      - migrations
    profiles:
      - ci

  migrations:
    image: gcr.io/my-org-12345/server:migrations
    build:
      context: .
      target: migrations
    command:
      - "./migration"
    depends_on:
      - postgres
    profiles:
      - ci

  postgres:
    image: postgres:12
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:

Removing the ci profile from the migrations service fixes the issue for me.

[sentientmachin3]

Uhm @bdashrad I don't think this is the case, in my case the list of profiles is always empty. Furthermore, I noticed this only happens when running docker-compose restart, all other commands seem to be working as expected.

[chrisfrancis27]

I'm hitting this issue in Docker Compose version v2.19.1 (macOS ventura).
Simple repro: https://github.com/docker/awesome-compose/tree/master/official-documentation-samples/rails/

[ahpooch]

I'm encountered the same bug and then upgrade my Docker Desktop.
My context is just this simple demo from awesome-compose repo. It contains depends_on: instruction and attempt to run docker compose run --no-deps was generating aforementioned warning preventing next steps.
It seems the issue is resolved in Docker Compose version v2.20.2-desktop.1

[c0deright]

Still not fixed for me, see #10892

[razaashai95]

Encountering the same issue, updating to docker-compose v2.20.2-desktop.1 didn't fix anything.

[phatt-millips]

Seeing this on 2.23.0 as well

[alexburner]

Still seeing this too, on latest Docker Desktop:

Version: 4.25.0 (126437)
Engine: 24.0.6
Compose: v2.23.0-desktop.1

[alexburner]

Ah I see, the behavior I'm seeing is the same as mentioned above:

A little more background. It appears sometime between 1.18.2 and now docker compose began following the profiles rule outlined here: https://docs.docker.com/compose/profiles/#auto-enabling-profiles-and-dependency-resolution Previously, a service with no profile would trigger a build on a dependent service even if that one had a profile.

Also discussed in this article: https://nickjanetakis.com/blog/optional-depends-on-with-docker-compose-v2-20-2

And confirmed by this comment over here:

The main reason here is that we fixed the behaviour of depends_on with profiles according to the specification. You were using a bug as a feature (and this our fault not yours 😄 ), this has been discussed #9795

Basically we've got a setup like:

services:
  db:
    image: postgres

  redis:
    image: redis

  web:
    image: web
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy

  backend1:
    image: backend1
    depends_on:
      web:
        condition: service_healthy
    profiles:
      - full_stack

  backend2:
    image: backend2
    depends_on:
      web:
        condition: service_healthy
    profiles:
      - full_stack

  celery:
    image: celery
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
      web:
        condition: service_healthy
      backend1:
        condition: service_healthy
      backend2:
        condition: service_healthy

We use a full_stack profile to determine whether we're just spinning up the web frontend/middleware, or also the backend services.

And we want celery to always start last, whether or not the full_stack profile is in use.

But in docker compose v2.19+ we get the error:

service backend1 is required by celery but is disabled. Can be enabled by profiles [full_stack]

And it seems like profiles is now unable to handle this? The article and this comment mention adding required: false:

 celery:
    image: celery
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
      web:
        condition: service_healthy
      backend1:
        condition: service_healthy
        required: false
      backend2:
        condition: service_healthy
        required: false

But that caused celery to be started twice, once before the backends and then "recreated" after — which led us to the same "celery task resume on startup failing with connection error" we were trying to avoid.

Is there a plan to support profile + depends_on in the future? Or do we need to use compose extend/merge instead?

[ndeloof]

@alexburner such a "select by profile" pattern is not supported, profiles are designed to add optional services to a compose model, not to switch components. An approach I have in mind to address this need relies on compose-spec/compose-spec#404

[Vigilans]

such a "select by profile" pattern is not supported, profiles are designed to add optional services to a compose model, not to switch components.

@ndeloof You say profiles are designed to add optional services, but googling for selectively start components in docker compose, people always suggest you use profiles because docker compose just doesn't have any alternative for this.

Now compose team fixed a "bug" that many people used as "feature" because they don't have another alternative, these people just felt left out because not a single solution is available anymore after version 2.19.0 (see #9795 (comment), #10993), especially for me requiring include keyword which is supported since version 2.20.3.

---

Ok, now we stick to add optional services and forget about switch components things. Say we have service a in profile a and service b in profile b. They are all optional services, but service b depends on service a to work.

Now, #9795 says docker compose --profile b up should not start a because a is not enabled by profile, we should inhibit this implicit setup. Ok this argument is indeed valid, but shouldn't it be a proper way to explicitly specify this requirement in compose configuration file, thereby making auto enabling profile a a defined behavior?

Suggested proposal like this:

services:
  a:
    profiles: [a]
  b:
    profiles: [b]
    depends_on:
      a:
        profiles: [a]
        condition: service_started

By making depends_on.profiles, the a dependency explicitly states that "profile a should be activated when starting service b".

[ndeloof]

"switch components" is a perfectly valid usage. The simplest approach for now is to rely on multiple compose files. We don't have a better way yet, maybe some will come.

About your proposal, same can be achieved already by declaring service a with profiles: [a,b]

[Vigilans]

Since I'm inclined to make a proposal of depends_on.profiles at compose-spec, let me elaborate on the use case more formally.

Example use case

Say we have following directory layout:

homelab/
├── .env
├── compose.yml
├── gitlab/
│   └── compose.yml
├── minio/
│   └── compose.yml
└── sharelatex/
    └── compose.yml

Where root compose.yml is:

networks:
  ...

secrets:
  ...

include:
  - minio/compose.yml
  - gitlab/compose.yml
  - sharelatex/compose.yml

And root .env contains:

COMPOSE_PROFILES=* # all

Here:

• We call each of homelab's subfolder a component, and all its services in sub compose.yml will have the same profile, .e.g. sharelatex profile in sharelatex/compose.yml services.
• minio is a component for S3 storage, therefore all other services using S3 as backend should include it as dependency, e.g. gitlab and sharelatex.

Now, we can select one of the components through:

$ docker compose --profile sharelatex config

And manage all services by not providing profile (because COMPOSE_PROFILES set to *):

$ docker compose config

Proposal about depends_on.profiles and component switch

@ndeloof You said We don't have a better way yet, maybe some will come., so let me be the one to come with the way for it.

Proposal is: use profile + depends_on.profiles [+ include].

Before explaining what depends_on.profiles will do in detail, let's see the drawbacks of current ways you mentioned.

1. Multiple compose files

The simplest approach for now is to rely on multiple compose files.

By using multiple compose files, one selects components with:

$ docker compose -f compose.yml -f sharelatex/compose.yml config

But sharelatex depends on minio component. Therefore you have to include one more file in command:

$ docker compose -f compose.yml -f minio/compose.yml -f sharelatex/compose.yml config

This:

1. Breaks the convention that user should not care about what the dependencies are once written in config.
2. Even if user takes the trouble to manage it, how can user know how many dependencies a component has once the dependency chain gets deeper? It also cannot be easily discovered through external script.

It also doesn't have a good way to manage all services without using external script.

2. Profiles + write child component's profile to parent dependency's profiles

About your proposal, same can be achieved already by declaring service a with profiles: [a,b]

By using profiles in this way, one selects components with:

$ docker compose --profile sharelatex config

Since minio component also gets written with sharelatex profile, above command need not change:

minio:
  profiles: [minio, sharelatex]

Seems good, but still with notable drawbacks:

1. Breaks the convention that children dependencies should be transparent to parent.

   Why should minio component's yml should know it has a sharelatex children? When there are more services that relies on S3 storage, they all need be written to minio's compose file.

   What's more, minio is a component with mutiple services. Say we have m minio services with n children components, then the profiles have to be written m*n times to minio's compose file. This is really a burden to maintainer.

2. It is not optimal to select parent services into management.

   When selecting sharelatex profile without minio services, only sharelatex services will be selected for management:

   $ docker compose --profile sharelatex restart
   [+] Restarting 5/5
   ✔ Container sharelatex-mongo    Started
   ✔ Container sharelatex          Started
   ✔ Container sharelatex-texlive  Started
   ✔ Container sharelatex-mount    Started
   ✔ Container sharelatex-redis    Started
   

   However, when specifying minio services with sharelatex profiles, restart of compose will also restart minio services:

   docker compose --profile sharelatex restart                                                                                                                           
   [+] Restarting 7/7
   ✔ Container sharelatex-mongo    Started
   ✔ Container sharelatex-mount    Started
   ✔ Container sharelatex          Started
   ✔ Container sharelatex-texlive  Started
   ✔ Container minio-mount         Started
   ✔ Container minio               Started
   ✔ Container sharelatex-redis    Started
   

   This is not optimal and even unacceptable, since minio is an important base service for storage, how can a restart of sharelatex component also restarts minio component that breaks all other components relying on it?

3. Proposed solution for component switching: Profiles + depends_on.profiles

With above use cases, let's see how is depends_on.profiles used and what it will do:

services:
  sharelatex:
    depends_on:
      minio:
        profiles: [minio]
        condition: service_healthy

By explicitly specifying minio profile in minio service dependency, starting of sharelatex service will start minio service (and other services minio relies on in minio profile) and wait until it's healthy, but do not select minio services into management.

That is, docker compose --profile sharelatex restart command will 1) only restart sharelatex services, and 2) start minio service if not up, and wait for it to be healthy.

For managing all services, use root .env with:

COMPOSE_PROFILES=* # all

So docker compose up command will just bring up all services the old-school way.

It perfectly solves all drawbacks above, and presents a very good solution for component switching because it only uses simple docker compose command as entrypoint, without resorting to external script.

[ndeloof]

COMPOSE_PROFILES=*

sounds to me like a terrible pattern!

Breaks the convention that user should not care about what the dependencies are once written in config.

this is why include has been introduced, so you can rely on third-party components without having to know about implementation details

You can then use:

include:
    - ${BACKEND:-sharelatex}/compose.yml

.. which will load sharelatex implementation by default, but you can select another one by just setting BACKEND variable so this point to another compose file, as long as same service name is declared

[Vigilans]

COMPOSE_PROFILES=*

sounds to me like a terrible pattern!

* can be replaced by explicitly specifying all (and exclude some optional) components, like:

COMPOSE_PROFILES=minio,gitlab,sharelatex

It doesn't matter, because we can still use docker compose up so manage all default services the old-school way.

In fact, since these components are already explicitly declared in root compose.yml:

networks:
  ...

secrets:
  ...

include:
  - minio/compose.yml
  - gitlab/compose.yml
  - sharelatex/compose.yml

Explicitly write default profiles in .env file is also a better choice.

Command line provided COMPOSE_PROFILES will also override the value in .env file:

COMPOSE_PROFILES=sharelatex docker compose config

So .env provided COMPOSE_PROFILES will not interfere existing behaviors of COMPOSE_PROFILES=... docker compose and docker compose --profile ....

---

You can then use:

include:
    - ${BACKEND:-sharelatex}/compose.yml

.. which will load sharelatex implementation by default, but you can select another one by just setting BACKEND variable so this point to another compose file, as long as same service name is declared

This still has the problem that if sharelatex depends on minio component, we should also include minio/compose.yml:

include:
    - minio/compose.yml
    - ${BACKEND:-sharelatex}/compose.yml

But this selects minio component as default. What to do when BACKEND is something that does not rely on minio component? It comes back to the problem to manually figure out dependencies of specific BACKEND component.

If we include minio/compose.yml in sharelatex/compose.yml:

include:
    - ../minio/compose.yml

This:

1. Using .. in child component compose.yml seems not a good pattern.
2. Comes back to the problem 2.2 that selecting minio services for management may not be optimal, like restarting sharelatex component will also restart minio component that breaks other services that depends on minio.

[ndeloof]

This still has the problem that if sharelatex depends on minio component, we should also include minio/compose.yml

no. If sharelatex includes minio (as it depends on this component) you don't have to. includes is transitive

[Vigilans]

This still has the problem that if sharelatex depends on minio component, we should also include minio/compose.yml

no. If sharelatex includes minio (as it depends on this component) you don't have to. includes is transitive

I've mentioned this in second example (include - ../minio/compose.yml), and it still has the problem that selecting minio services for management may not be optimal, like restarting sharelatex component will also restart minio component that breaks other services (e.g. gitlab) that depends on minio.

---

I've opened a PR at compose-spec/compose-spec#488 to introduce depends_on.profiles attribute to fully solve this issue. @nickjj @alexburner @ShahimEssaid May you share some opinions on it?