Unable to override environment variables if they are present in .env files

[yanneg]

Description

v2.9.0 does not allow environment variables to be overridden if they are already present in .env.

Steps to reproduce the issue:

1. With the following files:

docker-compose.yml:

version: '3.9'

services:
  app:
    image: app:${APP_CONTAINER_TAG}

  proxy:
    image: proxy:${PROXY_CONTAINER_TAG}

And .env

APP_CONTAINER_TAG=bar
PROXY_CONTAINER_TAG=bar

2. When I run APP_CONTAINER_TAG=foo PROXY_CONTAINER_TAG=foo ./docker-compose-2.9.0 convert, I receive the following

name: test
services:
  app:
    image: app:bar
    networks:
      default: null
  proxy:
    image: proxy:bar
    networks:
      default: null
networks:
  default:
    name: test_default

In versions prior, like 2.7.0, I would get the expected behavior:

name: test
services:
  app:
    image: app:foo
    networks:
      default: null
  proxy:
    image: proxy:foo
    networks:
      default: null
networks:
  default:
    name: test_default

Output of docker compose version:

Docker Compose version v2.9.0

Additional environment details:

I noticed this on my GitHub Actions environments. The runner recently updated Docker Compose to v2.9.0.

[adrolter]

This has been broken as of v2.8.0, presumably caused by this commit

[ulyssessouza]

This is the new expected behaviour. Environment file now has precedence over the OS Environment.
To bypass the value, in the .env file, please use:

APP_CONTAINER_TAG=${APP_CONTAINER_TAG:-bar}
PROXY_CONTAINER_TAG=${PROXY_CONTAINER_TAG:-bar}

The :- is the POSIX notation for "if empty or unset, use this default value".

[yanneg]

Thanks for the clarification @ulyssessouza . Just wondering, but why the sudden change in behaviour? IME, it's counter-intuitive to how most other tools and libraries handle env vars.

[madhavajay]

First you broke the ability to pass in ENVs to compose as a regression #9495, so we started using --env-file, now this is broken.... zzzz Seriously. 🙄

[madhavajay]

Just to clarify, from what I can see docker-desktop does not yet include anything above 2.7.0 does that mean that compose 2.8.0 was not shipped with docker desktop and these issues were reverted in 2.9.0 ?

[adrolter]

@madhavajay No, I've built both v2.8.0 and v2.9.0 from source and both have the same behavior. It has not been reverted and judging from Ulysses' comment there appears to be no intention to do so.

[pvlg]

It broke my CI/CD

[otavio]

Being a breaking change it should have been done in a major (e.g 3.0.0) release.

[gustavosbarreto]

Thanks for the clarification @ulyssessouza . Just wondering, but why the sudden change in behaviour? IME, it's counter-intuitive to how most other tools and libraries handle env vars.

This was a deliberate breaking change that changes the behavior of a component which is being used in the same way for years and as you said @yanneg, it goes against expected behavior of most other tools and libraries.

[ulyssessouza]

That was me trying to give more flexibility to Docker Compose in the way that it would have the possibility of overwriting OS Env from the Environment file.
By your messages, I understand that it wasn't the best decision, given 2 main reasons:

• Env files are not an exclusivity of Docker Compose and other tools don't use it in that specific way
• Docker Compose has been used in another way for years and changes on its behaviour can be painful.

I would like to thank you all for the feedback and present my excuses for the inconvenient. It was in the best intentions of evolving the software.

I'm working on a PR and a patch release to revert that behaviour. Also a better documentation on precedence is being cooked.

[otavio]

@ulyssessouza awesome. Thank you for your prompt action on this issue.

[madhavajay]

@ulyssessouza thanks so much for listening. I think everyone here can agree, the tooling is great, we love docker, we love docker compose, so much so were building our tooling on it, which is why were here quickly googling why our stack has broken. I personally noticed this on gitpod which uses a custom compiled compose v2.8.0. It would be great if this bug isn't shipped with any versions of Docker Desktop because then it becomes something we kind of have to support in the general population for a time.

I also think that considering docker wants to accelerate adoption of new versions its extra important that regressions don't get introduced because users are being actively encouraged to upgrade so we should expect regressions to have large stop the world, impact.

Having said that, please do keep innovating. ❤️