Environment variables priority

[ulyssessouza]

What I did
This PR is meant to define the precedence of the environment variables evaluation.

For the environment variables to be available in the container:

1. Command Line (docker compose run --env <KEY[=VAL]> https://docs.docker.com/engine/reference/commandline/compose_run/#options)
2. Compose File (service::environment section: https://docs.docker.com/compose/compose-file/#environment)
3. Compose File (service::env_file section file: https://docs.docker.com/compose/compose-file/#env_file)
4. Container Image ENV directive (https://docs.docker.com/engine/reference/builder/#env)

---

For the environment variables available for docker compose's runtime:

1. Command Line parent command option docker compose --env-file <FILE> (or .env in the root of the project by default)
2. OS Environment Variables

Obs: Note that the variables available for runtime will not be available in the container unless it's imported by one of the methods on the list above

Related issue
Resolves #9521
Resolves #9638
Resolves #9608
Resolves #9578
Resolves #9468
Resolves #9683

Depends on compose-spec/compose-go#280

Documentation update PR

(not mandatory) A picture of a cute animal, if possible in relation with what you did

[ulyssessouza]

Note that this points to compose-spec/compose-go#280
This PR shouldn't be merged before this dependency is merged and updated here.

[milas]

Wahoo! Looks good for approval once the compose-go side of the house is merged + tagged 🥳

(Sorry for nits - as you're well aware, the env code is pretty complex, so I was trying to identify places I got confused and had to dig in more)

[nicksieger]

Overall this looks good, just have some questions about the tests and what the new priority order is.

[nicksieger]

So there's a difference between -e WHEREAMI with the var defined, and -e WHEREAMI=shell? In this case shouldn't 3. be the expected result?

[ulyssessouza]

In the first, the variable value will be inherited from the environment.
The second one will have the explicit value passed (shell in this case).

[nicksieger]

Update to 3. Environment file <-- Result expected above?

[nicksieger]

Also, these tests so far seem to indicate that 1. Compose file is not the highest priority. Am I reading this right?

[ulyssessouza]

I just refactored the comments to fix this and reflect the order of this change.
BTW, the new doc on priority is in the oven docker/docs#15161

[glours]

LGTM

[K0Te]

Hi,

Sorry if this is not the correct place to ask a question, I've checked this PR source code and documentation update and am still not sure what the expected behavior is.
I've used .env file to override COMPOSE_FILE for local development: COMPOSE_FILE=docker/docker-compose.yml:docker/docker-compose.override.yml. This allowed to have all docker stuff in a separate folder and provide some secrets in docker-compose.override.yml.
And on CI I've just used COMPOSE_FILE=docker/docker-compose.yml as an env variable in GH Actions.

This worked fine as of docker compose 2.7.0, but with docker compose 2.9.0 this workflow doesn't work anymore. Value from .env started to take precedence over the environment variable.

Is this a new expected behaviour or a bug?