Interpolate environment variables in docker-compose.yml

[aanand]

(I'm creating a fresh issue for this as the old one has accumulated rather a lot of baggage.)

It should be possible to pass in environment variables to the value of any* configuration entry in docker-compose.yml. A lot of people want to do it, it's good for portability and I'm satisfied it's not going to cause chaos.

I have some reckons.

Required variables and optional defaults

It's useful to be able to specify that a variable that must be present in the environment, i.e. that Compose will refuse to run if it isn't. However, this will be a pain when you've got lots of them, so it should either be something you explicitly enable, or it should be possible to specify a default value.

The MVP implementation does not need to have either feature, but there should be a clear path to implementing both in a backwards-compatible way.

Syntax

There's a strong case for implementing an established standard, as long as it's not heavyweight - our requirements for functionality are minimal.

• POSIX parameter expansion is OK. It has a few too many features, but we could implement a subset of them:
• ${VARIABLE} - outputs empty string if VARIABLE is unset
• ${VARIABLE-default} - outputs default if VARIABLE is unset
• ${VARIABLE?} - errors out if VARIABLE is unset

#845 implemented a Bash-style ${VARIABLE:default} syntax, which is similar to POSIX parameter expansion but slightly different.

• Python's format syntax would be trivial to implement, but is tied to the Python language, and has rather too many features.

Implementation

Python's os.path.expandvars function implements the most basic case of POSIX parameter expansion:

>>> from os.path import expandvars
>>> expandvars('${HOME}')
'/Users/aanand'

However, it's got at least 2 problems:

1. An unset variable doesn't expand to an empty string - instead, it results in no expansion:

>>> expandvars('${UNSET}')
'${UNSET}'

2. Malformed syntax doesn't error out - instead, it also results in no expansion:

>>> expandvars('${HOME')
'${HOME'

So far, #845 is the closest we've got, but I'm fundamentally wary of an implementation that relies on regular expressions. Templating is a non-trivial job, and people are going to put all kinds of broken stuff in, so we need something that's robust, strict and errors out with helpful messages. Two important requirements:

• If someone puts in something malformed, Compose will not run.
• It is possible to escape any of the special characters used in the template syntax.

There may well be good Python implementations of Bash-like variable interpolation out there already - if not, creating something standalone would be far preferable to bloating the Compose codebase.

*Actually, are there any configuration keys for which we shouldn't allow interpolation?

[kojiromike]

How far do you want to go with these established UNIX standards? (FWIW, it's not a defacto standard, it's an actual standard.)

As someone who occasionally accidentally tries to use POSIX parameter expansions in Dockerfiles, if they were at all supported in docker-compose.yml it would make me a happy camper.

[aanand]

@kojiromike Hmm, so POSIX parameter expansion is actually what I was going for, but reading over the docs it looks like I misremembered the syntax/semantics.

Edit: I've updated my thoughts on syntax in the description.

[Vad1mo]

I have been following the old thread and we urgently wanted to have this feature. Finally the pain was too big and we created a yaml preprocessor bahs script to substitute variables in POSIX style. it worked fine but eventually we stopped using it, because it had one issue. You have to run the preprocessor first and set all the parameters before you get the final solution. Now we are using the docker yaml extends feature. Because it allows us to checkin the actual configuration and just execute it on the target. We know better what is going to happen.

Even though I was a supporter of docker-compose passing variables, I am now not so sure.

As an ideal solution I would rather see docker extends done right. In a sense that would be a solution that fits both. So what is broken in docker extends? It is basically the fact that you have to write all entries in the inherited file. It is not a merge where you enter only what you want to override.

Look at out actual example and how verbose it is. There are only two lines that matter.

#Common 
elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"

# inherited
elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

So my recommendation fix docker extends an make it less verbose. You don't even have to write that much code as YAML provides all the functionality you need. If you stick with standard YAML the file could be analyzed or created by other tools and UIs.

Take a look at YAML "node anchors" and YAML "file merge" it might be the perfect solution.

FYI: this discussion continuous now on #1380

[aanand]

@Vad1mo I agree that extends falls short in your case. There are a lot of things we can do to improve that experience - could you open a separate issue for it, so we don't get sidetracked here?

[Vad1mo]

Of course! I just wanted to highlight that this could be an easy and elegant alternative.
If compose extends gets you half the way too variable passing, then an improved compose-extends will make variable passing obsolete. Having less concepts to understand make easier for the user.

[mattes]

My use case is to allow $PWD in volumes, so every developer in the team can clone a repo to wherever and paths still get mounted correctly.

elasticsearch:
  image: zinvoice/elasticsearch
  volumes:
    - $PWD:/app

[dnephin]

@mattes I believe that is already supported, I think .:/app is supported as well

[kojiromike]

@aanand As a PoC I did a dirty hackup of POSIX PE in Python. For the Saturdays.

[aanand]

@kojiromike Looks great. Let me know if you plan to continue on it.

[kojiromike]

@aanand I intend to, but it definitely has a few bugs right now (and I think it may have been a bad idea to use shlex). Bug reports and PRs are welcome, of course.

[nafg]

@dnephin how about $HOME / ~?

[dnephin]

@nafg Both of those are supported for the host path of a volume

[nafg]

@dnephin interesting, b/c somehow I ended up with a directory named '$HOME'...

[andrerom]

@aanand Like the "${VARIABLE:default}" proposal, with global_extends (or "import") this would become rather powerful.

Q: Would this allow to specify port number that is exposed to host? like - "${WEB_PORT:80}:80"?
Use case is to be able to easily spin up several instances of an app on same machine/cluster, typically listening to different ports or assigned to different local domain names.

[aanand]

Yes, you'd be able to do that.

[alexanderilyin]

I'd like to use vars in volumes together with docker-compose scale my_app=3. I have this docker-compose.yml

server:
  image: alexanderilyin/docker-teamcity-server
  ports:
   - "8111:8111"
  volumes:
    - .TeamCity:/root/.BuildServer
  links:
   - mysql
mysql:
  image: alexanderilyin/docker-mysql
  volumes:
    - .MySQL:/var/lib/mysql
  environment:
    MYSQL_DATABASE: teamcity
    MYSQL_USER: teamcity
    MYSQL_PASSWORD: teamcity
    MYSQL_ALLOW_EMPTY_PASSWORD: yes
agent:
  image: alexanderilyin/docker-teamcity-agent
  links:
   - server

And I want to be able use scale for agents and use dynamical volumes for them to keep data between launches, for example:

agent:
  image: alexanderilyin/docker-teamcity-agent
  volumes:
    - .agent_{$AGENT_INSTANCE_ID}:/opt/buildAgent
  links:
   - server

[Maxim-Filimonov]

I hope it would be possible to interpolate variables as part of image name too
We are using https://github.com/openshift/source-to-image to build local container on CI for every branch and then run tests on it using docker-compose.
Running tests with dynamic image is quite complicated with docker-compose and requires manual template rendering.. 👎

[andrerom]

But you can set COMPOSE_PROJECT_NAME to control prefix per run to be able to do that already right? If so, no need to have complex logic and unreadable yml files around names.

[Maxim-Filimonov]

@andrerom don't follow. According to docs that controls the following Sets the project name, which is prepended to the name of every container started by Compose while we are trying to set an image property instead:

web:
  image: <I_AM_DYNAMIC>

[andrerom]

ah, my mistake.

Thought you meant

<I_AM_DYNAMIC>:
  image: nginx

Dynamic image (and build) reference would indeed make a lot of sense. For instance switching between debug and non debug base containers for your programming language for instance would be a good use case for this.

[andrerom]

Additional use case (which might be what @Maxim-Filimonov has in mind): Being able to override which tag to use of an image, so you can use :latest by default, but change to easily test something else without changing yml file (needed for CI use cases basically).

[Maxim-Filimonov]

@andrerom that is exactly our use case 👍

[ghost]

Will this also work for things like??

web:
  environment:
    - FOO=${whoami}

[thaJeztah]

@k0377 I don't think they will, because that's really something that's handled by the shell, but you could add the result in an environment variable and use that.

In this particular case, the $USER environment variable will probably give you the same.

[askoretskiy]

@aanand Why not use any of existing template engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task (and regexps are not that cool) so that we should use already existing one, that proven to be solid.

[askoretskiy]

Alternatively we might use YAML ancors and references https://gist.github.com/bowsersenior/979804

But then we are limited on variables usage (inject variable name into middle of content).

[marcellodesales]

The default value should be there... Very useful... Developers and OPS are using either the docker logs or syslog... So, They usually have to create the default LOG_FORMAT shown below... We could just have the default value and only use it when switching to syslog...

default:
  extends:
    file: base.yml
    service: base-${LOG_FORMAT:docker}
  labels:
    - "net.company.npmjs.datacenter=${DATA_CENTER}"
    - "net.company.npmjs.env=${ENV}"
    - "net.company.npmjs.hostname=${HOSTNAME}"
    - "net.company.npmjs.role=${NPMO_ROLE}"
    - "net.company.npmjs.log=${LOG_FORMAT}"

base-syslog:
  log_driver: syslog
  log_opt:
    tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"

base-docker:
  log_driver: json-file
  log_opt:
    max-size: "128m"
    max-file: "4"

When will this be available? I'm on Compose 1.7.0 and this is still not there :(

[nottrobin]

Please please please give us default values!

[pataquets]

@marcellodesales: Perhaps you can take advantage of using a docker-compose.override.yml file in some way. Check that feature.

Also +1 on env vars. It is our major pain point with docker-compose nowadays.

[metal3d]

I would insist on my PR #3367 to be able to get certain values from host. :)

[marcellodesales]

@pataquets I don' think I want to create yet other override file... our base.yml file, as shown above, shows all the supported stuff in terms of logger driver, etc... I just want to switch and have default values. I would think that we would need to maintain yet more yml files. But I will keep that in mind just in case.

[zerthimon]

+1

[trippel]

+1

[gbmeuk]

+1

[panga]

+1

[zlberto]

Any notice about use environment variables in docker-compose?

+1

[ChrisCooney]

+1

[darkn3rd]

FYI: Environment vars for docker-compose works as of 1.7.0. You can also set docker-compose default variables in .env in the same directory as your root docker-compose.yml file. This is not to be confused with docker engine envfiles, as that's a different thing.

[LouWii]

Is there a way to set the service name as a variable ?

Instead of writing this

services:
   site_db:
     image: mysql:5.7

We could write

services:
   ${CONTAINER_NAME}:
     image: mysql:5.7

My goal is to keep the same docker-compose.yml across multiple sites, and only change the .env file. Right now, I still need to modify the container name because I'm running several apps on the same host. And I'd like to have each service to have its own name for clarity.

[thaJeztah]

@LouWii you can use

services:
    site_db:
      container_name: "${CONTAINER_NAME}"
      image: mysql:5.7

or (compose-file format 2.1 and up)

services:
    site_db:
      container_name: "${CONTAINER_NAME:-defaultname}"
      image: mysql:5.7

But why not set the project-name? The project name is intended for that, as it prefixes/namespaces the container-names that are created to prevent conflicting with other projects on the same host. See https://docs.docker.com/compose/reference/envvars/#/composeprojectname

[LouWii]

@thaJeztah Thanks ! I'm still learning how Docker and docker compose work. Setting the project name seems to be a better option, it makes total sense in my use.

[samhh]

Is there any way to script within the interpolated braces? For example ${HOST_PORT + 1}.

[nafg]

You could pipe the file through Jinja or something...

…

On Tue, Jan 24, 2017, 5:36 AM Sam A. Horvath-Hunt ***@***.***> wrote: Is there any way to script within the interpolated braces? For example ${HOST_PORT + 1}. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1377 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGAUN5ZrU39dnVVVASwIHr5mGqJFxh3ks5rVdRIgaJpZM4EMysO> .

[logicminds]

Am I able to escape $?

environment:
   PATH: "$PATH:/home/appuser/.bundler/bin"

Currently this results in the host's PATH variable being interpolated and not the container

[alaagalal]

is a docker-compose.yml file only one found ?
how can i add or modify it?
Thanks in advance

[samrocketman]

@logicminds though I couldn't find it documented anywhere, I found $$ interpolates to an escaped $.

environment:
   PATH: "$$PATH:/home/appuser/.bundler/bin"

[binarymist]

@elquimista has a solution that I've found useful mhart/alpine-node#48 (comment)