Add support for extends feature in Compose v3 / docker stack deploy

[shin-]

As can be seen in docker/compose#4315 , the extends feature that exists in docker-compose seems to be popular among users despite its flaws. However, it has so far not been added in the Engine's implementation of the Compose format. So far, we have advised users to simply flatten their Compose file structure when using v3, but is this the long-term solution we want to go with? How can we provide a clear upgrade path for users who have come to rely on this feature?

cc @dnephin @vdemeester

[shouze]

I've added some notes docker/compose#4315 (comment) to me bringing back extends as it exists up to docker compose file 2.1 version is not a good idea but the main feature I miss is to be able to declare abstract services (that should never be runned but could be used to group common properties of services for convenience).

[jakajancar]

Agreed on comments from docker/compose#4315 that the way extends worked was a bit spartan.

I won't recommend a solution, but FWIW, to show the extent of abuse, here are the conventions that adorn the top of our compose file:

#
# Docker Compose configuration
#
# Due to lack of "expressivity" in Compose, we define our own couple of service
# "pseudo-types":
#
#   - image-only services (name: *-image)
#
#     The only goal of these is to build images. No other services build images.
#
#     These have entrypoint overridden to exit immediately.
#
#   - base services (name: *-base)
#
#     These contain common configuration and are intended to be extended.
#
#     Their command (not entrypoint, to keep the original one) is overridden to
#     exit immediately. Service must support a command to exit immediately.
#
#   - task services (name: *-task)
#
#     These are intended for running one-off commands.
#
#     Their default command is overridden to exit immediately. Service must
#     support a command to exit immediately.
#
#   - "real" services
#
#     These are actual services that stay up and running.
#
version: '2'
services:
  ...

[nicodmf]

I think the extends as in v2.1 is a good choice. Extends is actually simple and understandable, this is too a good pratice for each environnement to have a little and readable transformation between dev, prod and env.

Actually, i have

• a common docker-compose file which present the base rules
• a dev docker compose extending it and with facilities for developper
• a staging docker compose extending common too and with facilities for this environnement
• a production docker compose extending common too and with facilities for this environnement, specially container replication, rules about restarting, etc...

This works, and i don't understand why we should search in this ticket a complete rewrite, but more the keep of an interessing feature. Extends is a good part with special use cases that other techniques can't resolv easily. I'm happy with extends possibilities.

[sambernet]

Blocking us from upgrading to v3.x format as well.

We keep our docker container definitions in a folder-per-instance layout, where each folder contains a docker-compose.yml that defines the env specifics for the container instance at hand. To factor out the common stuff (DRY) we use base service definitions in a parent folder and use extend.

So when I need to manage a particular container instance, I just need to cd into the right folder and can then directly run docker-compose commands without further configuration (no -f flags required that team members need to lookup or know, just works as expected out-of-the-box).

[SC7639]

I'm blocked from using version 3 compose files.

I use services.yml to define the base layout of my services and then extend that with dev.services.yml for my development evironment (mostly adding volumes) but I like the (DRY) re usability of extends when it was added. It's not a deal breaker but it would keep me from moving to version 3 unless there is a must have feature.

[sambernet]

I'm not against improving the v2.1 'extends' solution with something more appropriate though. Something like abstract services could be both more safe to use and more powerful.

For example (since it's abstract) I could imagine supporting abstract volumes/mountpoints/networks, where the abstract base service defines a local mount point or a required network for a service, without defining the host part - meaning a deriving service could then define/map the host part of the mounts/volumes and networks to whats appropriate in his host/stage environment.

That's an example of something we can't do now with 'extends' as far as I know, and would open some interesting possibilities.

[alwaysastudent]

Taking away extends feature was not helpful at all. We have a lot of web services started with the same volumes mapped, environment variables and labels set. They also have same healthcheck policies. And Not to mention ports. Combining compose files using multiple -f option is tedious and error prone if you are dealing with multiple projects in the same host.

[alwaysastudent]

@shin- will this be brought back on 3.2 schema ?

[efrecon]

I really, really would love to get extends back into the file format. I have been using it to further refine abstract services for a number of projects. I understand that multiple -f options fits almost the bill, but it doesn't always work. For example, I rather often change the name of the abstract service into something that is more meaningful in the context of the project at hand. This is something that the overriding that occurs with multiple -f options does not support.

I would not mind loosing the exact extends though, as long as their is some other way to "instantiate" an abstract service in a file.

[jazzfog]

I have a setup with a common service file and bunch of services extending it, changing mostly volumes, so I would say I rely on extends feature and do not see other good way to describe my setup.

--frustrated

[rationull]

+1
I can see the logic behind the recommendation to flatten docker-compose files but I don't like the idea of duplicating the code that defines our development service topology across multiple projects. We'll use the -f workaround for now, with shell scripts so that developers don't have to remember which services to include in which cases.

I'm hoping a satisfactory solution can be found here to enable better factoring of compose structures!

[MadBomber]

At the top of my use case list is to DRYup my configuration managed hoard of services. I thought I'd be able to take advantage of 'extends' if v3. I don't want to go back to v2... and I don't want to have special cases where I must use the -f process work-around.

[cirocosta]

Hey, did anyone start working on this? I can't find a PR to keep track. It'd simplify a lot some things for us here as well (use-case very similar to some described above).

[JanNash]

Since version 3 is frozen and I needed a way to share common configuration, I hacked a small workaround, which I think I could share here (I'm not sure if this is the right place but feel free to tell me where else I can share this info :))

I'm not going to elaborate on it here, since there's a readme in the repo.
Have a nice one, everyone ☮️

Edit:

Sorry, here's the link :D

[gustavosbarreto]

+1

[fabiohbarbosa]

+1

[ccamp46]

+1

[JanNash]

Thanks for the +1s 💃
In the meantime, I've found another docker noop image, which smaller by a factor of 10^3 (due to the actual noop being written in assembly).

Unfortunately, there's no License in that repo. I already wrote a message to the owner on facebk but he didn't answer yet. Maybe he'll add a license if more people query him about it :)

[grayside]

Something that might help some of the extends use cases (those within a single file) would be support for YAML anchors: https://learnxinyminutes.com/docs/yaml/

It appears that the JSON Schema may be failing validation on them service must be a mapping, not a NoneType..

[JanNash]

Hey, @grayside, yaml anchors do work, at least for me. See my comment above for how I use them.

[shouze]

Ok but it's too sad to use some noop service no?

Especially for env vars, what kind of values those env vars handle? If it's about secrets, use swarm secrets feature (or any other secrets solution). If it's about settings, we're ok that most of the time settings are app/service specific, not intended to be shared between services.

If you need to share settings between services, most of the time it's when you launch the same container image but for different runtime purposes/tasks (consumer, producer, http worker, ...).

[JanNash]

If it's about settings, we're ok that most of the time settings are app/service specific, not intended to be shared between services.

I tend to disagree. In the project I'm currently working on, I use it for example for volumes:

# Volume paths
environment:
  - &volume_a        volume-a:/usr/share/my_project/volumes/volume-a
  - &volume_b        volume-b:/usr/share/my_project/volumes/volume-b
  - &volume_c        volume-c:/usr/share/my_project/volumes/volume-c
  - &volume_d        volume-d:/usr/share/my_project/volumes/volume-d

Now I can specify these volumes like that:

volumes:
  - volume-a:
  - volume-b:
  - volume-c:
  - volume-d:

services:
  some-service:
    image: some-image
    volumes:
      - *volume_a
      - *volume_b
  
  some-other-service:
    image: some-other-image
    volumes:
      - *volume_b
      - *volume_c

  some-third-service:
    image: yet-another-image
    volumes:
      - *volume_a
      - *volume_b
      - *volume_c
      - *volume_d

This makes it way easier to navigate different volumes without having to think about which container you're in. Imho, this is one way to make your docker-compose setup more consistent and easier to use and maintain.

[shouze]

Ok yes I understand @JanNash but in your example below you don't have any noop service right?

[yajo]

But anchors are not enough for many cases.

My case involves supporting several environments for the same project. You can see a scaffolding for our projects here.

When you develop, you use devel.yaml, but in production you use prod.yaml. There's also test.yaml. All of them inherit from common.yaml and get some common variables from an .env file.

Each one has its own peculiarities:

• devel environment aims for development speed, so it uses build args that produce faster builds, mounts volumes with the app code from the developer computer, and adds some dummy services that isolate the app from the outside world.
• prod aims for stability instead, so it downloads and compiles all code, and bundles it in the image itself instead of using volumes. Builds are slower, but more secure.
• test aims to be exactly like prod, but with isolation from external services, to avoid polluting the external world.

This simple separation allows to have a very agile and flexible DevOps pipeline, where everybody uses the same code among different stages, with just little tweaks depending on the environment in use.

I tried to move to compose file format v3, but not only extends is unsupported, but also .env, so right now it would be a maintenance nightmare (more because of the lack of .env, to be honest). When deciding between Swarm and DRY, we chose DRY for now, but some day we'll need Swarm and I hope that day both features are supported again... ☺️

... or at least we have a way to generate a valid DRY-less format from a DRY-ful solution. I thought docker-compose bundle was for that, but seems to be doomed to deprecation right now...

... or we have a different tool that makes everything (I'm keeping an eye on ansible-container too). But surely this is not "the fix".

[grayside]

The link in #31101 (comment) includes a README with the working example of YAML anchors. Looking it over and trying it again today, it works fine. Not sure what I'm doing differently.

@JanNash 👍

[JanNash]

@yajo I hear you and as said, it's a workaround and it would be better by an order of magnitude if there was a good, built-in, DRY solution supplied by docker/moby/docker-compose (whatever is the correct reference). Let's all hope that will come soon because apart from that, I'm pretty happy with docker compose 👍

For the sake of missing .env support, I also hacked a workaround (my project's not in production yet, so my talk is a bit cheap, I know :)). To support different sets of env vars (dependency and image versions/tags, for example) in different environments (for me right now, that is local-development and a small dev-server), I use two files, local.env and development.env and instead of running my commands by just docker-compose <command>, I either source the respective .env file into my shell before, or I run it like this: (. local.env && docker-compose <command>). Still a hack, but for now, I'm quite happy with this.

Have a nice one, y'all 🚶

[SC7639]

I forget but is there an actual reason it was taken away?

…

On Wed, 6 May 2020, 23:14 Julien Marechal, ***@***.***> wrote: A little reminder that a lot of people would like this feature :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31101 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOE6GGDIVGATP734YJA4UTRQHOLJANCNFSM4DANZGSQ> .

[a-abella]

The way yaml anchors handle this use case is a bit inconvenient compared toextend. It's power, in my opinion, came largely from recursive merging of elements. Anchors get you maybe 75% of the way there--they don't merge yaml recursively; all your merging happens at the top level. Referencing an anchored service template and then redefining the environment block will result in overwriting the anchored service's environment block instead of merging. You'll need to anchor and reference every dictionary down a recursive tree of dictionaries in order to match the behavior of the extend keyword.

An example:

# anchors for the service, environment, deploy, and deploy.placement blocks
# you'll need an anchor for every dict that you want to merge into
x-common-app: &common-app
  image: app:1.0
  environment: &common-app-environment
    common_option: a
    overwrite_option: b
  deploy: &common-app-deploy
    max-replicas-per-host: 3
    placement: &common-app-deploy-placement
      constraints:
        - 'node.labels.app_host==true'

services:
  myapp: << *common-app
    environment: << *common-app-environment
      foo: bar
      baz: xyzzy
      overwrite_option: quz
    deploy: << *common-app-deploy
      replicas: 15
      placement: << *common-app-deploy-placement
        preferences:
          - spread: node.labels.region

# The above yields the following:
services:
  myapp:
    image: app:1.0
    environment:
      common_option: a
      overwrite_option: quz
      foo: bar
      baz: xyzzy
    deploy:
      replicas: 15
      max-replicas-per-host: 3
      placement:
        constraints:
          - 'node.labels.app_host==true'
        preferences:
          - spread: node.labels.region

It might not look so annoying in this example, but it gets annoying and less readable if you're templating multiple (10+) services off one extension block.

In my mind an ideal scenario is a combination of the yaml anchor approach and the extend approach--allow extending only from a top level x- prefixed extension field block, with the smarter merging characteristics.

In my organization we found yaml anchors a bit syntactically sloppy so we basically re-implemented the extend feature in an external python script. That works for us, but it's a lame way to have to deal with something. Similarly we've had to create our own external tooling to deal with the removal of depends_on for v3/Swarm stacks.

[efrecon]

I have done a lot of gitlab CI YAML lately. It has exactly these features, which are sooo nice to achieve readable and manageable templates and final configurations:

• You can include other YAML files (good for templates)
• You can extend (even across projects/using remote resources through https). The extend documentation describe exactly what @a-abella describes for the compose format.
• You can also "hide" from being considered the real stuff. In compose format it's x-, in gitlab CI it's an initial . instead.

This is the exact set of feature that makes these files bearable.

[stellarpower]

I arrived at this issue from the docker docs, in my case I wanted to template my docker-compose setup, and as a 'workaround' I followed the advice above and did decide to look for an existing templating program. I won;t get those our hours back so am detailing a bit of what I found here, irrespective of any discussion on this actual feature request, however, it may be that those involved would feel the use of a YAML-based template system to generate a compose file rather than integrating this feature into docker-compose itself might be more appropriate in terms of encapsulation and picking the right tool for the job.

For some context, I am using a basic reverse proxy with Let's Encrypt and several application containers (Nextcloud for now, one for me, and some separated ones for friends) - in this case, I wanted to make a template of the Nextcloud containers so that I avoid mistakes and duplication with keystrokes for very similar setups. The following packages were what I tried:

• ytt
• Dhall
• CUE
• Jsonnet

ytt seems very comprehensive and is the only option to use YAML natively. It seemed powerful and the right tool for the job, and it uses Starlark, a superset of Python, directly inside the YAML file to performing processing. However after not long, the template became very messy, and the littering of fragments of code and fragments of YAML, plus mixing of Python data types like dictionaries and arrays and YAML fragments (that seem to be somewhat processed like text, a bit like using an HTML template engine that outputs tags like strings) eventually resulted in too many errors and too messy a file. Dhall also seems very comprehensive and uses a unique native language that can be output to a variety of formats; It seems more like a metaprogramming language than a template system, however given that the syntax is functional and it is pretty strictly typed, it quickly became more complex than was worth it for a simple template for unstructured YAML. As what seems a little like a blend of JSON and Haskell, it required too much thought to work what I needed done into the language.

Interestingly, sometihng that was difficult with both Dhall and ytt was using parameterised field names, both would have worked well enough otherwise, but I need to have the name of my instance appear in the services names and volume names, and in both of these achieving this was somewhat ugly; using arguments for the values in a hash is easy, but using those arguments in the names of the keys was messy or I couldn't find how to do it neatly, plus Dhall enforces type safety and this simply goes against that concept. With Jsonnet it was a simple as putting the expression in square brackets.

CUE and Jsonnet are both JSON-oriented, however running these through a converter is not hard at all, and it seems again, like Dhall, CUE has a lot of powerful features and was born out of shortcomings in Jsonnet, however partway into the documentation, it became apparent it was already overkill; perhaps with much more time to learn it properly it would be the superior option, but it seems CUE is more oriented to validation and schemata than a simple template job and so I quickly moved onto Jsonnet and finished the job quite quickly.

Finally, it was only after I had completed all this I realised the whole time I had been comparing these tools to the simplicity of Liquid tags or similar HTML templates, that I could actually probably simply have used Liquid in the first place. I have only used it in the context of a Jekyll site, so it just never ocurred ot me ot obtain a standalone package, however with basic looping and lists, and the ability ot evalate expressions straight into in-place text, this probably would have been a much better too for the job; Jsonify probably superior for JSON, but Liquid could operate in pure YAML and so the file becomes more readable again.

[jimwrightcz]

+1 docker-compose was one inspiration behind the bespoke solution I've implemented at work since this ticket was created to support migrating a large number of test envs to k8s. I was very careful to avoid bells and whistles but pretty quickly justified an analogous feature. The philosophical discussion (composition versus inheritance etc.) looks to me like a distraction from common sense (with the benefit hindsight - still unresolved nearly 3 years later). Evidently it is required by people who might continue to use docker-compose.

[darkn3rd]

+1 👍

I used this feature heavily before for dev/test/ci environments, where I could extend from a compose file in subdirectory paths of ./config/{dev,test,ci}/compose.yaml. I would have a .env that had COMPOSE_ENV=dev, but developers could override, and obviously I would override in ci.

I'm shocked by deprecating the feature and not replacing it with something that can do something similar. Maybe just allow us to use jinja2 and do what we want. I hope Docker-Compose would be less anti-DRY. :'(

[Vanav]

Seems that extends is supported by docker-compose since v1.27 (docker/compose#7588).

[yvess]

One use case where I use the feature heavily, is to version docker images to code. My docker dev and prod compose files both extend from docker-images.yml where only the basic service is listed and a tagged version of the image of the service.

Didn't found an easy workaround for this in v3.

[petrprikryl]

Is this issue also for docker stack deploy? Because I didn't find any in swarmkit repo.

And using docker stack deploy ... with extends in compose file results into:

extends: Support for `extends` is not implemented yet.

It would be good to know if it is planned feature for swarm beacuse its presence in new compose.

[pkit]

The way yaml anchors handle this use case is a bit inconvenient compared toextend. It's power, in my opinion, came largely from recursive merging of elements. Anchors get you maybe 75% of the way there--they don't merge yaml recursively; all your merging happens at the top level. Referencing an anchored service template and then redefining the environment block will result in overwriting the anchored service's environment block instead of merging. You'll need to anchor and reference every dictionary down a recursive tree of dictionaries in order to match the behavior of the extend keyword.

An example:

# anchors for the service, environment, deploy, and deploy.placement blocks
# you'll need an anchor for every dict that you want to merge into
x-common-app: &common-app
  image: app:1.0
  environment: &common-app-environment
    common_option: a
    overwrite_option: b
  deploy: &common-app-deploy
    max-replicas-per-host: 3
    placement: &common-app-deploy-placement
      constraints:
        - 'node.labels.app_host==true'

services:
  myapp: << *common-app
    environment: << *common-app-environment
      foo: bar
      baz: xyzzy
      overwrite_option: quz
    deploy: << *common-app-deploy
      replicas: 15
      placement: << *common-app-deploy-placement
        preferences:
          - spread: node.labels.region

# The above yields the following:
services:
  myapp:
    image: app:1.0
    environment:
      common_option: a
      overwrite_option: quz
      foo: bar
      baz: xyzzy
    deploy:
      replicas: 15
      max-replicas-per-host: 3
      placement:
        constraints:
          - 'node.labels.app_host==true'
        preferences:
          - spread: node.labels.region

It might not look so annoying in this example, but it gets annoying and less readable if you're templating multiple (10+) services off one extension block.

In my mind an ideal scenario is a combination of the yaml anchor approach and the extend approach--allow extending only from a top level x- prefixed extension field block, with the smarter merging characteristics.

In my organization we found yaml anchors a bit syntactically sloppy so we basically re-implemented the extend feature in an external python script. That works for us, but it's a lame way to have to deal with something. Similarly we've had to create our own external tooling to deal with the removal of depends_on for v3/Swarm stacks.

It fails for me with ERROR: yaml.scanner.ScannerError: mapping values are not allowed here
The correct working syntax is:

services:
  myapp:
    << : *common-app
    environment:
      << : *common-app-environment
      foo: bar
      baz: xyzzy
      overwrite_option: quz
    deploy:
      << : *common-app-deploy
      replicas: 15
      placement:
        << : *common-app-deploy-placement
        preferences:
          - spread: node.labels.region

[Persi]

Seems that extends is supported by docker-compose since v1.27 (docker/compose#7588).

Confirmed, works for me with docker-compose 1.29.

[panique]

Whoever decided to remove this extremely useful and widely used feature from Docker should be fired. This is an excellent example of how "improvement" of a product actually makes it less usable. This is one for the history books of software development ;)

[SC7639]

I can't believe it's been 4 years I've been tracking this issue and still no solution

[BretFisher]

@panique and @SC7639 have you read the recent comments? I'll just summarize to make sure we're all up to date:

The Compose Spec is the latest version of compose files and supports extends. The Compose CLIs (docker-compose and the new docker compose) support extends. For all cases but Swarm, "v3" is outdated and you no longer need the version statement in compose files. When you remove version: x.x the compose CLIs will use the latest Compose Spec features.

Now for Swarm Mode (which this ticket is about), where you can use a compose file for Stacks, the compose file still requires the older version: "3.8" in the file Swarm Mode has never supported extends. It was never "removed" from Swarm Mode, it was simply never developed for Swarm Mode.

Technically I believe this extends feature would have to be in the CLI, as Stacks is currently a CLI feature. That's actually good news, as the team may be more likely to help with Docker CLI updates than building a SwarmKit feature.

It looks like this CLI issue for moving to compose-go could go a long way in helping Swarm Mode Stacks support a feature like extends, which I don't believe would require any SwarmKit API changes (just an educated guess). Please give it a thumbs up!

Also, please add your thumbs up here for continued Swarm Mode support: docker/roadmap#175

And on these related issues:
docker/roadmap#194
compose-spec/compose-spec#156
docker-archive/compose-cli#1350

[tinducvo]

I was following the (docker-compose.yaml) => docker-compose config => (docker-composed.yaml) method to get a yaml that works with docker swarm. Things were working fine a month ago, after digging, I found out my setup was broken because docker-compose version 2.5.0 does not resolve the extends keyword whereas 1.29.2 did. This is even if I specify docker-compose config file version 2.1 in my all yamls.

[oliv3r]

Will it be possible to extend configs, networks or anything else? Or will it still be limited to just extending service. Which by itself is good, but still an oversight.

I wonder how hard it would be to just add support for the "#include" keyword, that would just merge compose files ... if the end-result would be the same as just doing 'cat *.yaml >> compose.yaml' then we'd solve 99% of the need :)