compose "version 3" is not a version

[zaquestion]

Context
Primary spun off from #4513 and #4673

@shin- It is completely understandable that the team would want to create an optimal compose file for use with docker swarm and tying in closely with the docker stack. Personally I'm a fan of docker swarm and I hope for elegant compose tooling to use with it.

Issue
I strongly urge you to consider what it means to call this a "version" of the compose schema. "version" implies things like improvement, displacement, and ultimately standardization. What we have here is an alternate "mode", which is even the terminology used here: #4513 (comment)

@shin- Here #4513 (comment) you talk about compose v2 continuing support, but also how you hope the docker stack will take over most usecases then this #4673 (comment) seems to imply that the community can just take on updating compose.

Moreover, compose v3 is marked as the "current, recommended" format. Likewise there are steps on "upgrading" https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading

THIS DOES NOT INSPIRE CONFIDENCE

Alternate Routes

• Create a clear distinction between swarm and non-swarm modes. Potentially with a mode: "swarm" and bring everything up to version 3. Or even by splitting out the shared libs and making some swarm-compose

• Honestly, haven't broken that much, yet, volumes_from and extend are a matter of convenience and IMHO its good to be explicit, leave 'em out. Resources however is a decent chunk of lost functionality. We can move resources: out from under deploy: and support all of the limits that docker and compose v2 does. Additionally ensuring that documentation is clear about the subset of limits for swarm and the docker stack command can warn on ignored limits.

Either option would allow everyone to use the recommended version of docker-compose.

In any case it would be nice to have a more clear explanation of dockers plans for compose and potentially when the community should expect to support non-docker stack functionality on our own.

[ftenaglia]

Agreed. Talking about functionality, I miss resources for example - since I upgraded to v3 and don't use swarm mode.

The deploy section in v3 documentation does mention that, but the point is that upgrading to the latest and greatest version brings the problem of some things not being supported anymore. One upgrades all the parameters and then finds that the latest version was intended to swarm users.
https://docs.docker.com/compose/compose-file/#deploy

If no solution will happen prior to v3+, maybe a flag/note in the Compose File Reference stating that latest version is intended for swarm only, and not recommending for general usage.
https://docs.docker.com/compose/compose-file/compose-versioning/

[drdaeman]

Is there any reason why docker-compose cannot use parts of deploy dict that make sense for non-swarm scenarios? I believe Compose is also about service deployments.

I understand that it doesn't make sense with some keys, e.g. deploy.placement. However here's my summary, based on deploy's documentation:

• ✔️ While mode generally makes sense with Swarm, I believe it fits Compose model. If it's "replicated" then there are no behavior changes. If it's "global", replicas should be ignored and scale command should result in error.
• ✔️ replicas can be passed to Compose's Service.scale (and this would solve Feature request: add scale parameter in yml #1661)
• ❌ placement only makes sense with multiple nodes. There should be a warning if this key is present.
• update_config:
  • ✔️ parallelism and delay seems to be not supported by Compose's Service.scale at the moment, but don't see any reasons why they can't be. Would make a great feature.
  • ❓ I'm not sure about monitor, failure_action and max_failure_ratio. They probably can be also implemented, but I'm not sure how they work.
• resources:
  • ✔️ limits.memory and reservations.memory map well to --mem-limit/ --mem-reservation
  • ✔️ limits.cpu maps to --cpus, although docker-py doesn't support this yet (but it still can be set in _get_container_host_config using NanoCpus). If NanoCpus is not supported, code can always fall back to --cpu-period 100ms and --cpu-quota.
  • ❓ I'm not sure about reservations.cpu - don't think docker run or API has this one.
• restart_policy:
  • ✔️ condition and max_attempts maps to --restart.
  • ❌ delay and window seem to be not supported and use hardcoded values ("double the previous delay, starting at 100 milliseconds"). Use of those should probably result in a warning.
• ❓ labels are for services instead of containers, and since with Compose services are outside of Docker, those don't map to anything. Unless I'm missing something, I think those can be safely ignored silently, and if someone would need them, they can extend Compose ps with --filter support etc.

Based on this, I believe, most options can be supported, and only a few that cannot should result in warnings. Would appreciate further discussion/opinions on this.

If there will be an agreement that Compose indeed can and should support those (and how exactly), I can try to work on a PR.

[shin-]

I'm going to try and answer that to the best of my ability. Please understand that I don't necessary hold all the information here. I'll try to see if I can have people responsible for docker stack stuff and decision makers to take a look at this, but this is a very busy time for us all at Docker, so it may take some time.

First off, a point of semantics: "Docker Swarm" and "Swarm Mode" are technically 2 different things. The former is a product that has existed for some time now and is separate from the Docker Engine. The latter is a feature of the Docker Engine that was added in 1.12.0. Functionally they both fulfill the same role (clustering and orchestration), but in practice they function differently. [1]
Please believe me when I say I empathize and understand that this naming is confusing, but it's an important distinction. In the comment you link (#4513 (comment)), I say v3 is designed to work with "Swarm mode", the product - I'm not saying that this is a different "mode" for Compose.

As far as support of v2 is concerned, nothing indicates right now that we are going to discontinue or stop maintaining it anytime soon. If and when sunsetting it is decided, I trust Docker will announce this well in advance so there will be ample time for the then-users to decide whether they're ready to switch over to v3, or if they need to fork and maintain a community version of docker-compose when the time comes.

Compose v3 is the "current, recommended format" because it is the format we want new, oncoming users to adopt, as it is specifically designed for distributed applications scaling over a Swarm cluster. If you consider this is the goal of most application developers in the long run, it makes a lot of sense to recommend the format that achieves that painlessly rather than the one that attempts it, but struggles due to its technical limitations [2].

Realistically, we are not going to see major changes to the v3 format that you suggest, because it would straight-up break versioning. I'm not sure I got your point about shared libraries, but please consider that docker stack and docker-compose are written in different languages and function completely differently. Any attempt to have one of those products support the features of the other would be very tedious duplicating and might not even be possible. [3]

Regarding resources: I think the typical use-case we have in mind internally is that people using Compose v3 right now probably want to benefit from the simplicity of docker-compose in their dev environment (aggregated logs, no swarm cluster needed, etc.) and deploy to production using docker stack, where things like resource limits and replication become relevant. Does your experience not match this for some reason? If that's the case, we'd certainly be curious to know why so we can better match expectations.
On the other hand, if you're never using docker stack (for various reasons), I'd just reiterate what I said earlier: v2 is the better format for that.

---

@drdaeman Those options have a specific meaning in the context of Swarm services. It would be a mistake to try and map those to seemingly equivalent container properties, or worse, try to re-implement Swarm logic in Compose as you suggest for parallelism and delay.

---

[1] If you're still confused, "Swarm mode" is essentially Docker's second attempt at getting clustering right, in a more integrated fashion, building upon the lessons learned with "Docker Swarm" (including re-using some of the code and concepts).
[2] Mostly, docker-compose problem in the distributed scenario is that it's outside of the cluster and only interacts with Docker via its API. It's possible to do, but it's difficult - and it's probably never going to be perfect
[3] See the above point - docker stack benefits immensely from its integration with the Engine and has access to real time information about the cluster that we couldn't access through the Engine API.

[drdaeman]

@shin- Thank you for your detailed reply!

It would be a mistake to try and map those to seemingly equivalent container properties

Oh, no, sorry, I must've clarified it better. I've assumed that Compose - just like Swarm - also has a proper notion of services, with the main difference that Compose services are completely external to Docker and are managed solely by Compose, with no other piece of software being aware about those.

So I wrote not about containers, but about services (given that a single Compose service may consist of multiple containers, e.g. with docker-compose scale foo=3). And I thought it won't be wrong to apply some of deploy options to Compose services just like they apply to Swarm services.

That is, unless deploy.resources.limits.memory is for Swarm services (as a group) and not for individual containers that compose them. Then it surely would be a mistake, because Compose can't track summary resource limits. But I believe deploy.resources options apply to individual containers, don't they? My apologies if I got this part wrong.

or worse, try to re-implement Swarm logic in Compose as you suggest for parallelism and delay.

I don't see why it would be a mistake to implement some comparable Swarm Mode features (when possible and when makes sense) in Compose as well. Can you please clarify this a bit further?

I just don't see anything wrong if docker-compose up would notice parallelism and delay options when re-creating a container and use them appropriately. Yes, this means implementing a new feature in Compose, but is the project being sunset?

Does your experience not match this for some reason?

Unfortunately, no. Fun thing is, I'm currently doing things in exactly opposite manner. I'm developing and testing stuff with docker stack deploy - on a local single-node Swarm Mode "cluster", and deploying the production system with docker-compose up -d plus a shell script that sets the resource limits. I'd want to use docker stack there as well, but can't do this right now.

Swarm Mode lacks some features at the moment, but I'm planning to switch as soon as all the relevant issues I'm tracking are resolved. My desire for Compose to support v3.x deploy options comes from desire to maintain docker-compose.yml with the modern version schema to make sure things are as future-proof as possible.

[aaronjensen]

I think the typical use-case we have in mind internally is that people using Compose v3 right now probably want to benefit from the simplicity of docker-compose in their dev environment (aggregated logs, no swarm cluster needed, etc.)

We use docker-compose in dev and for CI right now (and actually don't deploy docker containers at all) and the main change that is preventing us from moving to v3 is not being able to set cpu_quota to -1, which seemed to be the only way to get docker compose services to be able to take more than one cpu. Are we missing something?

[EugenMayer]

Even though i see that some that some of the initial concerns have been covered by @shin- - the lack of volumes_from has not been covered.

I understand that most usage of volumes_from has been wrong, should have been simple named volume usage and thats basically it. But there is one scenario which is very common for docker stacks and that scenario has no longer an option to be implemented at all.

Assume you have this general case: 2 services, one "app" and one "httpd(proxy)". What we want is to have httpd serve static content and offer a upstream server for dynamic content ( ruby/php/java whatever ). So the very usual way of implementing the reference web application.

At this point, my question is, do we agree, that the above is the reference web-app application, language agnostic ( well at least we take out client side only application from this, but thats not like those are the common ones ).

---

App includes our code under /var/www - and also the static assets ( because 99% of the frameworks out there do mix those contents in one nested codebase ). So what we generally will do is, create APP, but then use volume /var/www to expose this folder as a anon-volume. Sure will have a data folder somewhere, mounted as a named volume and so on ( for the persistent data )

We do this, because we want to access this folder on httpd to be able to serve the static assets of the APP codebase. and thats where we will use volumes_from, semantically to actually mount "the anon volume of APP`.

---

Ok, i guess some people will jump onto that with the argument - use a named volume instead - and its done? No it is not. The problem is the upgrade. We want the codebase to be part of the APP image, and if the app image has been rebuild due to codebase changes, we want to obviously distribute this.

When you do a docker-compose pull && docker-compose down && docker-compose up with a anon volume for /var/www - that is exactly what happens and works.

When you will use a named volume for /var/www - it will not. docker-compose down does delete anon volumes (thus the old codebase) but will not delete ( for good reasons ) the name volume. Thus after doing docker-compose down && docker-compose up with a named volume for /var/www - we end up starting the stack with /var/www being overlayed by the older codebase in the named volume - thus our app runs with the new image but the old codebase.

And there is no way to work around this beside starting to do crazy things like removing some "named volumes" yourself on upgrade.

And there is more to that - your forced people to adopt v3 since it felt like it is the successor of of v2, similar to times when you introduced v2 over v1. And now the all start using named volumes in this scenarios and they will all be surprised by the fact, there applications will run on old code bases. So this is a huge inconvenience, waiting in the dark to bite your ass.

The upgrade mode "docker-compose pull / down / up" is common for all orchestrators i know - thus they will no longer work and people are left without an alternative.

---

Bottom line, removing volumes_from was maybe planned to be in a positive manner to stop people abusing it - but you removed a important tool which know completely blocks us ( and a lot more, as they will find out ) to use v3 ever.

[andyliddle]

We now have no way of Setting Memory limits (mem_limit) for containers. Currently we don't have a need to run swarms, as we only have one host in production, so we have to stick with version2 even though version 3 is now the preferred and recommended approach?

I still don't understand the rational for this.

[tsloughter]

Compose v3 is the "current, recommended format" because it is the format we want new, oncoming users to adopt, as it is specifically designed for distributed applications scaling over a Swarm cluster.

I think this is part of the disconnect. I started looking today to use compose for CI testing against a clustered version of our app. Seeing 3 is the newest and recommended I started there only to finally discover I actually have to use v2 to be able to define # of replicas, unless I want to deploy to swarm.

I think there is a disconnect between those who want to only use docker compose for development and its direction as a tool for use with swarm.

[krystalcode]

This is still an issue and confusing to users.

I spent lots of hours learning and adapting all my compose files (20+) to v3 thinking that this is the way moving forward, to keep things updated, and to take advantage of the latest features. I now find I cannot limit resources and I have to convert everything back to v2, including applications running in production.

I use docker compose extensively for development environments, and I also run multiple applications that do not need scaling even in production. I just need to organize my containers, I do not want to have to learn and maintain swarms neither do I need for mentioned use cases, it's an overkill.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zaquestion]

It's still a relevant issue StaleBot

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[antonagestam]

Just chiming in with some perspective here. I think the version 2 is not only usable for dev environments, but anytime you want to run Docker containers in a "single-host" mode. As a DBA I've used docker-compose to setup monitoring services on database hosts, in which case it makes no sense to use Swarm features.

Is there any concrete reason why this issue isn't being addressed? It seems like it would be a really easy win in terms of usability to just deprecate v2 and v3 and introduce a v4 that has a top-level compose-mode: "single-host" | "swarm" field to discriminate between the two modes. Like, it's now been quite a few years of pointing peers to this issue because everyone (justifiably) takes for granted that 3 is a version that succeeds 2 ...

[kjkent]

Just in case others land here after reading #4513, I think it's the case that the new Compose Specification (Compose 1.27.0+, superseding v2/v3 versioning) addresses some of the concerns -- at least expressed in the thread I link.

NB. In case folks encounter similar confusion, even those the compose spec says mem_limit & others are deprecated in favor of deploy-contained variables, I believe it's the case that these deploy variables aren't swarm-specific.

Apologies if I'm wrong; perhaps @shin- / @ndeloof can confirm.