Feature request: add scale parameter in yml

[jmmills]

As a user of compose (formally fig), I would like to be able to specify the number of nodes started for any given definition (a.k.a scale) from inside the manifest (yaml configuration file), so that I can ship my cluster definition with my service orchestration.

E.g. syntax:

worker:
    build: rqworker
    scale: 5
    links:
       - redis
    command: rqworker -u tcp://redis 

[aanm]

@jmmills Can't you start your containers such as: "docker-compose scale worker=5" and that service would start with 5?

[jmmills]

@aanm Yes, but I think that functionality should be mirrored as a default in the service definition. Maybe I have a minimal set of workers that should always be running and I want that to be clearly declared as a default.

[prologic]

@@aanm Do you expect that a docker-compose up should take the scale parameter into account? The only problem with this that I see is that we're introducing parameters into the declarative configuration that are NOT compatible with the underlying Docker / Docker API concepts but very specific to Docker Compose.

If we were to do this going forward; I'd suggest something like:

worker:
    build: rqworker
    $scale: 5
    links:
       - redis
    command: rqworker -u tcp://redis 

Where $<parameter> denotes a Docker Compose specific thing.

[aanand]

We've gone back and forth on whether scale belongs in the YAML; there's been a PR to add it before (#630). I'm still of the opinion that most people shouldn't put scale numbers in their config, because it makes it less portable, but I understand the use cases for doing so.

Now that we've got a rudimentary way to augment Compose files with extends (and hopefully better ones soon - #1380, #758), the concerns I raised in #630 (comment) are perhaps less of an issue.

[jamshid]

I'd like to set scale=0 in my yml for test-related services that I don't normally want started. I only want to create those services explicitly with docker-compose run or an explicit scale .

[jmmills]

@jamshid I've often wanted that, a definition that sets up an environment but doesn't run by default. I've been relegated to creating a base image (which a zero/no-op scale would also help with) in which I run my unit tests out of (via docker run), and then my container composition consumes the base image.

Something like this seems pretty useful for dev configurations

myproject:
    build: .
    command: nosetests
    scale: 0
    links:
       - redis
redis:
    image: redis
apiserver:
    image: myproject
    command: plackup
    links:
       - redis
workerserver:
    image: myproject
    command: rqworker
    links:
        - redis

[prologic]

@jamshid @jmmills What about a enabled parameter/key in the YAML file per service? Such that you can disable/enable a service?

[jmmills]

@prologic Why do that when the a "scale" parameter would solve both needs?
If you want to imagine a running process/container as an instance of a class, one could even name it instances

[prologic]

@jmmills I'm just trying to find a solution to your use-case that doesn't involve breaking the current docker-compose as such. I do tend to think scale=0 doesn't seem that fitting and I'm in two minds about whether scale=X should even be part of Compose itself.

[jmmills]

In my opinion scale (or number of copies) is part of the composition of a service, thus should be included in compose.

[prologic]

Well I think we either have a scale=0 or disabled key.

[JeanMertz]

👍 on having the capability of setting a default scale size for an instance. And I agree, once scale is in, there is no need for a disabled key, as you'd simply set scale to 0.

[Filirom1]

+1

[jmmills]

Also, another use case: What if I want to scale the number of containers but don't want to background all of the services, or have to jump over to another terminal (or process) and set my scale numbers...
e.g:

$ docker-compose up && docker scale thing=4

Doesn't work because up doesn't exit.
But if my composition file sets the scale of my containers...

$ docker-compose up

Becomes DWIM.

[prologic]

I'm not sure that I really like this; all of a sudden up takes on two capabilities:

• Bring any containers up that don't have scale parameter.
• Bring any containers up that have scale=0.

We're now abusing the "up" command really. "Scale" also takes on new meaning in that it now does two things:

• scale containers up/down
• Disable containers/services by way of ``scale=0`

[jmmills]

Why would up bring up containers with a scale=0?
Build would build images with a scale=0, thus facilitating a base image need.

[prologic]

I could be wrong but reading your last comment it kind of implied that :)

[jmmills]

Let me elaborate:

base_thing:
    build: .
    scale: 0

thing:
    image: base_thing
    # scale: 1 implied by default

workers_for_thing:
    image: some_other_image
    scale: 4
    links:
      - thing

test_harness:
    image: base_thing
    command: nosetests --where=my_test_code_dir --all-modules
    scale: 0

Now, expected behavior: docker-compose build builds any containers with "build" (does compose pull in external images on build? don't remember), docker-compose up would run anything with a positive scale (default is 1), docker-compose run test_harness would build "base_thing" if needed and run my one command. Savvy? :)

Edit: docker-compose up would run 1 "thing" and 4 "workers_for_thing"

[prologic]

Okay :) Thanks; your example makes a bi more sense and a bit clearer as to the intention of scale=0

I think docker-compose "pulls" images on up / run.

[Miuler]

I need create recipe indicate number of de intances (scale), for test, production, qa, etc.

[mclate]

+1 for scale=X. This would be very helpful.
And +1 for @jmmills comment with configuration description and expected results.

[verges-io]

Yay! for scale=x. Initializing a set of containers would definately help to identify potential race conditions when setting up cluster configurations.

[michaelcontento]

+1 for scale=x (including scale=0 to disable services for the initial docker-compose up)

[ymmijgno]

+1 for scale=x.

[aanm]

x is NaN, I would propose -1 instead.

[samuelololol]

+1 for scale=x.

[sutyrin]

+1

[PaulPetring]

+1

[westlakem]

Services in the new version of Docker does offer a 'replicas' feature.

…

On Mon, Mar 20, 2017 at 3:28 PM, alwaysastudent ***@***.***> wrote: I support @dnephin <https://github.com/dnephin> proposal on #2496 <#2496> Please give us this basic feature. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1661 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AES98uWGEnbyCFyxNSSlv7QkXy5oBek1ks5rntNBgaJpZM4FS8ZQ> .

[alwaysastudent]

@westlakem - that feature is for swarm. It is not for the regular service scaling.

[cgarciae]

So many pros, few cons. What is the matter with this issue?

[matti]

There is a more recent proposal about completely removing the scale command in #2496

I really dont understand why or why there is no initial_scale: 3 option in compose file (I do understand why it can not be just scale: 3)

[shin-]

Implemented back in 1.13.0. Thank you everyone for your feedback and patience.

[greenscar]

@shin- Why is this considered closed? In managing large scale apps where EVERYTHING is automated & in version control, I now must modify my Ansible which executes Docker to define how many instances of a service will be running.

Ansible should be setting up everything so Docker can run. Docker should be defining how many instances of a service the app needs. Now it's Docker's responsibility to know the inventory of services but Ansible's responsibility to call Docker so it knows how many containers to spin up?

[shin-]

@greenscar Sorry, I'm not exactly sure what the problem is - are you unhappy about the way the feature has been implemented? If so, in what way can we make it better?

[cgarciae]

Why is this not present in version 3? Extreamly confusing since "deploy.replicas" and "up" are not equivalente, you could say this feature is missing in v3.

[shin-]

@cgarciae deploy.replicas serves the same purpose as scale. Why are they not equivalent in your view?

[cgarciae]

@shin- If I dont want to create a swarm and only use docker-compose up -d, docker compose tells me it will ignore the "deploy" settings.

[shin-]

@cgarciae Why are you using a v3 file if you don't want to create a Swarm?

[aphillipo]

I think everyone using docker expects the scale parameter to work for either docker 2 or 3 files. There is no reason for it not to and it could even set the default value for deploy.replicas as well. If deploy.replicas is set it therefore overrides scale in swarm. What is wrong with this picture apart from it being what everyone expects to happen (and of course if you don't like it no need to use it in your docker-compose.yml)?

[boscowitch]

The missing scale parameter made a servie fail, since i forgot to scale it up like it was before since the last redeloy -.- why is this not a thing it would save service and maybe even lifes depending on what its running

[tsloughter]

Just hit this issue as well. Seems there is no way to specify how many of a service to start from within the docker compose config? It is even more confusing that the docker compose documentation has a lot of documentation for things that don't work with docker-compose...

[tsloughter]

Oh wait, so if i change my docker compose config to version: "2.2" then I can use scale: 2 but if one were to want to go to a later version there is no longer an option to do this?

[tsloughter]

Ok, I see this issue of "version" not really meaning versions has already been brought up, #4693

[freesh]

I use v3.6 because i need some configs that does not exist in 2.2. For local development i use no swarm, but i want to set scale=0 to some containers in my docker-compose.overrides.yml. Because some of them are just for buildprocesses like the frontendbuild container. This container shares some volumes with other running containers, but should not started together with these other containers on docker-compose up. So the buildcontainer just runs with the run command. Yes i can set the --scale param on calling docker-compose up, but i think it is better to write it directly to the configuration. :)

[samrocketman]

Thanks for the scale parameter. I was able to use it in a non-production docker-compose.yml so that anybody could use a HA configuration of Consul and Vault. The scale parameter made provisioning a local HA configuration easy. https://github.com/samrocketman/docker-compose-ha-consul-vault-ui

In this case, only intended for somebody to try out Consul and Vault in a HA configuration and use Consul DNS.

[jjsaunier]

Which version of docker compose are you running on ? Since v3 this feature has been dropped (By drop, I mean, never implemented), so it's ignored. See https://docs.docker.com/compose/reference/scale/

[samrocketman]

I don’t ever use v3 format due to missing features. I tend to use the lowest 2.1 or 2.2 format depending on what features I’m using.

Edit: not to say I avoid v3 format. Just that when I go to write a compose file it is usually missing what I want to use.

[jjsaunier]

Ok, I guessed you speak about docker compose with swarm stack, that's why. I never tried, but it's weird to talk about HA with docker compose without orchestrator and mutli node. Is it for "poc" or testing purpose ? Anyway, I never tried in this context

[samrocketman]

In this case, the HA simply refers to a consul cluster and a vault cluster. It is meant to serve as a proof of concept available for experimentation and ease of bootstrap. The HA does not refer to docker HA or even multi-machine HA. It's only HA in that the person experimenting can kill one of the cluster containers and see that the consul and/or vault service is not affected.