feature: including external docker-compose.yml

[dnephin]

The Problem

I'd like to use fig for acceptance testing of services

In the case of a single service with a single level of dependencies this is easy (ex: a webapp with a database). As soon as the dependency tree grows to depth > 1, it gets more complicated (ex: service-a requires service-b which requires a database). Currently I'd have to specify the service dependency tree in each fig.yml. This is not ideal because as the tree grows, we end up with a lot of duplication, and having to update a few different fig.yml's in different projects when a service adds/changes a dependency is not great.

Proposed Solution

Support an include section in the fig.yml which contains url/paths to other fig.yml files. These files would be included in a Project so that the top-level fig.yaml can refer to them using <proejct>_<service>.

Example config

include:
    # link using http
    servicea:
        http: http://localhost:8080/projects/servicea/fig.yml
    # local paths
    serviceb:
        path: ../projects/serviceb/fig.yml

webapp:
    ...
    links:
        - servicea_webapp
        - serviceb_webapp

[dnephin]

I'm looking to implement this features myself, but I would like to upstream it eventually, so I'm interested to see how you feel about this idea.

I've looked over the code, and it seems to be relatively easy to make this work.

Some unresolved issues are:

• including services which use image should be easy enough, but services with build would need to ensure that the naming remains consistent, and that the service was already built (and possibly pushed to a registry). See next two issues
• in the case of remote repos, the config may need to support a repo name as well (as part of the includes section).
• project naming. Instead of using the file path/directory for a project, it uses the key from the parent yaml config (the file with the include section). This poses a problem if the names don't match. I was considering adding a project_name key to the config to keep these consistent (this is not really critical)

For fetching the external configs. I was going to use requests since it's already a dependency, but I was also considering supporting git as another method, possibly using dulwich

[dnephin]

I have a working prototype with a single integration test in https://github.com/dnephin/fig/compare/include_remote_config

It still needs some cleanup and more tests, but I thought I'd link to the current progress.

[bfirsh]

Nice!

[suzaku]

@dnephin @bfirsh +1 for the new feature.

I'm looking for a way to reuse fig.yml from another project,
before something like include is available,
I'll have to fallback to using copy/paste/edit ;(

[pirelenito]

That is awesome! I'd love to get this feature as well. Any updates on merging this on master @dnephin ?

[dnephin]

Cool, glad you are interested! I haven't had much time to finish this off yet. I still plan on doing it, but it might still be a week or two out. I've been thinking it actually probably requires #457 to deal with including any services that use build instead of image (because you wont be able to build them without the whole project).

The idea would be to use one of the tags as the image, and assume the other project has properly pushed the images and they are available to be pulled (or already cached locally).

[pirelenito]

Nice!

For a remote or standalone fig.yml file your solution of using tags seems pretty reasonable.

What are your thoughts if the fig.yml file is available locally inside its own project (meaning everything would be there)? Could the build be triggered? It seems to make sense.

[dnephin]

I haven't thought much about using it with locally available fig.yml. In that kind of setup would you have different services in subdirectories and you want to include them so that each individual one is smaller?

I think defaulting to use the build if no tags are provided is probably reasonable. It might need to do something with relative paths (append the path given in the include section to any build paths in the included fig.yml).

[dnephin]

https://github.com/dnephin/fig/compare/include_remote_config is mostly ready. There are two TODOs to resolve:

• convert remote build services to use the first tag
• support reading hdfs and s3 urls for remotes (currently supports http/https, and local files)

[ThomasSteinbach]

+1 - absolutely nescessary feature

It should be possible, to extend the imports. For instance a service included should get 'volumes_from' a data volume container.

Thus you can configure smaller orchestrations close to the services (e.g. gitlabService=gitlab+redis+postgres) and combine those smaller orchestration to a bigger one (whole orchestration = gitlabService + liferayService + jenkinsService). In the final fig.yml you can connect the imports to each other.

[rudibatt]

+1 for more reusable configuration

[clifton]

This would be extremely useful for my team and I 👍

[hadronzoo]

👍 for include

[manuwell]

+1

[rseymour]

extremely useful +1 as well over here.

[rseymour]

Playing with this, it seems to solve the initial problem, but not the exact problem I'm facing. My issue is wanting to fire up app X then at a later time link app Y (w/ it's own fig) to it. With this proposed solution, the second fig up will bring down the first app and restart it.

That fits with the initial problem, but ideally this could check to see if a set of containers from the included fig.yml was already up. In that case it could just do the link, /etc/hosts and ENV vars changes to link to the already up containers.

[dnephin]

@rseymour Thanks for trying it out!

I wonder if fig up --no-recreate will accomplish this?

[rseymour]

🏆 --no-recreate totally worked @dnephin !

[dnephin]

PR is up at #758 is anyone is interesting in giving some feedback

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[henri9813]

@keemyb , yes, that would work, but pretty dirty :/ in my opinion.

[jamshid]

This issue has a lot of different ideas but fwiw you can read the compose file from stdin with -f -. Reading directly from http would be nice, especially when there are multiple compose files, but at least this avoids a temporary file.
curl -fsS http://test.fr/docker-compose.yml - | docker-compose up -f -

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pirate]

Boop

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hasnat]

Boop

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[jgentil]

This isn't stale. C'mon, this is dead simple. We need a sensible way to combine and chain yml files together. Even a YAML directive like !include would be great.

[bsdis]

This is something that would be very good to have

+1

[thaJeztah]

I think for local files, this would (to some extend) be possible already through the extends option; https://docs.docker.com/compose/compose-file/#extends

tree
.
├── backend
│   └── docker-compose.yaml
├── docker-compose.yaml
└── frontend
    └── docker-compose.yaml

frontend/docker-compose.yaml:

services:
  web:
    image: nginx:alpine

backend/docker-compose.yaml:

services:
  db:
    image: mysql:latest

docker-compose.yaml:

services:
  web:
    extends:
      file: frontend/docker-compose.yaml
      service: web
  db:
    extends:
      file: backend/docker-compose.yaml
      service: db

docker compose up
[+] Running 3/3
 ⠿ Network compose_extend_default  Created                                                                                                                                                                                                                0.0s
 ⠿ Container compose_extend-web-1  Created                                                                                                                                                                                                                0.0s
 ⠿ Container compose_extend-db-1   Created                                                                                                                                                                                                                0.0s
Attaching to compose_extend-db-1, compose_extend-web-1
compose_extend-web-1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
compose_extend-web-1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
compose_extend-web-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
compose_extend-web-1  | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
compose_extend-web-1  | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
compose_extend-web-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
compose_extend-web-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
compose_extend-web-1  | /docker-entrypoint.sh: Configuration complete; ready for start up
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: using the "epoll" event method
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: nginx/1.23.3
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r4)
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: OS: Linux 5.15.49-linuxkit
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: start worker processes
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: start worker process 30
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: start worker process 31
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: start worker process 32
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: start worker process 33
compose_extend-web-1  | 2023/01/27 14:54:27 [notice] 1#1: start worker process 34
compose_extend-db-1   | 2023-01-27 14:54:27+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
compose_extend-db-1   | 2023-01-27 14:54:27+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
compose_extend-db-1   | 2023-01-27 14:54:27+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
compose_extend-db-1   | 2023-01-27 14:54:27+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
compose_extend-db-1   |     You need to specify one of the following as an environment variable:
compose_extend-db-1   |     - MYSQL_ROOT_PASSWORD
compose_extend-db-1   |     - MYSQL_ALLOW_EMPTY_PASSWORD
compose_extend-db-1   |     - MYSQL_RANDOM_ROOT_PASSWORD
compose_extend-db-1 exited with code 1

[theoilie]

this still would be a great feature to have 😄

[ndeloof]

Working on definition for this in compose file format, we (the Docker Team) can't establish a consensus on the right term to use for this feature.

So I created a poll to collect community opinion :)
https://github.com/docker/compose/discussions/10711

[pirate]

Very exciting! I hope someday this could help us towards having a Docker Hub equivalent for sharing Compose files. Import features would be very powerful if you could import / depend on a compose project from the "Hub" just like you can base Dockerfiles on images from Docker Hub. Could steal a bunch of thunder from Helm, which is way overkill for most homelab needs.

[ndeloof]

@pirate you'd be interested by compose-spec/compose-spec#288 then

[mbrevda]

@ndeloof to clarify: does this issue being closed mean that include is available in the latest docker compose?

[ndeloof]

@mbrevda it will be "included" (🤪) in next release, which should be v2.20.0

[mbrevda]

Looks like this didn't make the cut? Or did it just not make the release notes?

[ndeloof]

@mbrevda not in the release note because this is implemented in compose-go library