Containers not at the same network even though they are

[hnykda]

Hi. I am following https://github.com/lucaslorentz/caddy-docker-proxy#basic-usage-example-using-docker-compose, but it simply doesn't work because I get:

[WARNING] Container 15a8aba9d9f20f3bc03297b9e161ed1ff9480029640219fd6d9a526a86f8a300 and caddy are not in same network

and the generated block based on {{upstream 8000}} is "empty" (as if it's not being resolved to proper upstream service):

whoami.example.com {
  reverse_proxy
}

but I can clearly see that caddy and the other file are at the same network by using docker inspect. When I hardcode the hostname, it works fine:

version: '3.7'
services:
  whoami:
    image: jwilder/whoami
    networks:
      - caddy
    labels:
      caddy: whoami.example.com
      caddy.reverse_proxy: "whoami:8000" # "{{upstreams 8000}}"

networks:
  caddy:
    external: true

but that's not a great solution. Is there a way how to fix that?

A similar issue seem to be discussed here, but I don't understand the resolution: #104

[edudobay]

Hi, I'm having the same issue and after comparing the caddy logs with the output from docker events I found that:

• There is a small interval between the container being created and it being connected/attached to the network. In my environment this falls mostly on the 10~30 ms range,
• The message Container <ID> and caddy are not in same network seems to be printed ~100 ms AFTER the container is started (and is already attached to the network). At first this seems out of place, maybe this is a delayed response to the container creation?
  • Message timestamp: I'm relying on the timestamp from the logging driver itself, with docker logs -t
• This message is printed when I add a label caddy.reverse_proxy: {{upstreams}} (with optional arguments after upstreams), but not when the label is caddy.reverse_proxy: 'myhostname'

[edudobay]

By inspecting the source code, I found that {{upstreams}} works as expected if I set the environment variable CADDY_INGRESS_NETWORKS to the name of the Docker network that Caddy is connected to.

With the variable unset, this is the log output:

2021/04/22 03:43:21 [INFO] Running caddy proxy server
{"level":"info","ts":1619063001.2065825,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1619063001.2067482,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2021/04/22 03:43:21 [INFO] Running caddy proxy controller
2021/04/22 03:43:21 [INFO] CaddyfilePath:
2021/04/22 03:43:21 [INFO] LabelPrefix: caddy
2021/04/22 03:43:21 [INFO] PollingInterval: 30s
2021/04/22 03:43:21 [INFO] ProcessCaddyfile: true
2021/04/22 03:43:21 [INFO] ProxyServiceTasks: true
2021/04/22 03:43:21 [INFO] IngressNetworks: []
2021/04/22 03:43:21 [INFO] Connecting to docker events
2021/04/22 03:43:21 [INFO] Swarm is available: false
2021/04/22 03:43:21 [ERROR] Cannot find container id in cgroups: 0::/

[INFO] Skipping default Caddyfile because no path is set
[INFO] Skipping configs because swarm is not available
[WARNING] Container dbf5762040b2aedcaf3516abe4b0ad1d3975c64ac1fe693fc217276e030ba1c9 and caddy are not in same network
[INFO] Skipping services because swarm is not available

With the variable set, I get this:

2021/04/22 03:39:11 [INFO] Running caddy proxy server
{"level":"info","ts":1619062751.4558387,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1619062751.455977,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2021/04/22 03:39:11 [INFO] Running caddy proxy controller
2021/04/22 03:39:11 [INFO] CaddyfilePath:
2021/04/22 03:39:11 [INFO] LabelPrefix: caddy
2021/04/22 03:39:11 [INFO] PollingInterval: 30s
2021/04/22 03:39:11 [INFO] ProcessCaddyfile: true
2021/04/22 03:39:11 [INFO] ProxyServiceTasks: true
2021/04/22 03:39:11 [INFO] IngressNetworks: [files_webserver]
2021/04/22 03:39:11 [INFO] Connecting to docker events
2021/04/22 03:39:11 [INFO] IngressNetworksMap: map[2c0ba056e53e17cc0b7f2cf7800bf51fdee8674a0b82f4df871be228144377fc:true]
2021/04/22 03:39:11 [INFO] Swarm is available: false
2021/04/22 03:39:11 [INFO] Skipping default Caddyfile because no path is set
[INFO] Skipping configs because swarm is not available
[INFO] Skipping services because swarm is not available

[edudobay]

Running the Caddy container with the docker run --cgroupns=host option will restore the expected behavior.

(EDIT: Seems to me that the environment variable workaround would be ecommended over this one. I don't feel secure about recommending changes to cgroups settings without knowing exactly what that entails.)

This option is not yet supported in Docker Compose. docker/compose#8167

As far as I could understand it, this change in behavior is related to cgroups v2 — by default, the container ID is no longer visible in the child cgroup in which the container runs.

[hnykda]

I confirm it works when setting up CADDY_INGRESS_NETWORKS to the caddy's network 👍 , thanks for a workaround.

[lucaslorentz]

@edudobay Thanks for the detailed investigation.
Automatic network detection is not guaranteed and it's not working with cgroups v2.
I will update the readme with more details.

Closing as duplicate of: #207

[lucaslorentz]

Readme updated