[BUG] Intermittent "container <id> is not connected to the network <network>" errors on up #10668
Comments
|
Given that we're about to reconnect the network (with the full set of options? which I guess we can't do on create), would it make more sense to do the initial container create with network: none and then connect so we don't have to do the pre-emptive disconnect? |
|
This was implemented to replicate the |
|
I got the same error when I upgraded compose from |
|
If using the "docker in docker" image to deploy, reverting from |
|
We got the same error after upgrading to 2.19.0, after reverting to previous version things started working again. |
|
This is preventing DDEV from using v2.19.0, not sure what the fix is, fails on every platform |
|
@ndeloof is there some kind of workaround for this behavior on our end, or does it require a fix to compose? |
|
Compose v2.19.1 was just released with a fix for this: https://github.com/docker/compose/releases/tag/v2.19.1 |
@milas While this seems to fix the Example version: '3.9'
services:
mysql:
image: ${MYSQL_IMAGE_VERSION}
ports:
- "3308:3306"
environment:
MYSQL_USER: ${DATABASE_USER}
MYSQL_DATABASE: ${DATABASE_NAME}
MYSQL_ROOT_PASSWORD: ${DATABASE_ROOT_PASSWORD}
MYSQL_PASSWORD: ${DATABASE_PASSWORD}
volumes:
- mysql_data:/var/lib/mysql
networks:
- webapp
wait:
image: waisbrot/wait
environment:
TARGETS: mysql:3306
TIMEOUT: 300
networks:
- webapp
volumes:
mysql_data:
driver: local
networks:
webapp:
driver: bridge$ docker compose up -d
[+] Running 4/4
✔ Container webapp-wait-1 Started 3.0s
✔ Container webapp-mysql-1 Started 2.9s
$ docker compose run --rm wait
Waiting for mysql:3306 .mysql: forward host lookup failed: Host name lookup failure
.mysql: forward host lookup failed: Host name lookup failure
.mysql: forward host lookup failed: Host name lookup failure
$ docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
webapp-mysql-1 mysql:8.0 "docker-entrypoint.s…" mysql 8 minutes ago Up 5 minutes 33060/tcp, 0.0.0.0:3308->3306/tcp, :::3308->3306/tcpI'm reverting back to |
|
2.19.1 is not working for DDEV in many different ways (although the "not connected to network" seems to have gone away), hard to sort out all the test failures but all test types have some kind of new failure. |
|
Same. 2.19.1 not working. Reverted to 2.18.1. |
|
@ToshY I'm having trouble reproducing what you're seeing:
services:
mysql:
image: mysql
ports:
- "3308:3306"
volumes:
- mysql_data:/var/lib/mysql
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=1
networks:
- webapp
wait:
image: waisbrot/wait
environment:
TARGETS: mysql:3306
TIMEOUT: 300
networks:
- webapp
volumes:
mysql_data:
driver: local
networks:
webapp:
driver: bridgeIf I start |
|
Spoke too soon - I got a reliable repro and have a fix. I'm closing this issue to avoid confusion, follow #10777 for the |
As part of the fix for docker#10668, the logic was adjusted so that the default (highest-priority) network is used in the `ContainerCreate`, and then the remaining networks are connected via calls to `NetworkConnect` before starting the container. Unfortunately, `ServiceConfig::NetworksByPriority` is neither deterministic nor stable when networks have the same priority. It's non-deterministic because the order of networks from parsing YAML is random, since they are loaded into a Go map (which have random iteration order). Additionally, it's not using a `SortStable` in `compose-go`, so even if the load order was predictable, it still might produce different results. While I look at improving `compose-go` here to prevent this from tripping us up in the future, this fix looks at _all_ networks for a service and ignores the "default" one now. Before, it would always skip the first one in the slice since that _should_ have been the "default". Signed-off-by: Milas Bowman <[email protected]>
|
That said, although the "connected to the network" issues seem to be gone, there seem to be an interesting array of new problems in 2.19. Following #10777 |
As part of the fix for #10668, the logic was adjusted so that the default (highest-priority) network is used in the `ContainerCreate`, and then the remaining networks are connected via calls to `NetworkConnect` before starting the container. Unfortunately, `ServiceConfig::NetworksByPriority` is neither deterministic nor stable when networks have the same priority. It's non-deterministic because the order of networks from parsing YAML is random, since they are loaded into a Go map (which have random iteration order). Additionally, it's not using a `SortStable` in `compose-go`, so even if the load order was predictable, it still might produce different results. While I look at improving `compose-go` here to prevent this from tripping us up in the future, this fix looks at _all_ networks for a service and ignores the "default" one now. Before, it would always skip the first one in the slice since that _should_ have been the "default". Signed-off-by: Milas Bowman <[email protected]>
Description
Since I've run the E2E suite a bunch today, I've noticed the ddev test fails intermittently when bringing up the project:
That's coming from here:
compose/pkg/compose/convergence.go
Line 565 in 5155d22
When I modified the test locally to leave things running, I could see that the container was in fact connected to the network, so I think this is another Engine race condition with networks given that we create the container just before this
Steps To Reproduce
Run
TestComposeRunDdevE2E test a bunchCompose Version
Docker Environment
Anything else?
No response
The text was updated successfully, but these errors were encountered: