service in docker-compose resolved wrong ip ,resulting in a connection refused. #41766
Comments
|
Docker 18.09 reached EOL last year. Are you still able to reproduce this on a current (19.03 or 20.10) version of docker? We would also need to have a minimal reproducer (compose file using standard images and steps to reproduce), otherwise it will not be possible to investigate the issue. |
I can provide the docker-compose.yml the configuration serviceA proxy to serviceB service B |
|
@thaJeztah I deploy a stack to docker swarm, I use docker-compose to create three services and add them to the same overlay network. I should be able to ping one from the other using It is looking at IP 10.0.4.7, however if I inspect the network, I see the actual IP for that service is 10.0.4.8 and indeed, I can ping 10.0.4.8 and it works. This behaviour is the same for all the services that I deploy via a stack and is always the actual IP -1. Notes:
Steps to reproduce:
and System Info: and |
|
I rolled back to the previous stable version: 19.03.14 and it has the same problem. The adminer service is on 10.0.2.8 but trying to ping the service from another container and it looks for 10.0.2.7. |
|
@laxmanpradhan the IP address you're seeing is likely the VIP of the service itself; if you inspect the network with the |
|
@thaJeztah, thanks for your reply. Yes, you are correct, using the verbose option I see that the IP I'm seeing is the VIP of the service (which is 1 less than the container IPs). So I guess that leads to the question of why is it not able to connect on that IP? Why can't I ping the service from another container on the same overlay network? Using the adminer example, I try to connect to the database and it gives an error of "Host is unreachable". Shouldn't any requests to the service VIP get re-directed to one of the container IPs in that service? Note: I can connect using this |
|
ok my post above can be ignored, I don't think it is related to docker. I was running docker in a LXC container on proxmox. I switched the a VM and the DNS service works as expected. |
|
EDIT: Please use the updated patch of the post below #41766 (comment) I can confirm this behavior of the DNS system reporting the wrong container IPv4 address with one increment difference between the actual and reported IPv4 address when using docker in swarm-mode inside a LXC container (Arch linux image) created by LXD. I do however found a small workaround fix by adding the "hostname: " entry to the docker-compose stack file. The hostname field is then made equal to the service name, resulting in the correct IPv4 address when querying by the service name. What the fix looks like for the latest discussed situation: What the fix looks like for the OP situation: Version info of the test system used: |
|
@debugtux that hostname entry seems to be working, but do you have any documentation link for that? I cannot find its official document on how and what it is doing. |
|
[PATCH UPDATE] I did some more debugging as I noticed a very rare instability on the containers (failed connections due to wrong ip addr). Turns out the patch creates a race-condition on the name to be resolved by giving the name both the correct and incorrect ip address at the same time. Therefore, every so often the wrong ip is provided by the resolver. However, the patch still works and is stable with a different servicename to hostname. Only query by the hostname given and the correct ip address will be provided consistently; the servicename ip address will then again be off by one increment. My preferred way of implementing this is adding "service_" to the servicename and query the desired hostname as previously done. An example of the updated patch with a nginx container to be resolved at 'nginx': Last thing to note is that I am unable to replicate this problematic ip assignment behavior on docker-compose. I only encounter this problematic behavior with docker swarm (docker stack deploy); as this issue is originally directed to docker-compose and I am not the OP. |
Unfortunately I couldn't find any specific documentation on the network implications of these compose options other than the standard reference: https://docs.docker.com/compose/compose-file/#hostname But it appears that docker uses a special kind of networking and resolving for its servicenames. The hostname field is probably some form of an overwrite to name the container the standard way. But this is just my speculation and train of thought that inspired my patch. |
|
I do have the same issue and the workaround with hostname works aswell. I tried via network aliases which is not working. |
|
I found another workaround that might not have the race condition. It seems that is you use endpoint_mode dnsrr (https://docs.docker.com/network/overlay/#bypass-the-routing-mesh-for-a-swarm-service) it works as expected. I also noted that this bug (so without hostname or endpoint_mode fix) makes replicas undiscoverable. If you have a service replicated 4 times and you try to list them with PHP as follows: |
|
There is not much we can do without a reproducer. I'm closing the issue - though feel free to open a new one (and link to this issue) if you have reproducible steps. |
|
@Drallas which version of Compose are you using? Make sure that you use version 3.8 or higher. Example of my compose file which expicitly sets compose version: |
|
@fchapo This is my issue InvalidArgument desc = EndpointSpec: port published with ingress mode can't be used with dnsrr mode. The solution is to force host mode and not using the routing mesh. I guess I want Ingress Mode, in order to loadbalance my services. But if needed, |
|
It depends how you loadbalance. I use a separate service for that so I don't want Ingress Mode. And depending what you're doing you maybe don't need tu publish any port, so endpoint_mode: dnsrr should then work again as a workaround to the issue of this ticket :) |
Description
I am using docker-compose to manage my docker service.I have some containers which are running in a same docker-compose network.but it gave me surprise that when container A connect to container B by service name,it was refused as the ip of service was resolved to a wrong ip.
Steps to reproduce the issue:
it is my first time to see this strange behavior
I can not reproduce it
it works after I restarted the containert B
Describe the results you received:
another container in a same docker-compose network is refused when trying to connect it as it is resolved to a wrong ip
Describe the results you expected:
I hope the ip is resovled right
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
ubuntu 16
docker-compose info
The text was updated successfully, but these errors were encountered: