Not resolving using search domain

[NoumanSaleem]

Hostname is not properly resolving within the container when setting --dns-search on Docker.

Docker settings:

DOCKER_OPTS="--bip=172.17.42.1/16 --dns=172.17.42.1 --dns=10.0.2.15 --dns-search=service.consul"

gliderlabs/alpine

vagrant@local:~$ docker run --rm -it gliderlabs/alpine sh
/ # ping consul
ping: bad address 'consul'
/ # ping consul.service.consul
PING consul.service.consul (10.211.55.22): 56 data bytes
64 bytes from 10.211.55.22: seq=0 ttl=64 time=0.045 ms

progrium/busybox

vagrant@local:~$ docker run --rm -it progrium/busybox
/ # ping consul
PING consul (10.211.55.22): 56 data bytes

gliderlabs/registrator#111

[andyshinn]

Looks like musl doesn't support domain and search in resolv.conf: http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS. We're researching workarounds to this issue. Thanks for reporting!

[brycekahle]

@andyshinn the musl behavior of querying all nameservers in /etc/resolv.conf in parallel and using the the first response is unexpected. Perhaps that could be added to the documentation?

[andyshinn]

Yep, agree it should be noted somewhere. I'll add some more information to the docs about the caveats soon.

[pikeas]

@andyshinn The docs are a good start, but they raise further questions. It would be great if the section on dnsmasq (http://gliderlabs.viewdocs.io/docker-alpine/caveats) could be fleshed out to provide a full solution.

[andyshinn]

I tried to provide a general example of the use case where you want to send internal requests to a Consul server and the rest out to upstream name servers. What specifically do you think the example is missing?

[pikeas]

A brief description of dnsmasq and how to use it - on the host, as an additional container, etc. In other words, specific instructions on use, rather than "maybe try this workaround".

[progrium]

I think it could use a full dnsmasq config for the custom+upstream DNS
scenario so people can figure out how to run it on the host, then a pointer
to a good generic dnsmasq Docker container that would be easy to apply that
configuration for people that want to run it in a container, and lastly,
once available, a pointer to our magic resolve container.

On Tue, Mar 24, 2015 at 3:36 PM, Aris Pikeas [email protected]
wrote:

A brief description of dnsmasq and how to use it - on the host, as an
additional container, etc. In other words, specific instructions on use,
rather than "maybe try this workaround".

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

Jeff Lindsay
http://progrium.com

[thockin]

I was so hopeful for docker-alpine - I need a mini package for DNS utils. Alas, DNS in alpine seems to be totally busted. A shame.

[tifayuki]

I set up a dnsmasq container. In resolv.dnsmasq.conf, I added

nameserver 8.8.8.8
search example.com

In alpine container I set the dns pointing to that dnsmasq container.

The problem is when I do ping base in alpine container, the log of dnsmasq shows that only base is sent to 8.8.8.8, but base.example.com is never queried.

[andyshinn]

The 3 dnsmasq options that might append a domain to unqualified names is --expand-hosts, --domain, and --auth-zone. Can you try setting these to see if it changes the behavior? Unfortunately, the only other way to do this is with some host file manipulation on the dnsmasq container.

[neilellis]

This cannot be solved using dnsmasq, dnsmasq assumes glibc has expanded the hostname from the search entry in /etc/resolv.conf.

If you are in an environment (like Tutum) where links don't work because of this:

https://gist.github.com/neilellis/8983d6977f45f126df28

Is my workaround in conjunction with running dnsmasq:

dnsmasq --resolv-file=/etc/dnsmasq-resolv.conf --addn-hosts=/etc/hosts.links --no-daemon

The full image is at https://github.com/vizzbuzz/base-alpine

[progrium]

Alternatively we have a container called Resolvable that helps:
https://github.com/gliderlabs/resolvable

[hypergig]

Just because no one has mentioned it yet.. DNS search/domain resolution is really important in a kubernetes cluster. Kubernetes services are used for just about every integration point and they all resolve through DNS. Kubernetes injects the right information into /etc/resolv.conf to do this.

The fact that I can't connect to myservice.mynamespace from within an alpine container prevents me from doing anything useful. For example, I have a service called etcd.system that resolves to etcd host for http puts and gets. A very common thing is to write to etcd via curl for application configuration, etc. Alpine is the smallest base image that easily lets you download and use curl, but because my kubernetes service, etcd.system, isn't resolvable I am dead in the water.

I now have to use a 100 meg image to curl, when the 6 meg one would have done fine.

Just food for thought.

[progrium]

Totally agree. Hopefully this puts pressure on musl developers to address this problem. If anybody knows the best way to do this, please share.

In the meantime, we're probably adding search domain support to Resolvable.

[JeanMertz]

I guess this is still dead in the water. Ran into the exact same use-case as @hypergig, using Kubernetes, hitting DNS problems, finding out Alpine doesn't support DNS search.

The people at Kubernetes (specifically @thockin) seem to be aware of it at kubernetes/kubernetes#10163, but no dice so far.

[macropin]

I'd like to see this resolved too. We're using skydns for service discovery.

Has anyone raised this upstream with the musl devs?

[thockin]

I seem to recall not finding a way to file a bug with them.
On Sep 2, 2015 6:11 PM, "Andrew Cutler" [email protected] wrote:

I'd like to see this resolved too. We're using skydns for service
discovery.

Has anyone raised this upstream with the musl devs?

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[progrium]

I've not found a way yet but if anybody finds a way or files an issue be sure to share the link.

[andyshinn]

I've sent an email to the musl mailing list. It can be viewed at http://www.openwall.com/lists/musl/2015/09/04/3. I'm still gathering some information to answer all of Rich's questions (based on DNS use cases in Consul, Kubernetes and SkyDNS). If you have any specific use cases, please let me know or weigh in directly on the mailing list thread.

[thockin]

Awesome. Let me know if there is anything I can do to help.

On Fri, Sep 4, 2015 at 12:22 PM, Andy Shinn [email protected]
wrote:

I've sent an email to the musl mailing list. It can be viewed at
http://www.openwall.com/lists/musl/2015/09/04/3. I'm still gathering some
information to answer all of Rich's questions (based on DNS use cases in
Consul, Kubernetes and SkyDNS). If you have any specific use cases, please
let me know or weigh in directly on the mailing list thread.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[davidkbainbridge]

Any update on this? Just ran into it this morning.

[janeczku]

Sure, there is Alpine-kubernetes: A base image that comes with a special sauce for giving SEARCH domain powers to Docker Alpine. See here: https://github.com/janeczku/docker-alpine-kubernetes.

[mhart]

It looks like Rich's questions here are still unanswered: http://www.openwall.com/lists/musl/2015/09/04/5

Can anyone familiar with what Kubernetes et al need in this regard respond there? Or should we get Rich involved here? (I can't find him on GitHub though, so that might be out of the question)

[andyshinn]

I think @thockin just re-ignited the conversation. Sorry I've been so lax on the subject. I want to be respectful of everyone's goals and didn't really know how to reasonably answer the questions yet based on the communities goals.

[danielo515]

I'm also facing this issue on rancher, specifically connecting to mongodb databases. My image is unable to resolve the name of a link coming from another stack. However, if the links are within the same stack it works just fine. I don't understand why exactly.

I am now using a 300mb image when my average image size with alpine was 90mb. I would love to come back to alpine when this gets fixed.

[dchambers]

@danielo515 / @tuannvm: this is fixed from an Alpine perspective, but it's not as tolerant of non-conforming DNS servers, as was the case with Rancher's DNS until they recently fixed it for example.

[danielo515]

@dchambers do you know the minimum rancher version that works? I want to check mine. In wich Alpine version is this considered fixed ?

[dchambers]

I'm not 100% sure because I didn't perform the Rancher upgrade myself, but I know our cluster is definitely now working. More information in this issue.

[dchambers]

In wich Alpine version is this considered fixed ?

@danielo515, just saw the other part of your question and the answer is Alpine 3.4.

[tuannvm]

Still have an issue with alpine 3.4 / kube-dns:1.9
Hope that kube-dns:2.x.x would fix this.

[chenww]

alpine does not support multiple nameservers as well. Only one

[maxbbender]

Any updates on this? Using Jenkins based alpine images with dynamic slaves through kubernetes 1.8.3. Some alpine images can resolve external addresses and others can't. Both have the same /etc/resolv.conf files.

[Jean85]

I've stumbled onto this same issue, and I've solved it removing the a search directive, which was triggering a CloudFlare DNS (buggy) resolution.

I've found out about the solution with this blog post: https://blog.maio.me/alpine-kubernetes-dns/

[richfelker]

Cloudflare has EWONTFIX'd this issue according to:

kubernetes/dns#119 (comment)

[Jean85]

Exactly, that's why the only solution was removing the triggering domain from the search directive on my end.

[inodb]

This is closed, but I'm still having problems in the latest release running on kubernetes (kops on AWS):

kubectl run -i --tty alpine --image=alpine --restart=Never -- nslookup kubernetes.default
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

in busybox nslookup works fine:

kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup kubernetes.default
Server:    100.64.0.10
Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local

[Cloven]

This was eventually blamed on the alpine DNS resolver in musl, where it died a slow, forgotten death.

…

On Tue, May 22, 2018 at 2:16 PM, Ino de Bruijn ***@***.***> wrote: This is closed, but I'm still having problems in the latest release running on kubernetes (kops on AWS): kubectl run -i --tty alpine --image=alpine --restart=Never -- nslookup kubernetes.default nslookup: can't resolve '(null)': Name does not resolve Name: kubernetes.default Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local in busybox nslookup works fine: kubectl run -i --tty busybox --image=busybox --restart=Never -- nslookup kubernetes.default Server: 100.64.0.10 Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 100.64.0.1 kubernetes.default.svc.cluster.local — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAeOr3Q361ZnVQuDlqCOpyUmxy75CchCks5t1IAegaJpZM4DjZo8> .

[inodb]

@Cloven nice, thanks for clarifying! Seems like it could be nice to have a p big warning sign for using alpine images on k8s somewhere

[danlange]

I was experiencing a similar issue, and it was caused by a subtle configuration difference between alpine and glibc based linuxes.
If the "ndots" flag in /etc/resolv.conf is set to zero, then the search path will be disabled.
See: https://wiki.musl-libc.org/functional-differences-from-glibc.html