net: dns lookup sometimes fails with "no such host" when there are too many open files #18588

josselin-c · 2017-01-10T11:06:16Z

What did you do?

My network-intensive code would sometimes fail resolving domain names, returning "no such host" using the default resolver (pure go).The domains that fail resolving are valid domains that are successfully resolved using dig and the same dns server address.

I noticed that increasing the number of open files limit fix the issue.

I managed to reproduce the failure with: https://play.golang.org/p/MEZUS8h-o5

go build test.go
ulimit -n 20 # Set open file limit to something low
$ ./test 
2017/01/10 11:01:11 lookup munic.io on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
$ ./test 
2017/01/10 11:01:11 lookup munic.io on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
$ ./test 
2017/01/10 12:01:12 lookup munic.io on 127.0.1.1:53: no such host

What did you expect to see?

Always "too many open files"

What did you see instead?

Sometimes Lookup returns "no such host" when it shouldn't.

Does this issue reproduce with the latest release (go1.7.4)?

I first noticed the problem with 1.7.3. I reproduced the problem on 1.8beta2.

System details

go version go1.8beta2 linux/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/costanzj/projects/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build335463609=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
GOROOT/bin/go version: go version go1.8beta2 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.8beta2 X:framepointer
uname -sr: Linux 4.4.0-38-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.1 LTS
Release:	16.04
Codename:	xenial
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.23-0ubuntu5) stable release version 2.23, by Roland McGrath et al.
gdb --version: GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

The text was updated successfully, but these errors were encountered:

josharian · 2017-01-10T22:49:05Z

cc @mikioh @mdempsky

mdempsky · 2017-01-10T23:14:04Z

I think the issue is in goLookupIPCNAMEOrder. What's happening is the AAAA query fails with "no such host" (because munic.io has no AAAA records), but the A query sometimes fails with "too many open files". Currently if there's more than one error, which one we return to the caller depends on a race.

mikioh · 2017-01-11T10:13:01Z

There's a friend issue #18183. I'm inclined to return a list of effective errors when we cannot determine the cause of the Lookup API failure for surviving the dual IP stack era.

shuting-yst · 2017-05-09T05:27:48Z

I think the issue is in goLookupIPCNAMEOrder. What's happening is the AAAA query fails with "no such host" (because munic.io has no AAAA records), but the A query sometimes fails with "too many open files". Currently if there's more than one error, which one we return to the caller depends on a race.

@mdempsky I didn't catch the point reason, could you explain the detail?

bradfitz · 2017-06-08T21:33:12Z

We unfortunately didn't get to this, but if it's in 1.7 and 1.8, it's not a regression, so we'll look into addressing it for Go 1.10.

agnivade · 2017-08-30T07:28:00Z

I've looked into this a bit.

Yes, the issue is in goLookupIPCNAMEOrder. To be specific, its in the racer goroutine calls. I did some debugging and what is happening is it always returns the last error. Usually, there's only one error - "socket: too many open files", but sometimes the "no such host" comes up too. And when it does, "no such host" is always the last error.

Normal case - (I have added some debugging lines inside the racer lane handler which prints the error)

lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
2017/08/30 07:00:45 lookup munic.io on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files


lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
2017/08/30 07:00:45 lookup munic.io on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files

Other case -

lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: no such host
2017/08/30 12:30:46 lookup munic.io on 127.0.1.1:53: no such host

lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
lookup munic.io. on 127.0.1.1:53: no such host
lookup munic.io. on 127.0.1.1:53: no such host
lookup munic.io. on 127.0.1.1:53: no such host
lookup munic.io. on 127.0.1.1:53: no such host
lookup munic.io. on 127.0.1.1:53: no such host
lookup munic.io. on 127.0.1.1:53: no such host
2017/08/30 12:46:49 lookup munic.io on 127.0.1.1:53: no such host

I think if we change the logic to return the first seen error instead of the last error, its a more elegant solution. As mentioned by @pmarks-net on a slightly different but related thread #18183 (comment).

Combining the errors is also a possibility but as a user, I would prefer to see one error rather than a list of errors.

Overall, it seems to be a fairly simple enough fix. The real thing is to decide what it should be - returning all errors or the first encountered error. Or maybe something very different. Let me know what you guys think.

pmarks-net · 2017-08-30T20:46:03Z

According to goLookupIPCNAMEOrder, no ordering exists between the A and AAAA queries (until the actual addresses are sorted by sortByRFC6724).

So, "first" and "last" are meaningless when talking about A vs. AAAA; the only way I see to fix this case would be to define an ordering between errors, i.e. "too many open files" always replaces "no such host" when both are present.

Out of curiosity, does enabling StrictErrors make your example behave more predictably? That might only help if the two errors are generated by different elements of conf.nameList.

agnivade · 2017-08-31T17:17:05Z

So, "first" and "last" are meaningless when talking about A vs. AAAA; the only way I see to fix this case would be to define an ordering between errors, i.e. "too many open files" always replaces "no such host" when both are present.

"first" and "last" still hold meaning when you think of it as first error returned. As in whichever goroutine returned first with an error. Defining a precedence order among errors might be overdoing it IMO. There are lots of errors and lots of edge cases where one error might rank above another. Maybe a simpler solution might work here ?

Out of curiosity, does enabling StrictErrors make your example behave more predictably?

I re-ran the examples with StrictErrors. No such difference. I am not super familiar with the code base, but when I print the index inside the conf.nameList range loop, I see that it has only one element.

Here is the error case, with added logging -

21:44:09-agniva-~/play/gosource/go/src$./lookup 
[0]: fqdn- munic.io., error- lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
[0]: fqdn- munic.io., error- lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
[0]: fqdn- munic.io., error- lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
[0]: fqdn- munic.io., error- lookup munic.io. on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
[0]: fqdn- munic.io., error- lookup munic.io. on 127.0.1.1:53: no such host
2017/08/31 21:44:09 lookup munic.io on 127.0.1.1:53: no such host

mentioning the source code here for easy reference -

for i, fqdn := range conf.nameList(name) {
	for _, qtype := range qtypes {
		go func(qtype uint16) {
			cname, rrs, err := r.tryOneName(ctx, conf, fqdn, qtype)
			lane <- racer{cname, rrs, err}
		}(qtype)
	}
	hitStrictError := false
	for range qtypes {
		racer := <-lane
		if racer.error != nil {
			if nerr, ok := racer.error.(Error); ok && nerr.Temporary() && r.StrictErrors {
				fmt.Printf("[%d]: fqdn- %v, error- %v\n", i, fqdn, racer.error)
				// This error will abort the nameList loop.
				hitStrictError = true
				lastErr = racer.error
			} else if lastErr == nil || fqdn == name+"." {
				fmt.Printf("[%d]: fqdn- %v, error- %v\n", i, fqdn, racer.error)
				// Prefer error for original name.
				lastErr = racer.error
			}
			continue
		}
		addrs = append(addrs, addrRecordList(racer.rrs)...)
		if cname == "" {
			cname = racer.cname
		}
	}
	if hitStrictError {
		// If either family hit an error with StrictErrors enabled,
		// discard all addresses. This ensures that network flakiness
		// cannot turn a dualstack hostname IPv4/IPv6-only.
		addrs = nil
		break
	}
	if len(addrs) > 0 {
		break
	}
}

pmarks-net · 2017-08-31T22:20:41Z

"first" and "last" still hold meaning when you think of it as first error returned. As in whichever goroutine returned first with an error.

When two goroutines are literally racing each other, "first" and "last" both yield nondeterministic behavior, so I think it would be difficult to argue that either option is actually better. At least, I have no opinion in the matter.

tjmerritt · 2017-09-14T02:02:16Z

A different question is why do we get the 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files error in the first case. I hit this very quickly with 100 threads making outbound http requests. It seems that open files for make in UDP DNS requests are not be reused. This leak results in the too many open files error trigger when the open file limit for the user is hit. In my case this limit is set at 1024. With a max of 100 concurrent http requests, I don't really see the need for more than 300 open files. I could up the open file limit, but it increases the time to failure and does not eliminate it.

mikioh · 2018-02-16T01:13:42Z

@tjmerritt,

I filed a new issue, #23866, for your concern. Let's focus on "error handling" here.

simar7 · 2019-06-30T01:16:36Z

I'm running into this as well on go 1.12.5. I'd like to help if there's a decision made on how to approach on solving this.

mrichman · 2020-07-21T11:28:49Z

I think I just hit this issue on 1.14.5 as well:

RequestError: send request failed\ncaused by: Post https://dynamodb.us-east-1.amazonaws.com/: dial tcp: lookup dynamodb.us-east-1.amazonaws.com on 172.31.0.2:53: no such host

I get this when running one of my load test apps for a few minutes with 2000 or so concurrent goroutines.

davecheney · 2020-07-21T11:47:33Z

Disable cgo and it’ll work, this is a limitation of the libc resolver.

If you cannot rebuild the app without cgo, try GODEBUG=netdns=go

mrichman · 2020-07-21T12:09:08Z

@davecheney Thanks for that! I just tried GODEBUG=netdns=go without recompiling. Now I'm getting these too many open files errors:

{"level":"error","ts":1595333131.5405927,"caller":"ddbb/main.go:26","msg":"RequestError: send request failed\ncaused by: Post https://dynamodb.us-east-1.amazonaws.com/: dial tcp: lookup dynamodb.us-east-1.amazonaws.com on 172.31.0.2:53: dial udp 172.31.0.2:53: socket: too many open files\n\n","stacktrace":"main.worker\n\t/home/ec2-user/ddbb/main.go:26"}

I'm wondering if this isn't a ulimit issue on my EC2 instance.

EDIT: Also tried recompiling with GGO_ENABLED=0 and I get the same error, so I'm assuming it's a ulimit issue.

vkuznet · 2021-06-06T16:28:20Z

I experienced the same problem. And, I did tried to compile code with GODEBUG=netdns=go CGO_ENABLED=0 but problem persists. I did adjusted ulimit for number of files too (e.g. my limit was set to 1048576) and yet the same issue appears. And, I've seen this issue in Linux, macOS, and in k8s environments. What seems to be working is literally reduce number of concurrent calls and use a queue mechanism in the application.

matti · 2022-08-23T17:50:49Z

5 years later, this is still an issue

gopherbot · 2022-10-18T07:42:31Z

Change https://go.dev/cl/443255 mentions this issue: net: cgo resolver return correctly the too many open files error

bcmills · 2023-01-31T16:59:35Z

The discussion in #56192 seems to indicate that this is a glibc bug (https://sourceware.org/bugzilla/show_bug.cgi?id=29717) rather than a Go bug per se.

rsc · 2023-01-31T19:56:25Z

I can believe that it's a glibc problem, although I just saw this same problem happen on a Mac. On the Mac we're also invoking C code to run the lookup (because DNS traffic is blocked for ordinary Go code), so maybe the Mac library has the same bug.

mateusz834 · 2023-03-25T17:50:17Z

@rsc Side note: Today I stumbled across this: CL 121176, so it seems to be an old bug.

josharian changed the title ~~dns lookup sometimes fails with "no such host"~~ net: dns lookup sometimes fails with "no such host" when there are too many open files Jan 10, 2017

mikioh added this to the Go1.9 milestone Jan 11, 2017

bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label Jun 8, 2017

bradfitz modified the milestones: Go1.10, Go1.9 Jun 8, 2017

bradfitz added the help wanted label Jun 8, 2017

mikioh mentioned this issue Dec 1, 2017

net: a better builtin DNS stub resolver #22956

Open

r0wbrt mentioned this issue Dec 29, 2017

Moderate loads cause the Proxy server DNS resolving to fail r0wbrt/advertsieve#21

Open

ianlancetaylor modified the milestones: Go1.10, Go1.11 Jan 3, 2018

mikioh mentioned this issue Feb 16, 2018

net: better DNS transport for builtin DNS stub resolver #23866

Open

gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018

eur0pa mentioned this issue Jul 24, 2018

Massive refactoring OJ/gobuster#79

Merged

eur0pa mentioned this issue Aug 1, 2018

new features evilsocket/dirsearch#5

Closed

waveletlet mentioned this issue Mar 13, 2019

s2s lookup error ortuman/jackal#77

Closed

moredure mentioned this issue Aug 13, 2019

Does fasthttp.Client take care of half-open/half-closed connections? valyala/fasthttp#627

Closed

telendt mentioned this issue Sep 20, 2019

pubsub/awssnssqs: DNS resolution failure in tight receive-ack loop (AWS SQS) google/go-cloud#2672

Closed

ianmarmour mentioned this issue Sep 24, 2020

Bot not working ianmarmour/nvidia-clerk#73

Closed

gshively11 mentioned this issue Nov 3, 2020

rest client used by status/bundle plugins is leaking sockets open-policy-agent/opa#2859

Closed

andyw8 mentioned this issue Nov 23, 2020

Persistent unexpected error: DNS problem while connecting to Shopify, this indicates a problem with your internet connection Shopify/themekit#845

Open

This comment was marked as off-topic.

Sign in to view

This comment was marked as resolved.

Sign in to view

tb582 mentioned this issue Jan 10, 2021

dedup by-hash - dial tcp: lookup api.pcloud.com: no such host rclone/rclone#4918

Closed

vkuznet mentioned this issue Jun 6, 2021

Instabilities of DAS site dataset=/a/b/c queries dmwm/das2go#32

Closed

sfozz mentioned this issue Mar 18, 2022

Dogstatsd: dns lookup failed with "no such host" in container environment DataDog/datadog-agent#11338

Closed

yevgenypats mentioned this issue May 12, 2022

No such host and too many open files cloudquery/cloudquery#707

Closed

mateusz834 mentioned this issue Oct 18, 2022

net: cgo resolver return correctly the too many open files error #56255

Closed

bcmills mentioned this issue Jan 31, 2023

net: Misleading error "no such host" when running out of file descriptiors #56192

Closed

bcmills added the Unfortunate label Jan 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net: dns lookup sometimes fails with "no such host" when there are too many open files #18588

net: dns lookup sometimes fails with "no such host" when there are too many open files #18588

josselin-c commented Jan 10, 2017

josharian commented Jan 10, 2017

mdempsky commented Jan 10, 2017

mikioh commented Jan 11, 2017

shuting-yst commented May 9, 2017

bradfitz commented Jun 8, 2017

agnivade commented Aug 30, 2017

pmarks-net commented Aug 30, 2017

agnivade commented Aug 31, 2017

pmarks-net commented Aug 31, 2017

tjmerritt commented Sep 14, 2017

mikioh commented Feb 16, 2018

simar7 commented Jun 30, 2019

mrichman commented Jul 21, 2020

davecheney commented Jul 21, 2020 •

edited

Loading

mrichman commented Jul 21, 2020 •

edited

Loading

This comment was marked as off-topic.

This comment was marked as resolved.

This comment was marked as resolved.

vkuznet commented Jun 6, 2021

matti commented Aug 23, 2022

gopherbot commented Oct 18, 2022

bcmills commented Jan 31, 2023

rsc commented Jan 31, 2023

mateusz834 commented Mar 25, 2023

net: dns lookup sometimes fails with "no such host" when there are too many open files #18588

net: dns lookup sometimes fails with "no such host" when there are too many open files #18588

Comments

josselin-c commented Jan 10, 2017

What did you do?

What did you expect to see?

What did you see instead?

Does this issue reproduce with the latest release (go1.7.4)?

System details

josharian commented Jan 10, 2017

mdempsky commented Jan 10, 2017

mikioh commented Jan 11, 2017

shuting-yst commented May 9, 2017

bradfitz commented Jun 8, 2017

agnivade commented Aug 30, 2017

pmarks-net commented Aug 30, 2017

agnivade commented Aug 31, 2017

pmarks-net commented Aug 31, 2017

tjmerritt commented Sep 14, 2017

mikioh commented Feb 16, 2018

simar7 commented Jun 30, 2019

mrichman commented Jul 21, 2020

davecheney commented Jul 21, 2020 • edited Loading

mrichman commented Jul 21, 2020 • edited Loading

This comment was marked as off-topic.

This comment was marked as resolved.

This comment was marked as resolved.

vkuznet commented Jun 6, 2021

matti commented Aug 23, 2022

gopherbot commented Oct 18, 2022

bcmills commented Jan 31, 2023

rsc commented Jan 31, 2023

mateusz834 commented Mar 25, 2023

davecheney commented Jul 21, 2020 •

edited

Loading

mrichman commented Jul 21, 2020 •

edited

Loading