net: issue with DNS response > 512 bytes (cannot unmarshal DNS message) #21160
Comments
ianlancetaylor
changed the title
|
Have you tried the current Go release, 1.8.3? |
|
Yes, first I caught this bug with # kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"1dc5c66f5dd61da08412a74221ecc79208c2165b", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}After this, I installed |
|
Both of the hex dumps have prefixes before the message. As a result, neither is an RFC 1035 DNS message. The first message does not match the text you provided, so lets focus on the second message which both matches the text and has the same problem. The first two bytes of the DNS message should be the message ID (0x86fe). If you look closely, those two bytes don't show up until most of the way through line three. That is where the actual DNS message starts. I am not sure what the prefix is, but it isn't compatible with the current DNS library. |
|
Hi @iangudger, thanks for answer. But provided hex dump is dump of tcp-package, not only dns reply. |
|
@kvaps, what do you mean by tcp-package? |
|
@ianlancetaylor, I mean that this hex dump is contain the tcp/ip headers like source ip, dst ip addresses and other service information. You can use wireshark for open cap-files attached to my previous message and see all session between dns-client and dns-server. I can also attach the hex dump without tcp headers: |
|
@kvaps, I got confused when you were talking about TCP. The hex dumps you posted did not look like TCP DNS packets. I have confirmed from your packet capture that these messages were all exchanged over UDP, and they do look like UDP DNS packets. I am actually working on a solution to this problem. I have confirmed that the problematic message parses just fine with the new DNS library (golang.org/x/net/dns/dnsmessage). I am working on converting the standard net package to use it for DNS resolution. I am aiming to get this in Go 1.10. Once that lands, this problem should go away. |
|
@iangudger, No problem, thank you! |
ianlancetaylor
added
the
NeedsFix
|
What's the conclusion? An issue on some DNS recursive server that doesn't support DNS transport selection including TCP (RFC 7766)? If so, What's your solution? |
|
@mikioh, good point, but this is still RFC 1035. That longer packet is 530 bytes long with the UDP headers removed. The limit for UDP DNS messages is 512 bytes long. Well behaved DNS servers are supposed to truncate the message and set the truncated bit. See RFC 1035 section 4.2.1. |
|
I just wanted to know about the detail of "a solution to this problem." you mentioned above. Just guessing that we have four options: a) marking #6464 Go 1.10, b) making the existing DNS stub resolver relax to be able to grab a DNS message over 512 octets long on UDP transport (I'd rather not), c) returning more user friendly error values, d) other. |
|
Sorry, I still don't get it. I think we need to focus on the behavior of builtin DNS stub resolver here instead of message parser implementation because RFC 1035 clearly states that the UDP message size limit, and subsequent RFCs for DNS transport never try to change the limit for various reasons. Just for clarification, my question is: Do you want to change the behavior of the existing DNS stub resolver in Go 1.10 even if that violates RFC 1035? |
|
@mikioh, is there a good argument for doing so? |
Well, I'd like to ask what exactly means "for doing so." |
|
Is there a good reason to violate RFC 1035? For example, is there a later RFC that we should follow instead or is it commonly violated in a specific way? |
|
|
Sure, but I don't think implementing RFC 6891 will help as the server did not appear to implement it. I reviewed the packet capture, and I did not see any evidence of an attempt to negotiate a larger datagram size. |
|
@ianlancetaylor , this is just answer from my dns server manufacturer, about my message that their DNS Server is not responding by RFC 1035:
I should also add, that any other dns-client have no this problems unlike dns-client built in golang. |
|
As far as I can tell, the libc resolver makes no attempt to follow the RFCs on this matter: Using FIONREAD is inherently racy and using it reasonably correctly would probably not be possible with the net library. We could use 65536 byte buffers like libc's fallback behavior, but that does not seem like a good idea to me. According to RFC 5966:
This is the strategy used by Go's DNS resolver. You should suggest that your DNS server vendor follow the RFCs. |
|
Yes, I see, but why libc uses so large buffer? - maybe there were reasons like this for do this? For me, as user, this is really problem that all tools is working fine, but go written tools have the different look. |
|
glibc is, in my opinion, not very well written and should not be copied. In some modern versions of glibc, if the 65536 byte buffer code path is taken, glibc will allocate two such buffers, double free one, and leak the other. Basically they always use the FIONREAD code path. Using FIONREAD in the Go DNS client will likely be non-trivial, and, in my opinion, is a very bad idea. As far as I am concerned, the Go DNS client is working as intended here. I can tell you from experience that other popular DNS clients have this same issue. For the DNS server in golang.org/cl/51631, I originally just implemented a UDP server without truncation. DNS responses longer than 512 bytes did not work with the version of Node.js I was testing with. As a result, the version I sent out for review includes truncation for the UDP server and a TCP server. This seems to work well in practice, and I suggest that you request that your DNS server provider do the same. |
|
Sure, I will ask them about it again.
|
|
@iangudger , Sure.
nameserver 10.10.10.10
search domain.tld
|
|
That is because of #13281. If the response doesn't unpack, we ignore it. This could be fixed by only unpacking the header and not progressing to other search domains if we find a matching header. That will require golang.org/x/net/dns/dnsmessage as the current parser does not support incremental unpacking. |
|
Looks like we are on the same page. If we want to transmit/receive a DNS message over 512 octets:
Is my understanding correct? |
|
@mikioh, that all sounds good to me. The other option is that we might be able to get away with doing partial unpacking of large responses. For example, if all of the answers fit in 512 bytes but the total length is longer, we can just unpack the answers and ignore the truncated part. This shouldn't be relied on in the general case, but I think it will fix the case mentioned earlier in this issue if memory serves. |
|
Yup, that might be an option as long as we cannot provide a one-fits-all solution (and we cannot.) I personally hope that the stuff in x/net/dns helps people under heterogeneous environments. |
|
Change https://golang.org/cl/37879 mentions this issue: |
Vendors golang.org/x/net/dns/dnsmessage from x/net git rev 892bf7b0c6e2f93b51166bf3882e50277fa5afc6 Updates #16218 Updates #21160 Change-Id: Ic4e8f3c3d83c2936354ec14c5be93b0d2b42dd91 Reviewed-on: https://go-review.googlesource.com/37879 Run-TryBot: Matthew Dempsky <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]>
|
@kvaps, can you retry with tip? |
|
Hi, I was already changed my DNS server, sorry :) |
|
Here's a before/after with 1.9 and with Platform: Program: On 1.9: And with tip: The dig response for this name shows that it's over the limit.. |
|
Well, I am pretty sure that the original issue should be resolved in the specific case mentioned in this issue (the message was over the UDP limit, but the part we cared about was within the limit). The general case to allow arbitrary length UDP responses would require doing platform specific syscalls to allow these RFC violating DNS responses. I don't think that is something we want to do. |
|
fwiw I make no commentary with my repro above, but have been following this issue since it affects us fairly frequently due to a misbehaving upstream DNS server. Far as I can tell it's resolved, but I can't comment on 'correctness' of the resolution. |
|
The new incremental parsing will still fail if the wanted parts of the message fail to parse. The difference is that it will succeed if if parts of the message that it does not care about are unparsable. As a result, the incremental parsing should not affect correctness. |
|
As far as I can tell from the comments this issue is fixed. Please comment if you disagree. Thanks. |
We used to only accept up to 512 bytes in a DNS packet, per RFC 1035. Increase the size we accept to 1232 bytes, per https://dnsflagday.net/2020/, and advertise that larger limit in a EDNS(0) OPT record. Fixes #6464 Fixes #21160 Fixes #44135 Fixes #51127 Change-Id: I496a294e9a8015de4161cbc1825b0dc5b4e9f5d8 Reviewed-on: https://go-review.googlesource.com/c/go/+/385035 Trust: Ian Lance Taylor <[email protected]> Run-TryBot: Ian Lance Taylor <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Matthew Dempsky <[email protected]>
The existing value of 512 bytes as is specified by RFC 1035. However, the WSL resolver reportedly sends larger packets without setting the truncation bit, which breaks using the Go resolver. For 1.18 and backports, just increase the accepted packet size. This is what GNU glibc does (they use 65536 bytes). For 1.19 we plan to use EDNS to set the accepted packet size. That will give us more time to test whether that causes any problems. No test because I'm not sure how to write one and it wouldn't really be useful anyhow. Fixes #6464 Fixes #21160 Fixes #44135 Fixes #51127 For #51153 Change-Id: I0243f274a06e010ebb714e138a65386086aecf17 Reviewed-on: https://go-review.googlesource.com/c/go/+/386015 Trust: Ian Lance Taylor <[email protected]> Run-TryBot: Ian Lance Taylor <[email protected]> Reviewed-by: Damien Neil <[email protected]> Reviewed-by: Matthew Dempsky <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version)?What operating system and processor architecture are you using (
go env)?What did you do?
I have
get-ip.gowith this code:What did you expect to see?
What did you see instead?
# ./get-ip dial tcp: lookup storage.googleapis.com on 10.36.1.10:53: cannot unmarshal DNS messageAdditional information:
It works with
GODEBUG=netdns=cgo, but not withGODEBUG=netdns=go# GODEBUG=netdns=go ./get-ip dial tcp: lookup storage.googleapis.com on 10.36.1.10:53: cannot unmarshal DNS messageIt also started working after flushing dns-cache on my dns-server.
So I made capture of answer from dns-server:
before: (got error)
after: (is working)
And hexdump of package:
before: (got error)
after: (is working)
I presume that it connected with:
The text was updated successfully, but these errors were encountered: