This repository has been archived by the owner on Sep 12, 2024. It is now read-only.
[solo] Use unbound to proxy to skydns; fixes shit #900
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Skydns now needs to be compiled with go 1.5+.
Current coverage is
|
davidxia
force-pushed
the
dxia/fix-solo-skydns2
branch
from
88248a8 to
c3b4743
Compare
davidxia
changed the title
davidxia
force-pushed
the
dxia/fix-solo-skydns2
branch
from
c3b4743 to
0d1483b
Compare
davidxia
changed the title
| @@ -23,7 +23,8 @@ done | |||
| curl -XPUT http://127.0.0.1:4001/v2/keys/skydns/${SKYDNS_PATH} \ | |||
| -d value="{\"host\":\"$HOST_ADDRESS\"}" | |||
|
|
|||
| skydns $SKYDNS_OPTS & | |||
| skydns $SKYDNS_OPTS -verbose & | |||
There was a problem hiding this comment.
I'm not sure if -verbose should be put in the image. Seems like you could run the docker image with -e SKYDNS_OPTS=-verbose at runtime if you really wanted verbose logging.
|
👍 overall |
davidxia
force-pushed
the
dxia/fix-solo-skydns2
branch
from
0d1483b to
8989006
Compare
TL;DR When two DNS servers don't work, add one more! When running some integration tests with HeliosSoloDeployment on Docker hosts that use a local unbound instance as its DNS resolver (i.e. specified in `/etc/resolv.conf` on the Docker host), we saw tests failures due to failed SRV queries to skydns. Skydns is running in the solo container and forwards DNS queries it doesn't know about to nameservers specified in `/etc/resolv.conf` via logic in `start.sh`. The skydns error output from the helios solo container spawned by HeliosSoloDeployment looked like: ``` skydns: failure to forward request "dns: failed to unpack truncated message" ``` Our guess is that large UDP responses from the upstream unbound have the "Message Truncated" DNS flag set. When this type of response reaches skydns, skydns blows up and doesn't tell the client about the error. The client times out without retrying in TCP mode. The client would've retried if it had received an error message from skydns. Running `dig` against skydns works. We think this is because `dig` adds an OPT record to its query that sets "udp payload size: 4096". Here are outstanding issues in skydns that seem related: * skynetservices/skydns#242 * skynetservices/skydns#45 Solution: We start an unbound instance in the solo container and have it forward DNS queries via UDP to the upstream skydns in the same container. Unbound will add the OPT section that makes everything work. Things are fixed. :) We admit this is super funky...And this only might work for UDP packets up to 4096 bytes, the default set by unbound in OPT. Much thanks to @gimaker for helping and suggesting unbound inside the container.
davidxia
force-pushed
the
dxia/fix-solo-skydns2
branch
from
8989006 to
886f1dc
Compare
|
@gimaker If it looks good to you, I'll merge and try to get a release out. |
|
@davidxia what happened to the nice commit message? :( |
|
Oh nevermind. I see that it's still there. |
|
👍 |
gimaker
pushed a commit
that referenced
this pull request
Jan 24, 2017
TL;DR; skydns does not handle TCP well. We already worked around this in #900. See thar PR for more context. However, that change only fixed the problem to an extent as we still have the same issue once the responses are >4096 bytes. This change extends that workaround to allow us to survive responses up to 32768 bytes in size. This change does not fix the issue, but should make it more rare.
vbhavsar
pushed a commit
that referenced
this pull request
Aug 28, 2018
and use ServicesResourceTransformer to relocate class names in META-INF/services/. fixes #900
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[solo] Use unbound to proxy to skydns; fixes shit
TL;DR When two DNS servers don't work, add one more!
When running some integration tests with HeliosSoloDeployment on Docker
hosts that use a local unbound instance as its DNS resolver (i.e.
specified in
/etc/resolv.confon the Docker host),we saw tests failures due to failed SRV queries to skydns. Skydns is
running in the solo container and forwards DNS queries it doesn't know
about to nameservers specified in
/etc/resolv.confvia logic instart.sh.The skydns error output from the helios solo container spawned by
HeliosSoloDeployment looked like:
Our guess is that large UDP responses from the upstream unbound
have the "Message Truncated" DNS flag set. When this type of response
reaches skydns, skydns blows up and doesn't tell the client about the
error. The client times out without retrying in TCP mode. The client
would've retried if it had received an error message from skydns.
Running
digagainst skydns works. We think this is becausedigaddsan OPT record to its query that sets "udp payload size: 4096".
Here are outstanding issues in skydns that seem related:
Solution:
We start an unbound instance in the solo container and have it forward
DNS queries via UDP to the upstream skydns in the same container.
Unbound will add the OPT section that makes everything work.
Things are fixed. :)
We admit this is super funky...And this only might work for UDP packets
up to 4096 bytes, the default set by unbound in OPT.
Much thanks to @gimaker for helping and suggesting unbound inside the
container.