Unable to send UDP packets larger than MTU to container

[rasky]

Expected behavior

I'm running a container with --net=host. From within this container, I'm connecting to a UDP service on the Mac host, using the Mac's wifi IP address (on en0).

I would expect the container and the host to be able to exchange UDP packets larger than MTU (at least up to Darwin's configured UDP max size, which is 9216 on my computer).

Actual behavior

If the Mac tries to send a UDP packet which is larger than 1500 bytes, the UDP packet never arrives. To debug with wireshark, I've used this container that allows me to monitor the traffic, and I can see that UDP packets larger than 1500 are never delivered.

In fact, I don't even see IP fragments in wireshark. It looks like the packet is lost somewhere.

Information

Diagnostic ID: C7708C76-F5E1-4567-960A-F9DDAC423F4E
Docker for Mac: version: 1.13.1 (94675c5a7)
macOS: version 10.12.3 (build: 16D32)

[ijc]

Thanks for your report.

The functionality of --net=host with docker for mac is not 1:1 with the behaviour of docker running on just plain Linux. There are a few issues about this e.g #155, #1031, #68 (and probably others).

Your request is an interesting new twist on that in that it's seems to not be purely a "routing" issue. I've added a note to that effect on our internal ticket.

The question of what semantics are expected for oversized packet being sent from a local socket to a local interface in an interesting one, I suspect there might be as many answers as there are UDP/IP stacks ;-). Is this is something which you know works between normal processes (ignoring d4m and containers for now) on MacOS?

The reason I mention this is that in d4m the concept of "host" is a bit split-brained. There is obviously the MacOS host but there is also a Linux "host" running in the VM. Docker operations which refer to "host" are normally in terms of that second Linux host, whereas users (rightly) expect them to be in terms of the MacOS host and we have various glue code which tries to produce that apparent effect. Currently there is no such glue for --host=net which is the subject of the tickets I mentioned above.

This is particularly relevant to you and to the question of expected semantics for local oversized frames is that even if the --host=net glue were in place the result would be unlikely to be truly "local" since they would be passing between two different network stacks (this is somewhat exposed today since you say you are using the IP of en0, TBH I'd expect with that being used the --host=net would be unnecessary/redundant).

So, after all that onto your actual problem: Are you sure your UDP frames don't have the DNF bit set in the IP header? That would be one obvious reason for them not to be showing up.

Would it be possible to put together a simple end to end repro of what you are seeing, e.g. a simplified version of your MacOS application and a Dockerfile for the containerized client along with the CLI commands we can use run to run them to demonstrate the problem?

[MagnusS]

This issue has been inactive for more than 14 days while marked as status/0-more-info-needed. It is being closed due to abandonment. Please feel free to re-open with more information about the problem.

MORE_INFO_EXPIRY_TIMEOUT

[Habbie]

Simple problem demonstration:

$ docker run -ti --rm debian:9
root@d5247c347cf7:/# apt-get update
...
root@d5247c347cf7:/# apt-get install dnsutils
...
root@d5247c347cf7:/# dig +ignore +tries=1 +timeout=2 txt  @37.139.13.17 1405.v4.big.7bits.nl

; <<>> DiG 9.10.3-P4-Debian <<>> +ignore +tries=1 +timeout=2 txt @37.139.13.17 1405.v4.big.7bits.nl
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43192
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;1405.v4.big.7bits.nl.		IN	TXT

;; ANSWER SECTION:
1405.v4.big.7bits.nl.	3600	IN	TXT	"len=1405 qname=1405.v4.big.7bits.nl. who=95.96.86.198 ecswho=nil bestwho=95.96.86.198 dnssecOK=false ednsPKTSize=4096 tcp=false xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

;; Query time: 13 msec
;; SERVER: 37.139.13.17#53(37.139.13.17)
;; WHEN: Tue Nov 27 19:05:37 UTC 2018
;; MSG SIZE  rcvd: 1472

root@d5247c347cf7:/# dig +ignore +tries=1 +timeout=2 txt  @37.139.13.17 1406.v4.big.7bits.nl

; <<>> DiG 9.10.3-P4-Debian <<>> +ignore +tries=1 +timeout=2 txt @37.139.13.17 1406.v4.big.7bits.nl
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Observation on the outside Ethernet interface on macOS:

$ sudo tcpdump -nli en7 -s 0 -v host 37.139.13.17
Password:
tcpdump: listening on en7, link-type EN10MB (Ethernet), capture size 262144 bytes
20:06:39.293347 IP (tos 0x0, ttl 63, id 26376, offset 0, flags [none], proto UDP (17), length 77)
    192.168.0.17.51399 > 37.139.13.17.53: 11678+ [1au] TXT? 1405.v4.big.7bits.nl. (49)
20:06:39.307018 IP (tos 0x0, ttl 51, id 41190, offset 0, flags [DF], proto UDP (17), length 1500)
    37.139.13.17.53 > 192.168.0.17.51399: 11678*- 1/0/1 1405.v4.big.7bits.nl. TXT "len=1405 qname=1405.v4.big.7bits.nl. who=95.96.86.198 ecswho=nil bestwho=95.96.86.198 dnssecOK=false ednsPKTSize=4096 tcp=false xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" (1472)
20:06:41.053054 IP (tos 0x0, ttl 63, id 44246, offset 0, flags [none], proto UDP (17), length 77)
    192.168.0.17.60544 > 37.139.13.17.53: 12475+ [1au] TXT? 1406.v4.big.7bits.nl. (49)
20:06:41.066174 IP (tos 0x0, ttl 51, id 41384, offset 0, flags [+], proto UDP (17), length 1396)
    37.139.13.17.53 > 192.168.0.17.60544: 12475*- 1/0/1 1406.v4.big.7bits.nl. TXT[|domain]
20:06:41.066272 IP (tos 0x0, ttl 51, id 41384, offset 1376, flags [none], proto UDP (17), length 125)
    37.139.13.17 > 192.168.0.17: ip-proto-17

First and third packets are the two queries from inside Docker. Second packet is the exactly 1500 byte response that does make it to dig inside the container. Fourth and fifth packet are the two fragments of the bigger (1501 bytes, plus overhead due to fragmentation) response.

Observation from in between:

$ docker run --rm --net=host crccheck/tcpdump  -n -l
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:09:33.706112 IP 192.168.65.3.51138 > 37.139.13.17.53: 17759+ [1au] TXT? 1405.v4.big.7bits.nl. (49)
19:09:33.722765 IP 37.139.13.17.53 > 192.168.65.3.51138: 17759*- 1/0/1 TXT "len=1405 qname=1405.v4.big.7bits.nl. who=95.96.86.198 ecswho=nil bestwho=95.96.86.198 dnssecOK=false ednsPKTSize=4096 tcp=false xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" (1472)
19:09:36.148682 IP 192.168.65.3.58591 > 37.139.13.17.53: 47304+ [1au] TXT? 1406.v4.big.7bits.nl. (49)

There, we see the first query, the response to it, the second query, but not the fragments of the response.

[Habbie]

@MagnusS any way to reopen this? Thank you.

[andrewyager]

I can confirm that I have reproduced this today with packets that are fragmented. Testing outside container all traffic is delivered correct. TCPDump shows response received from external to Mac, but traffic never appears inside container. Requests smaller than MTU are delivered.

[SvenVD-be]

Any update? I encountered the same problem.