Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conntrack flush support #32505

Merged
merged 3 commits into from
Apr 11, 2017
Merged

Conntrack flush support #32505

merged 3 commits into from
Apr 11, 2017

Conversation

fcrisciani
Copy link
Contributor

@fcrisciani fcrisciani commented Apr 11, 2017

- What I did
Fixes #8795
Fixes #31610
Fixes #31414

On external connectivity removal, the bridge driver will take care of erasing all the conntrack flows relative to the endpoint that is going down

- How I did it
Introduced the conntrack support in the netlink library and then from libnetwork will cleanup the flows passing the IP address of the endpoint that is going down

- How to verify it
Added a test that creates a server container and a client container and establishes an UDP flow (using UDP for simplicity) between them.
The test verifies the presence of the flow fetching its from the conntrack table.
The server container is then destroyed and the tests validates that the previous flow got purged from conntrack.

- Description for the changelog

Conntrack flush on external connectivity removal

- A picture of a cute animal (not mandatory but encouraged)
15

Flavio Crisciani added 2 commits April 10, 2017 17:09
- Added conntrack support

Signed-off-by: Flavio Crisciani <[email protected]>
- adding conntrack flush fix for moby#8795

Signed-off-by: Flavio Crisciani <[email protected]>
@mavenugo mavenugo added this to the 17.05.0 milestone Apr 11, 2017
@mavenugo
Copy link
Contributor

@fcrisciani thanks for resolving this long standing issue and a good test to catch it as well.

LGTM

@vieux
Copy link
Contributor

vieux commented Apr 11, 2017

LGTM

c.Assert(err, check.IsNil)
var flowMatch int
for _, flow := range flows {
if flow.Forward.Protocol == 17 &&
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use constant for this magic number?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

@mavenugo
Copy link
Contributor

@fcrisciani the windowsRS1 failure seems genuine -

02:32:17 # github.com/docker/docker/vendor/github.com/vishvananda/netns
02:32:17 ..\vendor\github.com\vishvananda\netns\netns.go:27: undefined: syscall.Stat_t
02:32:17 ..\vendor\github.com\vishvananda\netns\netns.go:28: undefined: syscall.Fstat
02:32:17 ..\vendor\github.com\vishvananda\netns\netns.go:31: undefined: syscall.Fstat
02:32:17 ..\vendor\github.com\vishvananda\netns\netns.go:39: undefined: syscall.Stat_t
02:32:17 ..\vendor\github.com\vishvananda\netns\netns.go:43: undefined: syscall.Fstat
02:32:17 ..\vendor\github.com\vishvananda\netns\netns.go:57: cannot use int(*ns) (type int) as type syscall.Handle in argument to syscall.Close
02:32:17 FAIL	github.com/docker/docker/integration-cli [build failed]

@AkihiroSuda AkihiroSuda added status/2-code-review status/failing-ci Indicates that the PR in its current state fails the test suite and removed status/4-merge labels Apr 11, 2017
When a container was being destroyed was possible to have
flows in conntrack left behind on the host.
If a flow is present into the conntrack table, the packet
processing will skip the POSTROUTING table of iptables and
will use the information in conntrack to do the translation.
For this reason is possible that long lived flows created
towards a container that is destroyed, will actually affect
new flows incoming to the host, creating erroneous conditions
where traffic cannot reach new containers.
The fix takes care of cleaning them up when a container is
destroyed.

The test of this commit is actually reproducing the condition
where an UDP flow is established towards a container that is then
destroyed. The test verifies that the flow established is gone
after the container is destroyed.

Signed-off-by: Flavio Crisciani <[email protected]>
@thaJeztah thaJeztah removed the status/failing-ci Indicates that the PR in its current state fails the test suite label Apr 11, 2017
@thaJeztah
Copy link
Member

ping @mavenugo @vieux all green now; ready to merge?

@thaJeztah
Copy link
Member

adding impact/changelog, because #8795 has been a long standing issue, so worth a mention in the changelog

@mavenugo
Copy link
Contributor

@thaJeztah @vieux 👍

@thaJeztah
Copy link
Member

Thanks! I'll go ahead and merge 👍

@thaJeztah thaJeztah merged commit f30e94a into moby:master Apr 11, 2017
@fcrisciani fcrisciani deleted the conntrack_test branch April 11, 2017 15:06

// Launch the server, this will remain listening on an exposed port and reply to any request in a ping/pong fashion
cmd := "while true; do echo hello | nc -w 1 -lu 8080; done"
_, _, err := dockerCmdWithError("run", "-d", "--name", "server", "--net", "testbind", "-p", "8080:8080/udp", "appropriate/nc", "sh", "-c", cmd)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

appropriate/nc?

tisk tisk

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fcrisciani I think @seemethere was refering to the use of a 3rd party image, and it that case it was not updated in 2 years. could you replace it with a simple alpine for the official library ? it includes nc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vieux @seemethere Using that image is not accidental but is intended. The nc version in alpine does not support for example the option -q. There is a bug in nc that let the command never return also when specify the -w option.
Feel free to try to change it to the alpine removing the -q option to let the command work, but most likely the test will fail timing out because the nc command will hang forever.

thaJeztah pushed a commit that referenced this pull request Jun 3, 2022
There is a race condition between the local proxy and iptables rule
setting. When we have a lot of UDP traffic, the kernel will create
conntrack entries to the local proxy and will ignore the iptables
rules set after that.

Related to PR #32505. Fix #8795.

Signed-off-by: Vincent Bernat <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants