Deduplicate Iptables Rules with Dynamic ASG's

[klapkov]

Issue

We have noticed a strange case when we have Dynamic ASG's enabled. If there are two ASGs that define the same rules, and both are assigned to the same Space - in case of Static ASGs, when an application is deployed, in the iptables there will be only one rule for this ASG, and with Dynamic ASGs there will be two rules.

Steps to Reproduce

1. Enable dynamic ASG's
2. Create multiple security groups with the same rule
3. Bind them to a space and push an application
4. Ssh to the diego-cell that hosts the one of the app's instances.
5. Use iptables -L to list all the rules

Current result

We can see multiple identical rules

netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.207-10.0.25.207 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.207-10.0.25.207 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.206-10.0.25.206 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.206-10.0.25.206 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.213-10.0.25.213 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.213-10.0.25.213 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.212-10.0.25.212 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.212-10.0.25.212 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.211-10.0.25.211 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.211-10.0.25.211 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.207-10.0.25.207 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.207-10.0.25.207 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.206-10.0.25.206 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.206-10.0.25.206 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.213-10.0.25.213 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.213-10.0.25.213 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.212-10.0.25.212 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.212-10.0.25.212 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.211-10.0.25.211 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.211-10.0.25.211 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:http
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:https
netout--09cc0b12-2608-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:http

Expected result

Even though there are multiple ASG's defining our rule, there should be only one iptables entry for that rule.

Dump with the dedupe function added:

netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.213-10.0.25.213 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.213-10.0.25.213 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.212-10.0.25.212 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.212-10.0.25.212 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.211-10.0.25.211 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.211-10.0.25.211 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.210-10.0.25.210 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.209-10.0.25.209 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.208-10.0.25.208 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.207-10.0.25.207 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.207-10.0.25.207 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.206-10.0.25.206 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.25.206-10.0.25.206 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.2.32-10.0.2.63 tcp dpt:2222
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.2.32-10.0.2.63 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.2.32-10.0.2.63 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.1.32-10.0.1.63 tcp dpt:2222
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.1.32-10.0.1.63 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.1.32-10.0.1.63 tcp dpt:http
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.0.32-10.0.0.63 tcp dpt:2222
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.0.32-10.0.0.63 tcp dpt:https
netout--57942260-0d24-4--log  tcp  --  anywhere             anywhere            [goto]  destination IP range 10.0.0.32-10.0.0.63 tcp dpt:http

Possible Fix

#101

Additional Context

We introduced a temporary fix by using a patched up version of the iptables-restore binary, that removes duplicate rules when called. This solution has been running live for a few month and we have seen great results.

[ameowlia]

fix is released in https://github.com/cloudfoundry/silk-release/releases/tag/v3.40.0

Thank you all!