How is the bridge for multiacess links created ? #1324

DanPartelly · 2024-09-23T18:18:51Z

DanPartelly
Sep 23, 2024
Collaborator

As title says. I have to set net.bridge.bridge-nf-call-iptables=0 to be able to ping between devices on a multiaccess link. Is this the intended behavior ? Or should the firewalll process all bridge packets and be configured correctly ?

Thanks, Dan

jbemmel · 2024-09-23T18:42:18Z

jbemmel
Sep 23, 2024
Collaborator

For Containerlab it's done here:

netlab/netsim/providers/clab.py

Line 22 in 85421e3

def create_linux_bridge( brname: str ) -> bool:

We've not had complaints about this before afaik, but it could be that something similar to

status = external_commands.run_command(
      ['sudo','sh','-c',f'echo 65528 >/sys/class/net/{brname}/bridge/group_fwd_mask'],

is required on some platforms?

1 reply

DanPartelly Sep 23, 2024
Collaborator Author

i have to dig more into this. This is libvirt lab, with iosv devices on a multi-access link. The bridge created for this link is OK at layer 2, ARP works, CDP works, loop works. Didn't checked LLDP. Layer 3 does not.

Is there any bit in /bridge/group_fwd_mask which could influence IP unicast ? The fact that it works with net.bridge.bridge-nf-call-iptables=0 makes me think firewalling is playing a role.

ipspace · 2024-09-24T11:45:48Z

ipspace
Sep 24, 2024
Maintainer

Is this the same OpenSuse box we were discussing in #1313? If so, I can only give you vague pointers.

In the libvirt world, the networks (Linux bridges) are created by Vagrant (calling libvirt), and the only thing netlab does afterwards is to find out the underlying Linux bridge name and the change a few bits like the group_fwd_mask. You can see what's going on behind the scenes with '-v' (or '-vv') option of netlab up. However, to minimize the clutter I'd run netlab create first followed by netlab up --snapshot

This is a pretty good description of group_fwd_mask: https://interestingtraffic.nl/2017/11/21/an-oddly-specific-post-about-group_fwd_mask/ and this is the second hit of my Google search ;) https://blog.ipspace.net/2020/12/linux-bridge-lldp/ As you can see from that article, the group_fwd_mask affects only the IEEE group MAC addresses, not the IP traffic.

In you case, it looks like something created iptables that block traffic between the virtual machines -- the setting you mentioned disables the call to iptables for packets traversing a bridge (https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysctl.conf.html). FWIW, here are my net.bridge settings and as you can see the defaults work on Ubuntu:

~$ sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0

You could inspect the iptables before starting the lab and afterwards and see if there's any change. That will at least tell you whether any of the components used by netlab is the culprit (and no, netlab does not call iptables). iptables -L or iptables -S should do the trick, but the printouts are a bit cryptic.

Finally, we could disable iptables on bridges in netlab. Any thoughts on that one @jbemmel @ssasso @steffann?

0 replies

ipspace · 2024-09-24T11:58:17Z

ipspace
Sep 24, 2024
Maintainer

FWIW, here's my iptables after bringing up a topology that uses a libvirt isolated network (Linux bridge):

sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-N ts-forward
-N ts-input
-A INPUT -j ts-input
-A INPUT -j LIBVIRT_INP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -j ts-forward
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A LIBVIRT_FWI -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -d 192.168.121.0/24 -o libvirt-mgmt -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o libvirt-mgmt -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.121.0/24 -i libvirt-mgmt -j ACCEPT
-A LIBVIRT_FWO -i libvirt-mgmt -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr1 -o virbr1 -j ACCEPT
-A LIBVIRT_FWX -i libvirt-mgmt -o libvirt-mgmt -j ACCEPT
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i libvirt-mgmt -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i libvirt-mgmt -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i libvirt-mgmt -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i libvirt-mgmt -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o libvirt-mgmt -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o libvirt-mgmt -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o libvirt-mgmt -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o libvirt-mgmt -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xff0000
-A ts-forward -m mark --mark 0x40000/0xff0000 -j ACCEPT
-A ts-forward -s 100.64.0.0/10 -o tailscale0 -j DROP
-A ts-forward -o tailscale0 -j ACCEPT
-A ts-input -s 100.78.59.132/32 -i lo -j ACCEPT
-A ts-input -s 100.115.92.0/23 ! -i tailscale0 -j RETURN
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
-A ts-input -i tailscale0 -j ACCEPT
-A ts-input -p udp -m udp --dport 41641 -j ACCEPT

The magic happens in the LIBVIRT_FWX chain in which packets leaving through the same bridge they came in are accepted. There might be something in your iptables that drops those packets before LIBVIRT_FWX chain is called.

4 replies

DanPartelly Sep 24, 2024
Collaborator Author

Yeah, Ive read Robin's and your article concerning group_fwd_mask and associated link yesterday evening. I encountered the situation I mentioned on both Nixos and open suse. In addition Suse family uses nftables by a while and both systems come with default rulesets.

It was the firewall. I made netlab multiacccess links work on both. Of course p2p and management links worked flawlessly. I wonder on what distros ppl use netlab, statistically that they didnt encounter such issues in past years. The zoo has many animals and many, many different defaults.

ipspace Sep 24, 2024
Maintainer

It's nice to hear you made it work. It would be even better to have the solution(s) documented as caveats in the docs/install/linux.md document. Would you be willing to add them to that document and submit a pull request (I can prettify them later)?

As for "I wonder what other people use", it looks like it's mostly Ubuntu. The only other reported problems I remember were the crypto policies on AlmaLinux (https://netlab.tools/caveats/#ssh-access-to-cisco-iosv), and we have someone struggling to make WSL work (see #1318)

DanPartelly Sep 25, 2024
Collaborator Author

I have no problem doing that, except I hate to pass axe wielding solutions to general public. Like, remove those specific drop rules, then curse depreciation of nft backend for iptables in some OSes and docker , or just disable bridge processing by firewall globally (this might break some stuff). While this might be pragmatic, i think its better to spend some time understanding the problem and what a general pleasant solution might look like and work from here.

So I would spend the weekend looking more into this and update this thread to exchange ideas. how does that sound ?

ipspace Sep 25, 2024
Maintainer

I have no problem doing that, except I hate to pass axe wielding solutions to general public.

That's exactly like the --break-system-packages we're using in installation scripts ;) If you want to make something work in a throwaway VM... ;)

So I would spend the weekend looking more into this and update this thread to exchange ideas. how does that sound ?

That's great. Just don't spend too much time looking for a generic solution that might not exist.

DanPartelly · 2024-09-27T14:44:42Z

DanPartelly
Sep 27, 2024
Collaborator Author

The plot thickens. guys, does this look to you what it looks to me ? A prerouting chain drop due to an incorrect IP checksum.

2024/09/27 14:42:36 Attaching kprobes (via kprobe)...
1565 / 1565 [-----------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 907 p/s
2024/09/27 14:42:38 Attached (ignored 118)
2024/09/27 14:42:38 Listening for events..
SKB CPU PROCESS NETNS MARK/x IFACE PROTO MTU LEN TUPLE FUNC
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 vgif_r1_1:23 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) netif_receive_skb
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 vgif_r1_1:23 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) __netif_receive_skb
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 vgif_r1_1:23 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) __netif_receive_skb_one_core
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 vgif_r1_1:23 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) skb_pull_rcsum
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 vgif_r1_1:23 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) skb_ext_add
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) sock_wfree
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) nf_hook_slow
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) nf_ip_checksum
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) __skb_checksum_complete
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) sk_skb_reason_drop
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) skb_release_head_state
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) skb_release_data
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) skb_free_head
0xffff9bccc8551a00 12 ~-system-x8:5022 4026531840 0 virbr1:17 0x0800 1500 100 172.16.0.1:0->172.16.0.2:0(icmp) kfree_skbmem

5 replies

DanPartelly Sep 27, 2024
Collaborator Author

@ipspace the only good and simple way to deal with this, is to disable the distro specific firewall programs and do not let them load custom stuff with questionable mangle rules. Or fix the rules themselves but it really just doesn't worth for most people. If you are not an expert in them I swear to god,its easier to debug a linux firewall with EBPF tracing then the tools they themselves offer.

Netlab is run on workstations, you are already protected and inside your own network. Not worth to handle the bridge filtering sysctls, for one they may still break docker or kuber, god knows, and setting the flags on a single bridge, after lab start-up, with sudo ip link set dev br0 type bridge nf_call_iptables 0 didn't worked for me in this particular case.

If you still want me to write this in docs Ill doit. Axe the firewall.

jbemmel Sep 28, 2024
Collaborator

The plot thickens. guys, does this look to you what it looks to me ? A prerouting chain drop due to an incorrect IP checksum.

Could be due to checksum offload settings on a (virtual) NIC

ipspace Sep 28, 2024
Maintainer

@ipspace the only good and simple way to deal with this, is to disable the distro specific firewall programs and do not let them load custom stuff with questionable mangle rules.

Agreed.

Or fix the rules themselves but it really just doesn't worth for most people. If you are not an expert in them I swear to god,its easier to debug a linux firewall with EBPF tracing then the tools they themselves offer.

🤢

If you still want me to write this in docs Ill doit. Axe the firewall.

Please do.

DanPartelly Sep 28, 2024
Collaborator Author

Could be due to checksum offload settings on a (virtual) NIC

Could be. Thank you for signaling this, and for all your help in this thread. The tun interface has both RX / TX paths using sw checksums. The bridge port itself reports TX path as using hw offload (whatever that offload means for a SW interface. makes no sense to emulate this for linux bridge, even if it makes sense for a virtualized adapter like e1000). But Ive seen the kernel call stack , the packet is dropped in RX path in pre-routing hooks. , as soon as it hits the interface, so it doesn't even hit TX path. The tun interface has no checksum offload.

any further debugging would imply dumping skb at every call and analyze them / or using data access breakpoints on skb fields, and I have no desire and/or reason to do that. Nor it would bring anything useful to any of us at the moment.

DanPartelly Sep 28, 2024
Collaborator Author

🤢

There are a significant number of execution paths which can drop packets silently and no log or trace on firewall tooling would ever catch them.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How is the bridge for multiacess links created ? #1324

{{title}}

Replies: 4 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

How is the bridge for multiacess links created ? #1324

DanPartelly Sep 23, 2024 Collaborator

Replies: 4 comments · 10 replies

jbemmel Sep 23, 2024 Collaborator

DanPartelly Sep 23, 2024 Collaborator Author

ipspace Sep 24, 2024 Maintainer

ipspace Sep 24, 2024 Maintainer

DanPartelly Sep 24, 2024 Collaborator Author

ipspace Sep 24, 2024 Maintainer

DanPartelly Sep 25, 2024 Collaborator Author

ipspace Sep 25, 2024 Maintainer

DanPartelly Sep 27, 2024 Collaborator Author

DanPartelly Sep 27, 2024 Collaborator Author

jbemmel Sep 28, 2024 Collaborator

ipspace Sep 28, 2024 Maintainer

DanPartelly Sep 28, 2024 Collaborator Author

DanPartelly Sep 28, 2024 Collaborator Author

DanPartelly
Sep 23, 2024
Collaborator

Replies: 4 comments 10 replies

jbemmel
Sep 23, 2024
Collaborator

DanPartelly Sep 23, 2024
Collaborator Author

ipspace
Sep 24, 2024
Maintainer

ipspace
Sep 24, 2024
Maintainer

DanPartelly Sep 24, 2024
Collaborator Author

ipspace Sep 24, 2024
Maintainer

DanPartelly Sep 25, 2024
Collaborator Author

ipspace Sep 25, 2024
Maintainer

DanPartelly
Sep 27, 2024
Collaborator Author

DanPartelly Sep 27, 2024
Collaborator Author

jbemmel Sep 28, 2024
Collaborator

ipspace Sep 28, 2024
Maintainer

DanPartelly Sep 28, 2024
Collaborator Author

DanPartelly Sep 28, 2024
Collaborator Author