docs: add checksum hacks, openwrt

docs/building.md

@@ -23,6 +23,14 @@ $ make build-cli build-tools build-kmod
 
 To build statically linked CLI and tools, specify `STATIC=1`.
 
+Additionally, [checksum hacks](checksum-hacks.md) can be specified:
+
+```console
+$ make CHECKSUM_HACK=kfunc  # This is the default
+$ make CHECKSUM_HACK=kprobe
+$ make CHECKSUM_HACK=kprobe STRIP_BTF_EXT=1
+```
+
 ## Building Distro Packages
 
 Arch Linux users can use `mimic-bpf` or `mimic-bpf-git` packages in AUR:

docs/checksum-hacks.md

@@ -0,0 +1,44 @@
+# Checksum Hacks
+
+The Linux kernel comes with support to [*checksum offload*](https://www.kernel.org/doc/html/v6.12/networking/checksum-offloads.html), which leaves calculations of [RFC 1071](https://datatracker.ietf.org/doc/html/rfc1071) checksum in TCP, UDP and more (and also CRC for SCTP) to the NIC if possible, reducing CPU usage. This is done using three fields inside [`struct sk_buff`](https://github.com/torvalds/linux/blob/b86545e02e8c22fb89218f29d381fa8e8b91d815/include/linux/skbuff.h#L867):
+
+- `ip_summed`, if set to `CSUM_PARTIAL`, will make the `skb` do the offloading;
+- `csum_start` points to the start of the L4 packet to be summed; and
+- `csum_offset` points to the 16-bit checksum field where it should be written.
+
+(Encapsulations, as well as RX checksum offload are not discussed here for the sake of simplicity. We only focus on TX checksum offload.)
+
+This leaves us a problem: when Mimic transforms a UDP packet to fake TCP one, `csum_start` and `csum_offset` cannot be changed in eBPF alone; there is simply no such methods of doing that. This will make the NIC (or whatever that finishes the checksum) put the checksum in the wrong place and makes the packet malformed.
+
+To solve this issue, we have to extend eBPF on the kernel side, and we also have to maintain compatibility across kernel versions. Mimic implements two kinds of **checksum hacks** implemented in its kernel module:
+
+- `kfunc` (default): using [BPF kernel functions](https://www.kernel.org/doc/html/v6.12/bpf/kfuncs.html) to extend eBPF. This is the current way of extending eBPF in-tree, and requires BPF JIT (`CONFIG_BPF_JIT=y`) and BTF (`CONFIG_DEBUG_INFO_BTF=y`) support.
+
+- `kprobe`: change existing BPF helpers' behaviour using kprobe. This is way hackier, but could be used when kernel BTF is not present (with BPF program's own BTF information also stripped). It also allows the kernel module to be optional, since [not every case requires checksum hack](#when-is-checksum-hack-not-necessary). Pass `CHECKSUM_HACK=kprobe` to `make` to enable this behaviour (you would almost certainly need `STRIP_BTF_EXT=1` too).
+
+Normally with most Linux distros (Debian, Fedora, Arch, etc.), `kfunc` should be used. But in restricted environments like OpenWrt, `kprobe` might be the option.
+
+## When is Checksum Hack Not Necessary?
+
+When the two conditions below both matches, checksum hack is not necessary.
+
+1. Driver does not use `csum_offset`, or checksum offload is maunally disabled.
+
+    You can check if your NIC's driver source code contains `csum_offset` by simple searching either inside Linux kernel source code or out-of-tree somewhere. Realtek and Mediatek's Ethernet driver does not use it, while Intel and many others uses it.
+
+2. UDP socket in userspace.
+
+    Kernel `udptunnel{4,6}`, used by many in-kernel tunnels such as [WireGuard](https://www.wireguard.com/), and encapsulation protocols like [FOU](https://www.man7.org/linux/man-pages/man8/ip-fou.8.html) and [GENEVE](https://github.com/torvalds/linux/blob/master/drivers/net/geneve.c), is always `CSUM_PARTIAL` (see [udp4](https://github.com/torvalds/linux/blob/b86545e02e8c22fb89218f29d381fa8e8b91d815/net/ipv4/udp.c#L1041) and [udp6](https://github.com/torvalds/linux/blob/b86545e02e8c22fb89218f29d381fa8e8b91d815/net/ipv6/ip6_checksum.c#L115)). Userspace implementations are not affected, like [wireguard-go](https://git.zx2c4.com/wireguard-go), [OpenVPN](https://openvpn.net/) and [Hysteria](https://hysteria.network/). (However, since Hysteria uses (modified) QUIC, it is sometimes better to keep it as-is than turning it to something unknown to the middleboxes.)
+
+## Platform Support
+
+`kfunc` hack requires `CONFIG_BPF_JIT=y` and `CONFIG_DEBUG_INFO_BTF=y` to be set. Below lists kernel support for BPF JIT and kfunc call per CPU architecture:
+
+- x86_64, aarch64: >= v6.1
+- riscv64: [>= v6.4](https://github.com/torvalds/linux/commit/d40c3847b485acc3522b62b020f77dcd38ca357f)
+- i386: JIT will fail with `** NOT YET **: opcode 85` in kernel output
+  - This means [eBPF on i386 does not implement BPF_PSEUDO_CALL](https://github.com/torvalds/linux/blob/786c8248dbd33a5a7a07f7c6e55a7bfc68d2ca48/arch/x86/net/bpf_jit_comp32.c#L2092)
+- arm: Does not support kfunc call
+  - There are [patches](https://lwn.net/ml/linux-kernel/[email protected]/) to implement this, but not merged
+
+`kprobe` hack requires `CONFIG_KRETPROBE=1`, and any Linux version >= 6.1 will work.

docs/getting-started.md

@@ -1,21 +1,12 @@
 # Getting Started
 
-## Kernel Support
+## Platform Support
 
-To run Mimic, you need a fairly recent Linux kernel (>= 6.1, due to usage of [dynptrs](https://lwn.net/Articles/910873/)), compiled with BPF (`CONFIG_BPF_SYSCALL=y`), BPF JIT (`CONFIG_BPF_JIT=y`), and BTF (`CONFIG_DEBUG_INFO_BTF=y`) support. Most distros enable them on 64-bit systems by default. On single-board computers with custom kernel, recompilation with these options enabled is probably needed.
+To run Mimic, you need a fairly recent Linux kernel (>= 6.1, due to usage of [dynptrs](https://lwn.net/Articles/910873/)), compiled with basic BPF support. For more detail, see [checksum hacks](checksum-hacks#platform-support).
 
-To check if current kernel have these options enabled, run `` grep CONFIG_[...] /boot/config-`uname -r` `` or `zgrep CONFIG_[...] /proc/config.gz`, where `CONFIG_[...]` is one of the kernel configurations above.
+You will not need to worry too much about it; most desktop and server Linux distros enable these configurations by default and should work out of the box.
 
-BPF support varies depending on architecture, kernel version and distro configurations. Be ready if installed kernel won't load eBPF programs JITed even when enabled, or JIT does not support calling kernel function.
-
-### Platform Support
-
-- x86_64, aarch64: >= v6.1
-- riscv64: [>= v6.4](https://github.com/torvalds/linux/commit/d40c3847b485acc3522b62b020f77dcd38ca357f)
-- i386: JIT will fail with `** NOT YET **: opcode 85` in kernel output
-  - This means [eBPF on i386 does not implement BPF_PSEUDO_CALL](https://github.com/torvalds/linux/blob/786c8248dbd33a5a7a07f7c6e55a7bfc68d2ca48/arch/x86/net/bpf_jit_comp32.c#L2092)
-- arm: Does not support kfunc call
-  - There are [patches](https://lwn.net/ml/linux-kernel/[email protected]/) to implement this, but not merged
+OpenWrt support is on the way, though one might need to build the entire image; see [here](openwrt.md).
 
 ## Running Mimic
 
@@ -51,7 +42,7 @@ Optionally, load Mimic kernel module at startup:
 
 If you are building from source, load the compiled kernel module manually:
 
-```
+```console
 # insmod out/mimic.ko
 ```
 
@@ -70,4 +61,3 @@ See [mimic(1)](mimic.1.md) for more information on command-line options and deta
 Due to its transparent nature (i.e. UDP applications can work seamlessly with or without Mimic running), Mimic plays nice with existing firewall rules too.
 
 However, do note that since both TC happens after netfilter's output hook, and XDP before input hook, one should *treat traffic through Mimic as UDP instead of TCP*. TCP rules have no effect on Mimic's fake TCP traffic.
-

docs/openwrt.md

@@ -0,0 +1,7 @@
+# OpenWrt Support
+
+We are trying to bring Mimic to OpenWrt. The packaging effort is going on in the [`openwrt` branch](https://github.com/hack3ric/mimic/tree/openwrt). However, OpenWrt does not come with both kernel BTF and kprobe configuration enabled, and [checksum hacks](checksum-hacks.md) cannot be applied without recompiling the kernel.
+
+As some of the scenarios [does not require checksum hack](checksum-hacks.md#when-is-checksum-hack-not-necessary), we build Mimic on OpenWrt with `CHECKSUM_HACK=kprobe`. You can use Mimic normally without the kernel module, but if you do need the checksum hacks, you have to build the OpenWrt image manually.
+
+*TODO: image build tutorial*