Docs: clarify QUIC BPF operation

[moderation]

Title: Docs: clarify QUIC BPF operation

Description:
Prompted by the Twitter chat at https://twitter.com/mattklein123/status/1378172039870091265 I looked into the runtime flag that allows QUIC routing in the kernal via BPF - https://github.com/envoyproxy/envoy/blob/main/source/common/runtime/runtime_features.cc#L82

layered_runtime:                                                             
  layers:                                                                    
    - name: static-layer                                                     
      static_layer:                                                          
        envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true

Installing BPF rules like this requires one of:

1. Envoy is running as root
2. For kernels >= 5.8, Envoy is running with sudo setcap cap_bpf+ep <envoy binary>
3. For kernels < 5.8, Envoy is running with sudo cap_net_admin,cap_sys_admin+ep <envoy binary>

From initial testing Envoy doesn't display any different output when launched in different modes. QUIC / h3 listeners work whether Envoy was launched with the elevated permissions or not.

It would be good to clarify in the docs what steps need to be taken to enable QUIC BPF kernel routing and what platforms work and don't work. It looks like this is Linux only at the moment. It might be worthwhile logging whether the BPF rule has been installed successfully - https://github.com/envoyproxy/envoy/blob/main/source/common/quic/active_quic_listener.cc#L234-L298

Relevant Links:
Handy reference on determine what Linux Capabilities your system supports - https://linux-audit.com/linux-capabilities-101/

/cc @ggreenway @alyssawilk @danzh2010 @mattklein123

[mattklein123]

Yeah in addition to documentation, I agree we should have logging on which mode, and potentially even have it be configurable with error message if the kernel doesn't support it, not run as root, etc.

[alyssawilk]

I'll throw this Dan's way as part of the docs clean up for Alpha

[alyssawilk]

Actually @DavidSchinazi since you were looking for things to pick up, could you take on logging BPF success/failure to make it clear if it's working or not?

[alyssawilk]

I'm adding some of the above comments to my docs PR, and just tagging moderation as co-author. I think with that and a log message we can close this one off.

[DavidSchinazi]

Adding a log message when that function fails sounds good. @RenjieTang could you take this on please?

[DavidSchinazi]

/assign @RenjieTang

[RenjieTang]

When the BPF socket option is added, it will be applied in ActiveQuicListener constructor at https://github.com/envoyproxy/envoy/blob/main/source/common/quic/active_quic_listener.cc#L57, which calls down to SocketOptionImpl::setOption(), where a syscall return value is checked and warning logged.

So it looks like we are doing the correct logging? I'm confused on why these logs didn't show up in the output.

[alyssawilk]

So running as non-root
bazel-bin/source/exe/envoy-static --config-path configs/envoyproxy_io_proxy_http3_downstream.template.yaml -l warn

there's no warning becasue BPFwon't install with concurrency =1, so fine I can update docs and add an info log about that.

I'm still not seeing warnings or errors when I run with c=2 and I'm not running as root. I wonder if this is the casue of #15988 that we don't have permissions to install the filter but we're not getting an error somehow? Or maybe the assertions about root aren't correct?

[RenjieTang]

By running bazel-bin/source/exe/envoy-static --config-path configs/envoyproxy_io_proxy_http3_downstream.template.yaml -l warn --concurrency 2
I verified that setsocketop(SOL_SOCKET/SO_ATTACH_REUSEPORT_CBPF) is called and it returns no error.

[alyssawilk]

yeah, I think BPF works at least on our workstations without special permissions, so I'm going to update the docs PR to remove the "requires root" and just say "it tries, and may require root and will log warnings if it fails" and call it a day.