netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat' #24374

mispp · 2024-10-25T20:18:57Z

Issue Description

running a pod causes fails with an iptables6 error

netavark: unable to append rule '-j MARK  --set-xmark 0x2000/0x2000' to table 'nat'

root@core:/etc/containers/systemd# systemctl status forgejo
● forgejo.service - The sleep container
     Loaded: loaded (/etc/containers/systemd/forgejo.kube; generated)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Fri 2024-10-25 22:07:02 CEST; 9min ago
   Main PID: 1943 (conmon)
      Tasks: 1 (limit: 3970)
     Memory: 5.4M (peak: 24.5M)
        CPU: 464ms
     CGroup: /system.slice/forgejo.service
             └─1943 /usr/bin/conmon --api-version 1 -c 739b51ab8aad1294951a06a4ed70798cdd2c0f16c0a362c611f02357fb2d3e06 -u 739b51ab8aad1294951a06a4ed70798cdd2c0f16c0a362c611f02357fb2d3e06 -r /usr/bin/crun -b /var/lib/containers/storage/>

Oct 25 22:07:02 core forgejo[1932]: Containers:
Oct 25 22:07:02 core forgejo[1932]: e830c99d45dac1888caf57e911844c3c2d04626590be79318ea0e011100ecb50
Oct 25 22:07:02 core forgejo[1932]: a27ab24731bd283b22ebd20479af1d4e3571c2f0c58f3bf00aa6114185d1761b
Oct 25 22:07:02 core forgejo[1932]: starting container 2ef93257f5b1ac3fc2e57aebdf43dd7b4e90cd48a82ec33106abcb21d4f1459d: netavark: unable to append rule '-j MARK  --set-xmark 0x2000/0x2000' to table 'nat': code: 2, msg: Warning: Extensi>
Oct 25 22:07:02 core forgejo[1932]: ip6tables v1.8.10 (nf_tables): unknown option "--set-xmark"
Oct 25 22:07:02 core forgejo[1932]: Try `ip6tables -h' or 'ip6tables --help' for more information.
Oct 25 22:07:02 core forgejo[1932]: starting container a27ab24731bd283b22ebd20479af1d4e3571c2f0c58f3bf00aa6114185d1761b: a dependency of container a27ab24731bd283b22ebd20479af1d4e3571c2f0c58f3bf00aa6114185d1761b failed to start: contain>
Oct 25 22:07:02 core forgejo[1932]: starting container e830c99d45dac1888caf57e911844c3c2d04626590be79318ea0e011100ecb50: a dependency of container e830c99d45dac1888caf57e911844c3c2d04626590be79318ea0e011100ecb50 failed to start: contain>
Oct 25 22:07:02 core forgejo[1932]: Error: failed to start 3 containers
Oct 25 22:07:02 core systemd[1]: Started forgejo.service - The sleep container.

Steps to reproduce the issue

make files mentioned in additional information section of the issue in /etc/containers/systemd
systemctl daemon-reload
systemctl start forgejo

Describe the results you received

systemd service is runnning, but containers are not

Describe the results you expected

containers running

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 98.81
    systemPercent: 0.74
    userPercent: 0.45
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: server
    version: "40"
  eventLogger: journald
  freeLocks: 2035
  hostname: core
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.11.4-201.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1889120256
  memTotal: 3591000064
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.x86_64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 3590320128
  swapTotal: 3590320128
  uptime: 0h 17m 38.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 3
    stopped: 7
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 237908262912
  graphRootUsed: 3781160960
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.3
  Built: 1727136000
  BuiltTime: Tue Sep 24 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.3

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

bare metal server

Additional information

this doesnt work with 3 services: traefik, forgejo and vaultwarden.

config for forgejo

/etc/containers/systemd/forgejo.kube

[Unit]
Description=The sleep container
After=network.target

[Kube]
Yaml=forgejo.yaml
Network=systemd-traefik

[Install]
WantedBy=multi-user.target default.target

/etc/containers/systemd/forgejo.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    bind-mount-options: /var/run/podman/podman.sock:z
  creationTimestamp: "2024-02-18T16:02:00Z"
  labels:
    app: forgejo
    traefik.enable: true
    traefik.http.routers.forgejo.entrypoints: websecure
    traefik.http.routers.forgejo.rule: "Host(`git.mydomain`)"
    traefik.http.routers.forgejo.service: forgejo
    traefik.http.routers.forgejo.tls: true
    traefik.http.routers.forgejo.tls.certresolver: lets-encr-porkbun
    traefik.http.services.forgejo.loadbalancer.server.port: 3000
  name: forgejo-pod
spec:
  containers:
  - image: codeberg.org/forgejo/forgejo:8.0.3
    name: forgejo
    env:
    - name: FORGEJO__database__DB_TYPE
      value: postgres
    - name: FORGEJO__database__HOST
      value: localhost:5432
    - name: FORGEJO__database__NAME
      value: forgejo
    - name: FORGEJO__database__USER
      value: forgejo
    - name: FORGEJO__database__PASSWD
      value: x
    - name: MIN_PASSWORD_LENGTH
      value: 4
    volumeMounts:
    - mountPath: /data:z
      name: app-data-0
      readOnly: false
    - mountPath: /etc/localtime
      name: etc-localtime-2
      readOnly: true
  - image: postgres:latest
    name: forgejo-db
    env:
    - name: POSTGRES_USER
      value: forgejo
    - name: POSTGRES_PASSWORD
      value: x
    - name: POSTGRES_DB
      value: forgejo
    volumeMounts:
    - mountPath: /var/lib/postgresql/data:z
      name: db-data-1
      readOnly: false
    - mountPath: /etc/localtime
      name: etc-localtime-2
      readOnly: true
  restartPolicy: Always
  volumes:
  - hostPath:
      path: /data/forgejo/app
      type: Directory
    name: app-data-0
  - hostPath:
      path: /data/forgejo/db
      type: Directory
    name: db-data-1
  - hostPath:
      path: /etc/localtime
      type: File
    name: etc-localtime-2

/etc/containers/systemd/traefik.network

[Network]
IPv6=true

The text was updated successfully, but these errors were encountered:

mispp · 2024-10-25T20:22:04Z

i might add that 2weeks ago this was working find, not sure which update broke it since f40 running on that server is automatically updated overnight.

picsel2 · 2024-10-26T14:07:09Z

I have the same problem (interestingly also while using Forgejo and its agent) on Fedora IoT 40.

Downgrading to the previous deployment is a workaround for me. (I really start to love rpm-ostree based distros :D)
Here is the log output from downgrading. It shows the package versions involved:

Moving '63bb50518279ffcffd29c3a16cda92c973b71435ea2082f330d8a2a3559162b2.0' to be first deployment
Transaction complete; bootconfig swap: yes; bootversion: boot.1.1, deployment count change: 0
Downgraded:
  atheros-firmware 20241017-2.fc40 -> 20240909-1.fc40
  authselect 1.5.0-6.fc40 -> 1.5.0-5.fc40
  authselect-libs 1.5.0-6.fc40 -> 1.5.0-5.fc40
  brcmfmac-firmware 20241017-2.fc40 -> 20240909-1.fc40
  containers-common 5:0.60.4-2.fc40 -> 5:0.60.4-1.fc40
  containers-common-extra 5:0.60.4-2.fc40 -> 5:0.60.4-1.fc40
  crypto-policies 20241011-1.git5930b9a.fc40 -> 20240725-1.git28d3e2d.fc40
  device-mapper 1.02.199-1.fc40 -> 1.02.197-1.fc40
  device-mapper-event 1.02.199-1.fc40 -> 1.02.197-1.fc40
  device-mapper-event-libs 1.02.199-1.fc40 -> 1.02.197-1.fc40
  device-mapper-libs 1.02.199-1.fc40 -> 1.02.197-1.fc40
  firewalld 2.1.4-2.fc40 -> 2.1.3-1.fc40
  firewalld-filesystem 2.1.4-2.fc40 -> 2.1.3-1.fc40
  fwupd 1.9.26-1.fc40 -> 1.9.25-1.fc40
  fwupd-plugin-modem-manager 1.9.26-1.fc40 -> 1.9.25-1.fc40
  fwupd-plugin-uefi-capsule-data 1.9.26-1.fc40 -> 1.9.25-1.fc40
  iwlwifi-mvm-firmware 20241017-2.fc40 -> 20240909-1.fc40
  kernel 6.11.4-201.fc40 -> 6.11.3-200.fc40
  kernel-core 6.11.4-201.fc40 -> 6.11.3-200.fc40
  kernel-modules 6.11.4-201.fc40 -> 6.11.3-200.fc40
  kernel-modules-core 6.11.4-201.fc40 -> 6.11.3-200.fc40
  kernel-tools 6.11.4-201.fc40 -> 6.11.3-200.fc40
  kernel-tools-libs 6.11.4-201.fc40 -> 6.11.3-200.fc40
  libarchive 3.7.2-7.fc40 -> 3.7.2-4.fc40
  libjcat 0.2.2-1.fc40 -> 0.2.1-2.fc40
  libtirpc 1.3.6-0.fc40 -> 1.3.5-0.fc40
  libxmlb 0.3.21-1.fc40 -> 0.3.19-2.fc40
  linux-firmware 20241017-2.fc40 -> 20240909-1.fc40
  linux-firmware-whence 20241017-2.fc40 -> 20240909-1.fc40
  lvm2 2.03.25-1.fc40 -> 2.03.23-1.fc40
  lvm2-libs 2.03.25-1.fc40 -> 2.03.23-1.fc40
  python3 3.12.7-1.fc40 -> 3.12.6-1.fc40
  python3-firewall 2.1.4-2.fc40 -> 2.1.3-1.fc40
  python3-libs 3.12.7-1.fc40 -> 3.12.6-1.fc40
  realtek-firmware 20241017-2.fc40 -> 20240909-1.fc40
  shadow-utils 2:4.15.1-4.fc40 -> 2:4.15.1-3.fc40
  shadow-utils-subid 2:4.15.1-4.fc40 -> 2:4.15.1-3.fc40
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[sebastian@oldfriend ~]$ systemctl reboot

The broken revision is 40.20241023.0 of fedora-iot:fedora/stable/x86_64/iot .
The previous revision 40.20241017.2 is fine.

mispp · 2024-10-27T16:14:38Z

just stumbled upon the previously reported issue

kubernetes-sigs/kind#3765

it also links this bug

https://bugzilla.redhat.com/show_bug.cgi?id=2321325

Luap99 · 2024-10-28T10:42:01Z

Yes this looks like a kernel problem and not something we can fix.
You can try switching to native nftables like we do by default in f41 to see if this works around it, https://fedoraproject.org/wiki/Changes/NetavarkNftablesDefault#How_To_Test

The recent fedora kernel 6.11.4 has a problem with ipv6 networks [1]. This is not a podman bug at all but rather a kernel regession. I can reproduce the issue easily by running this test. Given many users were hit by this add it to the distro level gating which runs in the fedora openQA framework and then we should catch a bad kernel like this hopefully in the future and prevent it from going into stable. [1] containers#24374 Signed-off-by: Paul Holzinger <[email protected]>

The recent fedora kernel 6.11.4 has a problem with ipv6 networks [1]. This is not a podman bug at all but rather a kernel regression. I can reproduce the issue easily by running this test. Given many users were hit by this add it to the distro level gating which runs in the fedora openQA framework and then we should catch a bad kernel like this hopefully in the future and prevent it from going into stable. [1] containers#24374 Signed-off-by: Paul Holzinger <[email protected]>

mispp · 2024-10-28T11:24:13Z

Yes this looks like a kernel problem and not something we can fix. You can try switching to native nftables like we do by default in f41 to see if this works around it, https://fedoraproject.org/wiki/Changes/NetavarkNftablesDefault#How_To_Test

tried it, it works now.

thanks.

one difference though, i did it with making /etc/containers/containers.conf instead of using a .d dir.

mispp added the kind/bug Categorizes issue or PR as related to a bug. label Oct 25, 2024

Luap99 added network Networking related issue or feature and removed kind/bug Categorizes issue or PR as related to a bug. labels Oct 28, 2024

Luap99 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 28, 2024

Luap99 mentioned this issue Oct 28, 2024

test/system: add podman network reload test to distro gating #24389

Merged

This was referenced Oct 28, 2024

run cargo update containers/aardvark-dns#530

Merged

CI: e2e: pull/run and decrypt from local registry: timeout #24219

Closed

DO NOT MERGE: try new pasta and crun containers/automation_images#391

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat' #24374

netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat' #24374

mispp commented Oct 25, 2024 •

edited

Loading

mispp commented Oct 25, 2024

picsel2 commented Oct 26, 2024 •

edited

Loading

mispp commented Oct 27, 2024

Luap99 commented Oct 28, 2024

mispp commented Oct 28, 2024

netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat' #24374

netavark: unable to append rule '-j MARK --set-xmark 0x2000/0x2000' to table 'nat' #24374

Comments

mispp commented Oct 25, 2024 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

mispp commented Oct 25, 2024

picsel2 commented Oct 26, 2024 • edited Loading

mispp commented Oct 27, 2024

Luap99 commented Oct 28, 2024

mispp commented Oct 28, 2024

mispp commented Oct 25, 2024 •

edited

Loading

picsel2 commented Oct 26, 2024 •

edited

Loading