rootless podman network architecture diagram?

[iKloudly]

Hi everyone! First of all, thank you for clicking on this post to have interest in my question... I highly appreciate it.

I'm new to podman, and I've been looking into the usage of rootless podman for its security benefits.
I'm currently using rootless podman on RHEL9.2, with:

[vagrant@yellow test]$ podman --version
podman version 4.9.4-rhel

Podman Info

[vagrant@yellow ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 99.84
    systemPercent: 0.11
    userPercent: 0.04
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "9.2"
  eventLogger: journald
  freeLocks: 2034
  hostname: yellow.mls.local
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-284.11.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6222675968
  memTotal: 8061698048
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: runc
    package: runc-1.1.12-2.el9.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.21.7 (Red Hat 1.21.7-1.el9)
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 8388603904
  swapTotal: 8388603904
  uptime: 11h 35m 14.00s (Approximately 0.46 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/vagrant/.config/containers/storage.conf
  containerStore:
    number: 14
    paused: 0
    running: 14
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/vagrant/.local/share/containers/storage
  graphRootAllocated: 24403124224
  graphRootUsed: 809656320
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 21
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/vagrant/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1713184388
  BuiltTime: Mon Apr 15 12:33:08 2024
  GitCommit: ""
  GoVersion: go1.21.7 (Red Hat 1.21.7-1.el9)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

After some time of investigation, I realized that rootless podman uses a different approach for setting up and configuring networks, compared to rootful podman and rootful docker. It seems like the main point of entry is to utilize slirp4netns to create network namespaces, then connect the containers, etcs...

However, I'm not really familiar with the network stacks & how the flow works. To help me understand better, I tried to look for some architecture diagram out there, but I wasn't able to find any out there! (For rootless podman at least.. 😢)

So after doing some hands-on experiments, I realized that:

1. Each podman run calls without specifying --network, WILL spawn a new slirp4netns process with netns-.... from the process line.
2. If I do a podman run --network <new_network> to utilize a new user-defined bridged network, it summons a new slirp4netns with something like rootless-netns-.... from the process line.
3. Even if I deploy a new container on a different network stack, like podman run --network <different_network>, it still uses the previously existing slirp4netns process (the rootless-netns-... one), and does not spawn a new one.

So it makes me think "okay, then maybe by using user-defined networks, podman recognizes that it should be using a rootless namespace that's created by the slirp4netns process, and if I don't define any networks, podman believes that it should create a whole brand new network namespace, which is why it spawns a slirp4netns process for each runs.

Okay, I hope I understood correctly.
Now, I don't understand how the TAP device comes into play here. My slirp4netns process contains a tap0 at the end of the process line, but ALL the slirp4netns contains a tap0 only. So does a single slirp4netns process create a single TAP device?

I'm just not sure if it creates TAP devices for EACH NETWORKS, or if it just creates one TAP for all the rootless containers...
So I have something like:

which has a diagram of a single slirp4netns process, with a single TAP device (named tap0), connecting the bridges for two different networks.

OR, I have:

So although my ps aux does not show tap1, this was my assumption.. how it creates TAP devices for each podman network that we use pretty much.

Is any of these diagram correct? Am I understanding the network flow correctly?
Also, how does conmon come into play? Is it required to summon the slirp4netns process?

Any feedback would be REALLY, really highly appreciated.. I'll try to fix my diagram with it. Thank you so much in advance!

[iKloudly]

I've also read that podman consists of netavark, CNI, slirp4netns, and pasta as for its networkBackends;
However, I've read multiple posts on how rootless podman can currently only use slirp4netns and pasta, but not netavark or CNI. Is that correct as well?

(Considering I have slirp4netns processes popping up upon podman runs, I'm assuming I'm currently using slirp4netns as a default configuration.. right? haha)

[arizvisa]

Ftr, both slirp4netns and CNI are slated to be deprecated as I understand it (RHEL10?). The default_rootless_network_cmd option in your containers.conf(5) "should" be set to pasta(1) by default in the current version (if you're not using an old configuration on purpose).

There's some rudimentary network diagrams on the passt.top page, but it abstracts away podman, specifically, in favor of the more general description "network namespace".

To try and answer your question, (I might be wrong here as I've already made the transition from slirp4netns), the purpose of the tap interface is to simulate your network-stack in usermode. So, unless you're sharing that network between namespaces, they "should" be per-namespace. You can use lsof /dev/net/tun to see the pids/fds for each userspace process w/ a tun/tap device, and then /proc/$PID/fdinfo/$FD to see if it points to a specific interface. Hopefully that helps distinguish them, anyways.

[iKloudly]

Interesting! First of all, thank you so much for your response.

You can use lsof /dev/net/tun to see the pids/fds for each userspace process w/ a tun/tap device, and then /proc/$PID/fdinfo/$FD to see if it points to a specific interface.

That is awesome, thank you so much; I didn't know how to check for such things. I really appreciate it :)

And unfortunately, my work environment limits me to use a very specific version of RHEL9.2, and podman v4.9.4-rhel that comes with it. And from what I understand, it is currently using slirp4netns as default, with an option to switch to pasta.
And I've tried installing pasta, and configuring it to use pasta instead - which was successful.

I'm honestly not sure which network I'd prefer - but I do know that I definitely need some kind of an architecture diagram for it (which is why I tried to come up with those diagrams above for slirp4netns). But I'm very open to investigate into pasta as well, if that's a much recommended option to use (If so, I'd really appreciate some reasonings behind it...) I'd HIGHLY appreciate if you already have an architecture diagram for pasta's network with podman (if not, then I guess I'll try to create one, like I did with slirp4netns).

Again, I really, really appreciate your response. You're really so much helpful already :')

[sbrivio-rh]

But I'm very open to investigate into pasta as well, if that's a much recommended option to use (If so, I'd really appreciate some reasonings behind it...)

https://passt.top/#pasta

or, in more detail: https://2023.everythingopen.au/schedule/presentation/4/

I'd HIGHLY appreciate if you already have an architecture diagram for pasta's network with podman (if not, then I guess I'll try to create one, like I did with slirp4netns).

It's pretty much the same as what you have with slirp4netns, plus the "tap bypass" represented here: https://passt.top/#pasta-pack-a-subtle-tap-abstraction.

Or, in that diagram, replace "pod network namespace" with Podman. I know, "pod" might be a bit misleading there nowadays, we originally developed it for KubeVirt (where passt(1) connects guests in pods). A revamp of website and diagrams is long overdue (and somewhat in progress).

I'm just not sure if it creates TAP devices for EACH NETWORKS, or if it just creates one TAP for all the rootless containers...

If you just use podman run, without creating Podman custom networks (e.g. a bridged network), it will be one tap device per tap container. If you have a custom network, running as rootless, then there will be a single instance of pasta (or slirp4netns, it's conceptually the same) connecting that bridge and network namespace to the outside world, via tap device. What you have "inside" is typically a bridge of veth endpoints.

[iKloudly]

This has been extremely helpful... Thank you so much, I'm crying in gratefulness really :')
Understood! Thank you so much for the help once again! <3

[Luap99]

This is what I created for my devconf talk. Basically there is only one rootless-netns namesapce (podman unshare --rootless-netns) connected via pasta/slirp4netns and all the bridges veth are inside there.

[Luap99]

This is expected, as the diagram should show we are using rootless port to forward ports with custom networks which cannot keep the source ip. This issue is tacked in #8193

[d03j]

Thanks. I was aware it wasn't possible with port_handler=slirp4netns but was hoping it was with pasta.

So, what's the difference between doing podman unshare --rootless-netns; podman network create mynet vs simply podman podman network create mynet ?

I can not see any differences when I do a network inspect on networks created with each approach, or when I do an ip a inside container's using each.

[piotr-dobrogost]

The diagram shows port forwarder magically directly access containers bypassing all other depicted network subsystems which seems odd and hard to believe. Could you tell how this works in reality?

[Luap99]

Well that is exactly what is does. It bypasses the "rootless-netns", the rootlessport program is rather "stupid". It simply binds the the port on the host spawns a child process in the container netns and connects to the port there and then just proxies all the traffic between both processes (code used from rootlesskit builtin port forwarder). As such it is unable to preserve the source ip as the container just sees the connection made inside.
https://github.com/containers/podman/blob/main/cmd/rootlessport/main.go

[eriksjolund]

I can get to traefik through its exposed port and it finds whoami using the container name but whoami shows the traefik container's IP as the source IP. When I check traefik's logs - it too identifies its own IP as the source IP.

If you configure a container to use socket activation then the container will see the real source IP address.
Not all software support socket activation but luckily traefik 3.1.0 (and newer) has support for socket activation.
I wrote a demo of how to run docker.io/library/traefik with rootless Podman and use socket activation.
https://github.com/eriksjolund/podman-traefik-socket-activation/