resolv.conf inside container does not get updated when it changes on host

[martinetd]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug
(or feature?)

Description

Running containers before the network is fully setup (e.g. LTE internet that takes a minute to setup), or just changing network (e.g. laptop moving from work to home network, or using a VPN's dns server and stopping the VPN) can lead to having an invalid /etc/resolv.conf inside containers.
Unfortunately, it doesn't look like the container keeps up with such changes -- the /etc/resolv.conf inside my containers keep whatever original value it found when the container was created, and doesn't seem to pick up new values.

Would updating it be possible? (ideas in additional information section)

Steps to reproduce the issue:

1. connect PC to a network with one dns server.
2. start containers, note the containers use the dns servers configured on host as expected
3. disconnect PC from that network, reconnect to other network. Note /etc/resolv.conf gets updated on host
4. Note /etc/resolv.conf hasn't changed in container

Describe the results you received:

DNS no longer works in containers

Describe the results you expected:

DNS keeps working.

Additional information you deem important (e.g. issue happens only occasionally):

Okay, first: the "obvious solution" to this is using something like systemd resolved or dnsmasq, with /etc/resolv.conf pointing to localhost -- that's what I have on my laptop and why I hadn't noticed the problem until now.
If a dns proxy/cache is used, the containers can keep using that local address and the local server will just use appropriate dns; which even allows split DNS, local dnssec validation and all the cool things.
I've been using dnsmasq with NetworkManager forever so I'd be willing to make that step, but would like to explore alternatives first (this might have been more appropriate as Q&A, sorry -- but I think whatever comes out of it is worth adding to documentation, if only "if you change networks use resolved or another local resolver" to the FAQ)

So, as far as I can see, /etc/resolv.conf is mounted from <storage>/overlay-containers/<id>/userdata/resolv.conf
Updating that file obviously works as expected, and it'd be easy enough to create a NetworkManager dispatcher script that updates these when DNS change. I don't see any such script anywhere, is it known to have been done before?

Are there "helpful wrappers" to manipulate dns like podman run --dns once a container is already started?

Output of podman version:

# podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18.2
Git Commit:   551cccc8e86503cde282f9b8e904e676bd559c2e
Built:        Thu May 12 09:00:55 2022
OS/Arch:      linux/arm64

Output of podman info --debug:

# podman info --debug
host:
  arch: arm64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-r1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: c0150c44db9bcbc1de391118753939cd1bf6b3c7'
  cpuUtilization:
    idlePercent: 99.59
    systemPercent: 0.1
    userPercent: 0.31
  cpus: 4
  distribution:
    distribution: alpine
    version: 3.16.0
  eventLogger: file
  hostname: armadillo
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.118-1-at
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1292308480
  memTotal: 1741070336
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.4.5-r0
    path: /usr/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-r0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 40m 18.46s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/containers/storage_readonly
    overlay.mountopt: nodev
  graphRoot: /run/containers/storage_root
  graphRootAllocated: 348217344
  graphRootUsed: 1134592
  graphStatus:
    Backing Filesystem: tmpfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /run/containers/storage_root/volumes
version:
  APIVersion: 4.1.0
  Built: 1652313655
  BuiltTime: Thu May 12 09:00:55 2022
  GitCommit: 551cccc8e86503cde282f9b8e904e676bd559c2e
  GoVersion: go1.18.2
  Os: linux
  OsArch: linux/arm64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

# apk info podman
podman-4.1.0-r1 description:
Simple management tool for pods, containers and images

podman-4.1.0-r1 webpage:
https://podman.io/

podman-4.1.0-r1 installed size:
32 MiB

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No and Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
headless computer board

[Luap99]

Podman is not a daemon, there is no way for podman to monitor /etc/resolv.conf and update all containers when it changes.

That said I understand the usecase. Currently a container restart should make it work again. However I understand that this is not desirable in most cases. Maybe podman can provide a way to only update the /etc/resolv.conf, something like podman network reload.

[martinetd]

Yes, I'm not thinking of having podman do it itself, but NetworkManager (and any other dhcp client) have hooks :)

With a podman network reload command it would become trivial to make a NetworkManager dispatcher script that just does that for all running containers, that is a good idea!

If it's just copying resolv.conf to container userdata as I pointed out though I'm not sure it's worth adding the bloat, I see we can get '{{.ResolvConfPath}}' from podman inspect so the script itself isn't that hard to make even without a command; I was mostly wondering if anyone had done it before. I'll be happy to submit such a network dispatched script to contrib/networkmanager/refresh_containers_dns.sh or something if that sounds like a good idea.

The only "problem" I can think of is since that is a bind mount we can't atomically replace resolv.conf, but have to write to it -- so if doing it normally there's a short window where resolv.conf will be empty in the container.
In another old issue someone suggested making it a bind mount to the userdata directory somewhere and have symlinks to files instead of individual bind mounts, but maybe there's another way of doing atomic updates in that case?

[Luap99]

I'll be happy to submit such a network dispatched script to contrib/networkmanager/refresh_containers_dns.sh or something if that sounds like a good idea.

Sounds good.

The only "problem" I can think of is since that is a bind mount we can't atomically replace resolv.conf, but have to write to it -- so if doing it normally there's a short window where resolv.conf will be empty in the container.

We already do this with network connect/disconnect, so I think we can also do it here.

[martinetd]

Thanks!
I'm sorry after having suggested I'd submit a script, but after internal discussion here we've decided to use a local resolver instead.

Making a simple dispatcher script that just overwrites dns with new ones whenever a configuration change happens is easy, but there doesn't seem to be any event that includes the former dns if e.g. you have this chain of events: 1/ ethernet link, 2/ vpn up with their dns, 3/ vpn down -- at this point you'd be left with the vpn dns which isn't good.

With a local resolver networkmanager will make things work as expected so it's just easier that way.

I'm closing this as my issue is then resolved, I wouldn't know if it's worth adding to the FAQ or some wiki page about networking as advice instead.

Thank you very much for being reactive anyway :)