podman logs fails in the face of log file rotation

[portante]

/kind bug

Description

podman logs -f ${_contid} does not work in the face of log rotation.

Steps to reproduce the issue:

$ _contid=$( podman run --log-opt max-size=10mb -d my-container )
$ podman logs -f ${_contid}

Describe the results you received:

podman logs -f ${_contid} stops following output of container when log file is rotated.

Describe the results you expected:

podman logs -f ${_contid} follows output of container even when log file is rotated.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.2.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: gandalf.bos.redhat.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.18-200.fc32.x86_64
  linkmode: dynamic
  memFree: 3976982528
  memTotal: 33562722304
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 4278480896
  swapTotal: 4294963200
  uptime: 882h 0m 24.6s (Approximately 36.75 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/pportant/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/pportant/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 124
  runRoot: /run/user/1000/containers
  volumePath: /home/pportant/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438263
  BuiltTime: Tue Dec  8 09:37:43 2020
  GitCommit: ""
  GoVersion: go1.14.10
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

@haircommander I would figure this is more of a conmon issue then a podman issue?

[haircommander]

well, sort of. when podman logs is following a file, and that file is removed and recreated underneath it, I think the stream will stop (it'll get EOF). If this happens, I suppose it should try to reopen the file. I actually think this is a podman problem

[portante]

A tail -F .../ctr.log works fine, shouldn't podman logs -f ${_contid} also work fine?

Given tail works, seems like not a conmon issue.

[rhatdan]

Ok so Podman is not talking to conmon when this happens but reading the file directly?

[rhatdan]

@baude PTAL,
This is your code, I believe, I think we just need to reopen the log file when we get the EOF. And continue reading until the container exits.