CI: podman run check dnsname plugin: Error freeing pod ... lock

[edsantiago]

Seen in CI for #8277:

Running: podman [options] network rm -f dnsNetTest
time="2020-11-09T09:22:48-06:00" level=error msg="Error freeing pod 3d01e6ef3e993362552426a069b353e8e2a12787d6622d844416ef02e1cd94e4 lock: no such file or directory"
Error: error freeing lock for container 7c40432117a6bc3104934fad3692e556195af9835df57ec36ca67c273131f89f: no such file or directory

This happened on the first try (of three), but the next two then failed because of improper cleanup ("network name dnsNetTest is already used").

Log: int podman fedora-33 rootless host

Will retry and see if it's a flake.

[mheon]

The initial error indicates that something else freed the pod's lock before it was fully removed, and could be an indication of a lock being held by two different objects, which is very bad because it could easily lead to deadlocks. Would be very interested in seeing if you can reproduce this.

[edsantiago]

Followup: it's a flake. Re-run made it pass.

I tried the following, let it run for an hour, and could not get it to fail:

$ ~/go/bin/ginkgo -trace -debug -progress -r -focus "run check dnsname plugin" -untilItFails test/e2e

Maybe it's something in the CI environment, or maybe it's another e2e test running in parallel, or maybe something else entirely. I have no idea what else to do to reproduce this, so I suspect in thirty days we'll get a stale-issue label and someone will close.

[edsantiago]

Well, well. Here we are again. int podman ubuntu-19 root host (First one was fedora-33 rootless).

[baude]

are you able to reproduce through repeated podman calls? repeated test runs?

[edsantiago]

No, see my earlier comment. cirrus-flake-summarize is reporting four failures in the last week, starting 11-05. The dnsname test was merged 10-31 via #8203. @Luap99 could you chime in here please?

• gce_instance:fedora : int podman fedora-33 rootless host
  • PR Reenable 'exec' test on remote #8277
    • 11-09 10:24
    • 11-09 10:24
    • 11-09 10:24
  • PR rootless container creation settings #8245
    • 11-05 14:09
    • 11-05 14:09
    • 11-05 14:09
• gce_instance:prior-fedora : int podman fedora-32 root container
  • PR Update CI tests to run python docker library against API #8236
    • 11-09 13:30
    • 11-09 13:30
    • 11-09 13:30
• gce_instance:prior-ubuntu : int podman ubuntu-19 root host
  • PR system tests: skip journald tests on RHEL8 #8284
    • 11-09 16:28
    • 11-09 16:28
    • 11-09 16:28

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago Still seeing this?
@Luap99 Any ideas?

[edsantiago]

Still happening, although not often:

Podman run networking [It] podman run check dnsname plugin

• gce_instance:fedora : int podman fedora-33 root host
  • PR Handle --rm when starting a container #8688
    • 12-11 06:51
• gce_instance:prior-fedora : int podman fedora-32 root container
  • PR remote copy #8747
    • 12-18 09:21
• gce_instance:ubuntu : int podman ubuntu-20 root host
  • PR Implement containers/{id or name}/archive api #8126
    • 11-19 17:06
    • 11-19 17:06
    • 11-19 17:06

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

No occurrences reported since my last comment (December 18).

[rhatdan]

Do you want to close or wait another month?

[edsantiago]

Well, what I'd really like is for @baude or @Luap99 or @mheon to jump in and say "oh, yeah, I committed PR#XYZ around then, and it fixed a something-something that was almost certainly the root cause of this issue". But the Rolling Stones have a song about not always getting what you want...

[mheon]

I still see no direct correlation between the test and the error, so I'm afraid I can't be helpful in that area

[baude]

im with @mheon

[rhatdan]

Well I am going to close and risk this triggering again and giving @edsantiago the ability to say I told you so.