Replies: 17 comments
-
|
Usually a computer restart can fix this issue. |
Beta Was this translation helpful? Give feedback.
-
|
you can't restart a container that was created with Does podman show any error if it is left in a bad state and you try removing it? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I was not explaining correctly. I am not restarting the container. It's just that I have a script that's called postgresql-restart that does a The postgresql container seems gone when the problem occurs (not shown in My only solution is to restart the computer (or run my containers as another user) Probably |
Beta Was this translation helpful? Give feedback.
-
|
can you share the output of |
Beta Was this translation helpful? Give feedback.
-
|
I think I am encountering the same bug on Fedora 39, below my output. |
Beta Was this translation helpful? Give feedback.
-
|
that error is coming straight from the kernel and it happens when an error happened in the underlying file system: if (!ovl_should_sync(ofs)) {
ofs->errseq = errseq_sample(&upper_sb->s_wb_err);
if (errseq_check(&upper_sb->s_wb_err, ofs->errseq)) {
err = -EIO;
pr_err("Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.\n");
goto out_err;
}
}Does it make any difference if you run |
Beta Was this translation helpful? Give feedback.
-
|
I remember trying that but it did not help, but I am not 100% sure. Will try once more once it occurs again to be 100% sure (or someone else does it before me). |
Beta Was this translation helpful? Give feedback.
-
|
Not an FS expert, but I would also be surprised if sync really meant sync here, because if so, I would expect the kernel to do the sync automatically. |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure either, but from the code above it seems it is checking the upper layer superblock. I think this behavior is a consequence of this change torvalds/linux@335d3fc |
Beta Was this translation helpful? Give feedback.
-
|
@sargun any hint what it could cause this issue? Is it a pending error on the upper layer file system that was not checked yet? |
Beta Was this translation helpful? Give feedback.
-
|
This can happen if your upperdir has had filesystem errors. They should get cleared if you do sync (upperdir). If that doesn't fix it, can you give me a drgn dumb of the upperdir sb? |
Beta Was this translation helpful? Give feedback.
-
|
I have run |
Beta Was this translation helpful? Give feedback.
-
|
This just happened on my silverblue host (kernel-6.8.6-200.fc39.x86_64, podman-4.9.4-1.fc39.x86_64 and fuse-overlayfs-1.13-1.fc39.x86_64). I was running a heavy i/o workload on a bind mounted host volume, and it appears like this resulted in file corruptions: At that point, starting container always failed with that overlayfs error. Rebooting and removing the affected volume fixed the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Can you dump the superblock of the upperdir using drgn? |
Beta Was this translation helpful? Give feedback.
-
|
Let me see if it still exists. What is drgn? |
Beta Was this translation helpful? Give feedback.
-
|
Got this issue again, I was under the impression that to sync the filesystems, you just had to call I did it twice, getting an IO error the first time (in French, sorry) and not the second time. And hooray, podman works again |
Beta Was this translation helpful? Give feedback.
-
|
Moving to a discussion as this seems outside of the control of podman due some IO issues that we cannot do anything about |
Beta Was this translation helpful? Give feedback.
-
Issue Description
Using podman to run a postgresql server, the computer was under heavy load then the postgresql container crashed.
I tried restarting the postgresql container but I cannot:
At the same time, I see a kernel error in the system logs:
In partisular this kernel error looks suspect: overlayfs: Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.
Steps to reproduce the issue
Steps to reproduce the issue
Describe the results you received
Podman failed to recover and restart the container
Describe the results you expected
Podman shsould recover the bad state and start the container
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Additional environment details
Additional information
The container was removed prior to restarting it, any rootless
podman runfor the same user is failing.podman runas root or as other users continue to work.Beta Was this translation helpful? Give feedback.
All reactions