manifest unknown errors during pulls (seemingly after GC)

[nmcostello]

Problem

We turned GC on over this past weekend after having paused it for a bit due to other issues. When we returned to work this week, we began to see some images fail to pull with manifest or blob unknown errors.

$ podman pull myharbor.com/project/core/image:v3.2.231
WARN[0000] Failed, retrying in 1s ... (1/3). Error: parsing image configuration: fetching blob: blob unknown to registry 
WARN[0002] Failed, retrying in 1s ... (2/3). Error: parsing image configuration: fetching blob: blob unknown to registry 
WARN[0004] Failed, retrying in 1s ... (3/3). Error: parsing image configuration: fetching blob: blob unknown to registry 
Error: parsing image configuration: fetching blob: blob unknown to registry

After doing some debugging I can find the affected blob is: sha256:93f54f...0f. When I dive into the harbor UI this error pops up when I navigate into the previously mentioned artifact.

http status code: 404, body: {"errors":[{"code":"BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:93f54f02b5c84a9d626ad1d34fe8df11f50cde17c237e3c877e0a055b979550f"}]}

I went into the database and ran the query select * from artifact_blob where digest_blob like '%93f54f%', which returns:

id	digest_af	digest_blob	creation_time
6051788	sha256:96e655...52	sha256:93f54f...0f	2024-01-12 22:52:33.087429

Then when I check the path in our S3 bucket, that sha is not found, which explains the blob unknown error.

Questions

1. Could we go into our old s3 backups and copy the affected files and manually upload them back to the bucket?
2. What could have caused legitimate blobs to be deleted? Can we prevent this in the future?

[stonezdj]

1. Yes
2. Root cause is unknown, maybe related to the S3 storage, is it an AWS S3 or S3 compatible storage?

[nmcostello]

We are using AWS S3 for our blob storage. I am actively looking through our logs to see if there is anything that we can glean from them.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[Vad1mo]

@nmcostello, can you check if your issue is related to #20547?

[nmcostello]

@Vad1mo The issue appears connected. We observed the same behaviors. I can't confirm or deny that specific error code. We were running v2.7.x at the time.

edit: it happened long enough ago that I don't think we have logs from the event, but let me know if there is anything the team can provide you to help debug.