[frontend] "Failed to retrieve logs" in finished pipelines.

[drozzy]

Sometimes I click on old pipelines to check their logs and get the error "Failed to retrieve logs":

[ji-yaqi]

We automatically recycle the pods 24 hours after creation. You can try to re-run the pipeline to get logs. Please re-open this issue if you have any more questions.

[drozzy]

This was failing on microk8s install of kubeflow pipelines.
Once I switched to https://k3ai.github.io/docs/tutorials/kubeflow/ everything is great.

[danielschulz]

We also experience the very same issue, once our cluster Auto-Scales.

All logs are gone and NOT read from our mirror in MINIO (interface of KFP to access S3) / AWS S3 (physical location).

[AxoyTO]

Facing the same. Is there a way to increase or remove the 24h retention period? Logs disappear from the UI but they stay on the S3 bucket.