Does not support containers running with random uid

[JtMotoX]

What steps did you take and what happened:

• Create a deployment/statefulset with a runAsUser set to a random uid (our admission/policy engine does this) and PVC defined.
• Create a backup
• The backup fails with unable to create config directory: mkdir /udmrepo: permission denied

The following information will help us better understand what's going on:

It seems to attempt to write to $HOME/udmrepo (link to function), but since the container user does not exist, $HOME is blank which results in /udmrepo which the user does not have write access.
You can replicate this with the following:

> CONTAINER_UID=$(( 10000 + RANDOM % 90000 )) && docker run --rm -it -u$CONTAINER_UID:$CONTAINER_UID debian sh -c 'id && echo $HOME && mkdir $HOME/udmrepo'

uid=28800 gid=28800 groups=28800
/
mkdir: cannot create directory '//udmrepo': Permission denied

Anything else you would like to add:

Would it be possible for us to specify a custom workPath directory (e.g. /tmp/udmrepo)?

Environment:

• Velero version (use velero version): v1.14.0
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): 1.25

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

This might be implemented as a extensive part of #7963, that is, we allow users to specify a customized dir.
On the other hand, we should be careful to deliver this. E.g., I don't think it is a good idea to use /tmp, the files there may be deleted by the OS in some occasions. And once the config files are deleted, there will be failures when accessing the repo.

This increases the risk of failures and the cost of troubleshooting. Therefore, we need a further consideration.

[JtMotoX]

For security purposes, we have designed all of our application images to be completely read-only by the application process/user. This is why we run the containers with a random uid. This ensures that all directories and files within the container are immutable and cannot be modified if an attack were to take place. The only directory that most of our applications have access to is the /tmp directory. There are some cases where applications need to store persistent data so they use PVC's which is the only other directory that the application has write access to (e.g. /persist). Correct me if I am wrong, but if velero is backing up the PVC, I don't think storing it under /persist/udmrepo would be a good idea. So, our only option is storing it in the volume (/persist), or in /tmp. Also, we run most of our nodes on spot instances so pod restarts happen quite frequently, probably more frequently than the container OS deleting files in the /tmp directory.

[Lyndon-Li]

if velero is backing up the PVC, I don't think storing it under /persist/udmrepo would be a good idea

Yes, backup tool should not change anything to the backup source. This is a basic principle.

[Lyndon-Li]

we run most of our nodes on spot instances so pod restarts happen quite frequently

If so, I think you will probably encounter further problems when you backup/restore data, since the backup/restore may last long time and if Velero server/node-agent pods are restarted before the backup/restore completes, the backup/restore will fail

[JtMotoX]

As a workaround, we have reinstalled velero using the --uploader-type restic parameter and our backups and restores are working now and contain all our PVC data. We would really prefer to use kopia for the performance benefits, but with the limitations of kopia, we will need to use restic.

[reasonerjt]

Given there's concern about putting it in /tmp, maybe we can add an emptyDir for storing the repo's config?