Only one SOURCE-VOLUME is supported by bind-mount-options in generated Kubernetes YAML files

[dnso86]

Issue Description

Only one SOURCE-VOLUME gets referenced within the bind-mount-options entry in the Kubernetes YAML files generated using podman kube generate, even though there might be several more defined for the individual containers within the pod.

Looking at the code, this seems to be by design. However I think this behaviour might worth a short explanation in the documentation at least, as it might not be obvious for all users.

Steps to reproduce the issue

podman pod create --name example
podman create --pod example --name container_1 --volume /tmp/ex_1:/tmp/1:Z quay.io/podman/hello
podman create --pod example --name container_2 --volume /tmp/ex_2:/tmp/2:Z quay.io/podman/hello
podman create --pod example --name container_3 --volume /tmp/ex_3:/tmp/3:Z quay.io/podman/hello

2. podman kube generate example

Describe the results you received

The metadata section looks like:

# ...
metadata:
  annotations:
    bind-mount-options: /tmp/ex_3:Z
# ...

Describe the results you expected

• /tmp/ex_1:Z
• /tmp/ex_2:Z

should have appeared somewhere as well. bind-mount-options metadata pertaining to these mounts seems to be lost.

podman info output

N/A

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ygalblum @umohnani8 PTAL

[ygalblum]

The root cause here is that Podman uses a single annotation key bind-mount-options with a value expected to refer to only one volume.

As a result, kube generate overrides the value for this key. There is some sorting code before the volumes are processed, so we can maybe infer the rule that will tell the configuration of which volume will stay last. But, I don't think this is something we can really document.

But, this issue doesn't end there. When parsing the annotations, kube play expects the value of the bind-mount-options annotation key to refer to only one volume.

So, as I see it, the issue should be fixed by changing the value of the bind-mount-options annotation key so that it can refer to multiple volumes. For example, <sourceName>:Z,<sourceName>:Z. The proposed structure maintains backward compatibility as it does not change the structure of a single source, while it allows its extension.

Once kube play supports reading the complex string, kube generate should be fixed to produce such values

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ygalblum Interested in opening a PR to support this change?

[ygalblum]

I've played with my proposal and found that it is not a good solution. The reason is that the length of the value of an annotation is limited to 63 characters. As a result, once I had two paths each longer than 29 characters, I was already above the limit (add :Z to each and the delimiting ,).

I then though about building separate annotation keys for each case. But, ended up with the same issue. The maximum length for the name part of the annotation key is again 63 (https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) while the maximal path length on Linux is 4096.

This made me realize that the current solution (regardless to this issue) needs to be reevaluated.

I would like, however, to discuss the issue at hand.

First, comes the question of what is the purpose of kube generate.

If the output is to be used on K8S, then AFAIK bind-mount-options is private to Podman and therefore not used by K8S. If this is the case, genarate already has a mechanism in place to remove its private annotations: https://github.com/containers/podman/blob/main/libpod/kube.go#L445. So, maybe it should also remove this annotation?

But, if the output is to be used on Podman (with or without using on K8S), then as I wrote before, we need to first figure out the solution for kube play and only then fix kube generate accordingly.

@rhatdan @umohnani8 @vrothberg WDYT?

[K1DV5]

If I'm not missing something, play kube already supports bind mount optios for individual volumes like:

volumeMounts:
    - mountPath: /mounted-path:z
      name: vol-name

So the annotation is completely unnecessary and can be safely removed.

[quaintdev]

This is still happening in version 5.2.2
Any suggestions or workaround for this?

[ygalblum]

@quaintdev please see @K1DV5's answer: #17761 (comment)

[quaintdev]

@quaintdev please see @K1DV5's answer: #17761 (comment)

This works. Thanks.