Add podman play kube --build support for podman remotes

[1player]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

podman play kube --build is not supported on remote podman. The documentation doesn't specify why, and I wasn't able to find any discussion of why exactly it isn't working, as podman build works on remote mode.

The use case is developing on immutable distributions like Fedora Silverblue. I'm running podman inside a toolbox, I can build images manually with podman --remote build, I would like podman play kube to be able to build as well, so I only need one command to local development.

Steps to reproduce the issue:

1. Set up a toolbox with access to the host podman socket
2. podman play kube --build

Describe the results you received:

Error: unknown flag: --build

Describe the results you expected:

Build to work on remote podman.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18.1
Git Commit:   e4b03902052294d4f342a185bb54702ed5bed8b1
Built:        Fri May  6 19:18:30 2022
OS/Arch:      linux/amd64

Server:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 17:15:54 2022
OS/Arch:      linux/amd64

[Luap99]

@baude PTAL

The problem with remote support is that remote build has to tar the full context dir and send it to the remote server.
Currently the client only sends the yaml content to the server and the server does all the work. To make remote build work the client would need to parse the yaml and then build the images. This is significant more complex.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc PTAL

[axel7083]

The problem with remote support is that remote build has to tar the full context dir and send it to the remote server.

Why is it a problem ? It is what podman build is doing isn't it ? One possibility could be to have --build default to false on remote machine, so the context-dir is not send if it is not explicitly required

[Luap99]

The client doesn't parse the yaml so it has no idea what is specified in the file as such it can never perform any builds.

You would need a major rewrite of the API in order to allow such things and this is not trivial.

[axel7083]

The client doesn't parse the yaml so it has no idea what is specified in the file as such it can never perform any builds.

@Luap99 After looking a bit at the code, I still think that the client does not need to parse the file.

build mechanism

Let's look at the build command and how it provide the context to the server. The client never parse the Containerfile, it creates a tar from the ContextDirectory (and other elements such as secrets).

podman/pkg/bindings/images/build.go

Line 438 in acc78af

tarContent := []string{options.ContextDirectory}

Then it provide in the POST request at /build the tar file as the body

podman/pkg/bindings/images/build.go

Line 590 in acc78af

response, err := conn.DoRequest(ctx, tarfile, http.MethodPost, "/build", params, headers)

On the server side, it will ensure the content body is x-tar.

podman/pkg/api/handlers/compat/images_build.go

Lines 42 to 43 in bd00c6f

	case "application/x-tar":
	break

Then untar, and uses it as the ContextDirectory

https://github.com/containers/podman/blob/bd00c6fef9b8dce1784fd531a8f3037eafb69008/pkg/api/handlers/compat/images_build.go#L74C27-L74C41

kube play mechanism

When using kube play according to the documentation, the default ContextDirectory will be the parent folder of the yaml file provided. Or the value defined by --context-dir option.

The body of the request kube play is not the context directory as it is not supported, but simply concat all the yaml resources

podman/pkg/bindings/kube/kube.go

Line 70 in 72f1617

body = io.NopCloser(bytes.NewReader(yamlBytes))

and provide them as body.

podman/pkg/bindings/kube/kube.go

Line 78 in 72f1617

response, err := conn.DoRequest(ctx, body, http.MethodPost, "/play/kube", params, header)

Question

My question is, why just like the build command, where the Containerfile is not parsed by the client, and the context is simply send to the server, the same could be done for the kube play

[Luap99]

From the docs:

Kube play is capable of building images on the fly given the correct directory layout and Containerfiles. This option is not available for remote clients, including Mac and Windows (excluding WSL2) machines, yet. Consider the following excerpt from a YAML file:

apiVersion: v1
kind: Pod
metadata:
...
spec:
  containers:
  - name: container
    image: foobar
...

If there is a directory named foobar in the current working directory with a file named Containerfile or Dockerfile, Podman kube play builds that image and name it foobar. An example directory structure for this example looks like:

|- mykubefiles
    |- myplayfile.yaml
    |- foobar
         |- Containerfile

The build considers foobar to be the context directory for the build NOT . If there is an image in local storage called foobar, the image is not built unless the --build flag is used. Use --build=false to completely disable builds.

---

So without reading the file you cannot know what the dir on the client because the context dir is based of the image name foobar. Also without reading the file we wouldn't even know how to name the final image.

[axel7083]

So without reading the file you cannot know what the dir on the client because the context dir is based of the image name foobar. Also without reading the file we wouldn't even know how to name the final image.

I do not see the problem of sending the entire directory where myplayfile.yaml is, and let the backend uses what it needs after parsing it, just like the build command is sending all the context-dir without knowing in advance what the Containerfile will used.

[Luap99]

Sending a (possible) large directory by default is very bad.

[axel7083]

Sending a (possible) large directory by default is very bad.

I understand the concerns, context directory can be very big on build too, but we are still sending them. IMO the sizing issue is the user responsibility: when building an image or a pod making a context directory that is okey to deal with is on the user part.

I still think that this would be manageable, and could be improve by having the build option to false by default. Having this feature working on more platforms by supporting the remote feel like it would have a lot of benefits, overlapping the few drawback already facing when using podman build.

[Luap99]

If you want to propose a specific way on how to do it then sure please do so here.

You can also join our cabal meetings to discus it there, next one on July 2 (https://hackmd.io/gQCfskDuRLm7iOsWgH2yrg?both). So feel free to add this topic there.

[l0rd]

This issue has been mentioned by this PR that tries to fix it.

The right way to solve this would be to package and send the images-named folders and context-dir folders (and that requires the client to parse YAMLs and the API to be adapted) but as mentioned by @Luap99 this is not trivial and since it's not a top priority it's unlikely that we are going to implement it in the short term.

But there may be a workaround solution, easy to implement, that would address the default podman machine cases.

Podman machine automatically mounts some host folders inside the VM (mainly /Users/ for Mac, C:/ for Windows and $HOME for Linux). And there are already some podman commands that map a local folder to a folder in the VM:
For example in the command podman build ... --build-context context1=C:\User\bob\my-project ..., the path C:\User\bob\my-project is automatically mapped to /mnt/c/User/bob/my-project (see function ConvertWinMountPath()). On MacOS and Linux the mapping is not required as the host and guest folders paths are identical.
The same mechanism is used for local volumes paths such as in podman run -v C:\User\bob\my-project where the volume path argument is mapped to /mnt/c/User/bob/my-project using the same function.

For podman kube play, instead of preventing builds from happening, we could implement a similar "best effort" solution. That means that on windows we should convert the current working path and the optional --context-dir to the corresponding path in the VM (using ConvertWinMountPath()) and, server side, we should always look for the Containerfiles in the image folders and build them when they exist.

This doesn’t solve all the use cases (i.e. remote hosts, non-default machine mounts, build contexts in non-mounted paths etc…) but is simple to implement and unblock the podman machine cases.

@fixomatic-ctrl would this address your use case?

[fixomatic-ctrl]

The right way to solve this would be to package and send the images and context-dir folders (and that requires the client to parse YAMLs and the API to be adapted)

I understand that this is a complicated task, and clearly not trivial, thanks for the explanation and details.

@fixomatic-ctrl would this address your use case?

This would partially address some of my use cases, but I would not be able to continue using podman from inside a dev-container, the solution provided would not fix the problem as it would not be able to find a valid path to provide the resources.

Given the details of the explanation for the The right way, I think the work can be divided in two tasks:

• The libpod api should support the tar Content-Type, same as the build but should still support text/json/yaml for backward compatiblity
• The remote client should support (when --build provided) building a tar with the minimum contexts required in it.

[l0rd]

This would partially address some of my use cases, but I would not be able to continue using podman from inside a dev-container, the solution provided would not fix the problem as it would not be able to find a valid path to provide the resources.

@fixomatic-ctrl please provide more details. My understanding is that you are running podman kube play from VS Code terminal from within a devcontainer. Podman running inside the container should be able to build the Containerfile that is mounted in the container. That should already work now. But I may have misunderstood your scenario.

[john-westcott-iv]

Our devs have a mixture of both OS X and Linux. It would be really nice for this feature to work.