🐛 tilt: ensure .tiltbuild/bin directory is created early enough, add tilt troubleshooting guide

[bengentil]

What this PR does / why we need it:

The context + "/.tiltbuild/bin/" directory is created by a local_resource
but local_resources happens after local commands, including the one to create the Dockerfile for podman.

This PR ensures that the directory is present before writing the Dockerfile.
mkdir -p is idempotent so it's not an issue to call it twice for podman.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #9162

[k8s-ci-robot]

Hi @bengentil. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabriziopandini]

/ok-to-test
I don't have an environment to test, but the change seems ok
Also, it would be great to have tips&tricks to use podman (e.g #9162 (comment)) to surface in the CAPI book

[sbueringer]

I think I hit the same or a similar issue on a MacBook with Apple Silicon and Docker.

In my case I first started tilt and waited until everything was up. Then I added a new provider:

• Docker image build failed
• binary built then worked

I think the problem was that there is no dependency from the Docker image build to the binary build. Usually this is done in the right order because the Docker image build has to wait for the deployment of all CRDs etc. ("uncategorized"). But if this is not the case and the targets run in the wrong order the build will fail.

So I think this issue might not be podman specific but basically it can always occur when a provider is deployed for the first time and the binary was not build before the Docker image build was triggered

[sbueringer]

tl;dr. Maybe let's just add this mkdir for both cases and not only to Podman?

[bengentil]

Also, it would be great to have tips&tricks to use podman (e.g #9162 (comment)) to surface in the CAPI book

@fabriziopandini There is a documentation on how to use podman: https://cluster-api.sigs.k8s.io/developer/tilt.html#using-podman
but maybe a small troubleshoot guide or a summary like in the issue could be helpful yes 👍

@sbueringer This directory is required early in build_docker_image only for podman as a Dockerfile is written via a local call , docker receive the Dockerfile dynamically via the field dockerfile_contents when needed.

From my tests I saw that local are being executed immediately but local_resource (and other resources like docker_build) seems to be executed asynchronously (and re-called based on the dependencies)

What we could do is separate the local_resource in build_go_binary to have:

• 1 local to run the mkdir -p
• 1 local_resource to call the build command

So the directory will be always present before any other resources. This will not completely fix the issue you describe as if the docker_build is triggered before the build local_resource , the binaries will be missing.

Please advise, any solution is fine with me 😉

[sbueringer]

If I understand it correctly a local with mkdir would create the directory instantly, so this would be before local_resource and docker_build? (and then solve the issue completely)

[bengentil]

If I understand it correctly a local with mkdir would create the directory instantly, so this would be before local_resource and docker_build? (and then solve the issue completely)

Yes

[sbueringer]

I would go with that then

[bengentil]

@sbueringer I moved the mkdir to a local

@fabriziopandini I pushed a troubleshooting guide to explain how to fix all issues seen in #9162
let me know if you want me to push that in a separate PR

[fabriziopandini]

docs looks great, thanks
/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: aa8b989147fbd38685b28ced28c7b6b4941f30a9

[sbueringer]

Tested it with a MacBook with M2 Pro + Docker, worked as expected

Thank you very much!

Really nice troubleshooting guide!

[sbueringer]

/lgtm
/approve
/area devtools

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment