Remove stargz from builder bob

[Furisto]

Description

Remove stargz from builder bob

Related Issue(s)

n.a.

How to test

n.a

Release Notes

Remove stargz snapshotter from image build

Documentation

[csweichel]

Can we move this along? We're still seeing stuck image builds in prod.

cc @Furisto @aledbf

[aledbf]

@csweichel I am not sure this will help. I prefer to finish the image builder transition to workspace clusters

[csweichel]

I prefer to finish the image builder transition to workspace clusters

Same here - that's still a bit out though.
@Furisto if I understood you correctly, a possible cause of this issue is the use of FUSE when building the image, no? If so, remove this and reverting later seems like a low-hanging way to sort this out.

[aledbf]

Can we reproduce the issue in a workspace cluster?

[Furisto]

Can we reproduce the issue in a workspace cluster?

Checking now...

[Furisto]

/werft run

👍 started the job as gitpod-build-fo-remove-stargz.1
(with .werft/ from main)

[aledbf]

@csweichel can we update the meta clusters without merging this change?

[csweichel]

@csweichel can we update the meta clusters without merging this change?

Good call.
@geropl wdyt?

[csweichel]

it would be a change to the registry-facade configmap which automatically reloads. No service restart is required.

[kylos101]

it would be a change to the registry-facade configmap which automatically reloads. No service restart is required.

Dear @csweichel , first, thank you so much for your suggestion and guidance with this. 💯

I looked at registry-facade, and could not find a reference to image-builder-bob. Do you mean the image-builder-mk3 configmap, and its builderImage property?

I assume yes, but, wanted to double check. It looks like this. "builderImage": "eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:5ba78ec619601c7c08f485a0d4db3d9303bc3444"

@geropl wdyt? This seems like a nice way to solve the problem where image builds are lingering in a Terminating status. Worst case, we'd have to do a rolling restart of image builder mk3, if image-build-mk3 lacks hot reload after a config map change.

Stuck builds in EU:

imagebuild-0c027fc6-5b8a-40fd-b669-534a62e3c746   0/1     Terminating   0          8h
imagebuild-34b06ba9-54bb-46e3-9bb6-4e891b5d0b8a   0/1     Terminating   0          11h
imagebuild-984b3310-11f2-4d10-94a5-4c128c854574   0/1     Terminating   0          8h
imagebuild-99166f14-7a0e-4a7b-8b36-85d36597237e   0/1     Terminating   0          7h13m
imagebuild-9a542d54-66b5-4f8b-baa9-3aabd480c7bf   0/1     Terminating   0          18h
imagebuild-dba39041-f3ed-4673-8dd2-8ecd375c0eb7   0/1     Terminating   0          17h

[csweichel]

I looked at registry-facade, and could not find a reference to image-builder-bob. Do you mean the image-builder-mk3 configmap, and its builderImage property?

🤦
My bad - it is in fact the image-builder config where we choose this value, not registry-facade.
Changes to that config require an image-builder restart.

[geropl]

@csweichel can we update the meta clusters without merging this change?

Good call.
@geropl wdyt?

@csweichel Sure, we can update right away.

It feels I'm missing some context, though: Since when do we see stuck image builds? Yesterdays deployment? 🤔

[csweichel]

Since when do we see stuck image builds?

Not entirely sure - but since Friday last week for sure

[geropl]

Not entirely sure - but since Friday last week for sure

👍

@csweichel @kylos101 Going to bump bob to: eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:9c0c6ae5a34dab7507fa94ab357a004868111fdc from the last green build on this branch

Update: Internal link for x-ref: https://gitpod.slack.com/archives/C021LT6GUJG/p1651053749792269

[aledbf]

This change is not enough. The daemon still uses --oci-worker-snapshotter=stargz (the source of the error)

[AlexTugarev]

@Furisto, what's the timeline on this fix? I'm asking, because unintentionally we did an undo of the workaround with the recent deployment of WebApp.

[Furisto]

@aledbf I have removed the stargz snapshotter from the daemon args as well.

[aledbf]

@Furisto I would still apply this change only in the meta clusters.

We need to review this after the migration of image builder to the workspace cluster.

[geropl]

@aledbf @Furisto Thank you both! So we're going to apply the bob image as last time to both prod app clusters, right?

Edit: waiting for this build

[aledbf]

@geropl yes please 🙏

[csweichel]

/werft run no-preview

👍 started the job as gitpod-build-fo-remove-stargz.4
(with .werft/ from main)