Unexpected error loading prebuild

[aledbf]

Bug description

rpc error: code = FailedPrecondition desc = cannot initialize workspace: prebuild initializer: Git fallback: git initializer gitClone: mkdir /dst/spring-petclinic: no such file or directory

Workspace affected

gitpodio-springpetclini-8a38a5a57eu

Expected behavior

1. Log when this happens (we don't now).
2. Ideally we'd wait longer until the file is ready, before trying to clone

Example repository

None

Anything else?

How long do we wait before the file system is ready?

[sagor999]

We already have log entry for this:

gitpod/components/content-service/pkg/initializer/git.go

Line 75 in 6390f20

log.WithError(err).WithField("location", ws.Location).Error("cannot create directory")

Also looking at logs, it looks like workspace failed, and then 5 minutes later ws-daemon tried to run initializer for it (???):

I cannot quite understand what exactly happened to that workspace lifecycle.

[kylos101]

Thanks for looking at this one, @sagor999 ! @jenting prior to resuming PVC work, could you peek at this to see what you can find? It's the last of broken windows we found from gen59 traces.

[jenting]

1. Log when this happens (we didn't for now).

We log it already

2. Ideally we'd wait longer until the file is ready, before trying to clone

We haven't run git clone; the error reports on os.MkdirAll(ws.Location, 0775) failed 🤔

---

I thought the error happens at this line

gitpod/components/content-service/pkg/initializer/git.go

Line 75 in 3d97d55

log.WithError(err).WithField("location", ws.Location).Error("cannot create directory")

However, I did not find the span isGitWS in the tracing

gitpod/components/content-service/pkg/initializer/git.go

Line 64 in 3d97d55

span.SetTag("isGitWS", isGitWS)

🤔

[jenting]

I'm blocked for this issue now.

I can't reproduce it locally to make the os.MkdirAll(ws.Location, 0775) failed with the reason no such file or directory. Is this because the mount point /dst/ is not ready yet?

[kylos101]

👋 @jenting were you able to find anything meaningful via Google searches, or recreate similar misbehavior in https://go.dev/play/? I ask so that we can have that context when sharing this issue with a teammate next week.

For now, let's leave blocked. And later this week I'll inspect frequency for this error. Frequency will determine if we reassign to another teammate while you're out on vacation, etc.

[jenting]

👋 @jenting were you able to find anything meaningful via Google searches, or recreate similar misbehavior in https://go.dev/play/? I ask so that we can have that context when sharing this issue with a teammate next week.

I did some google searching and write a similar code locally to reproduce the error. But no luck so far.

[jenting]

This is odd. From the log, the error comes from here.
However, I can't see this line warning log within GCP log.

Note: we filter by instanceId="a7ad0fe1-3ebc-4786-a207-366f8c7c1e47"

[kylos101]

@jenting are you still blocked and need help from the team (if yes please reach out in #t_workspace), or, do you have more info to go on now because of this thread?

[utam0k]

If this PR doesn't fix this problem, we have to write code to check if the container is still alive.
#12215

[sagor999]

This is related to this: #12282
If StopWorkspace was called while workspace was still doing content init, then it may fail with this exact error, as ws-daemon does not know that workspace was stopped and /dst has disappeared.

[utam0k]

This is related to this: #12282 If StopWorkspace was called while workspace was still doing content init, then it may fail with this exact error, as ws-daemon does not know that workspace was stopped and /dst has disappeared.

I just wonder who deletes $wsRoot/dst? kubelet or our component? Do you know?

[sagor999]

Could be that our housekeeping job in ws-daemon does that? 🤔

[jenting]

I just wonder who deletes $wsRoot/dst? kubelet or our component? Do you know?

I have the same question.

Since we are not sure whether #12282 addressed this issue or not, we might need to consider reopening this issue.

[utam0k]

There are probably two patterns in this issue

• Doing Initializing after the stop request StopWorkspace is called prior to regular workspace transitioning to running state #12282
• For some reason, the initializeing starts very late.

I put the log links of those two patterns in this PR. Perhaps this PR will improve both, but it is unclear if they will be resolved.
#12215

So, I think if it happens again, we should reopen.

Perhaps this PR will improve both, but it is unclear if they will be resolved.

[jenting]

We need to check the jaeger tracing on the gen63 cluster to see if it still happens or not.

[utam0k]

It still happens 😭
https://cloudlogging.app.goo.gl/4Hk68KGGpKBS1wyk9

[kylos101]

FYI, we need webapp to add logging, so that we can know "why" stop workspace is being called via #12282. Once that is done, then we can proceed with this particular issue.

[kylos101]

Added Blocked label, because we're waiting for webapp to schedule and do logging in #12282

[kylos101]

This is no longer blocked as of #12283

[atduarte]

During the refinement meeting, @utam0k mentioned he saw the error recently. We don't know exactly how to approach other than looking at the new logs and trying to understand what's going on. There's currently no hypothesis.

[kylos101]

@jenting @utam0k As this is Scheduled, and not In-progress, I removed you both from assigned. This way, it is "free" for later, when someone has bandwidth, then it can be assigned and status changed accordingly. 😄 Have a nice day you two! 👋

[utam0k]

@kylos101 💯 Thanks

[kylos101]

@sagor999 could you peek at the new logs, to see why the workspaces are stopping, to help form a plan of attack for this? I'm going to move this from Scheduled to the Inbox for now.

[sagor999]

Hm. I looked in traces (US) and looked in GCP logs for that error and cannot find one.
🤔

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[utam0k]

There are a few error messages so I closed