x/build: s390x builder not receiving/detecting new builds #27182
Comments
bradfitz
changed the title
bradfitz
added
the
NeedsInvestigation
|
I see 23 entries of linux-s390x-ibm under active builds. They're all started anywhere between 51 and 46 hours ago, and stuck on this step: That might (or might not) be related. |
@billotosyr @mundaym Can you try restarting it (the buildlet process)? Perhaps it's running but disconnected, and hasn't detected that. You can also try updating to the latest version (i.e., |
|
restarted buildlet process - doesn't seem to have had any effect. |
|
I'm pretty sure its not a version thing -- as the buildlet was working up untl a benign unrelated CL yesterday ("skip TestGcSys on Windows"). |
Great to hear that it's running again! Do you know if the machine ran out of space due to the Go builder work, or was it for unrelated reasons? About the queue, after talking to @bradfitz about it, I don't think there's an existing way to clear it out. Sorry about that. We have an open bug for a scheduler (#19178) and that would be helpful here (e.g., prioritize more recent commits). Let's keep an eye on how the builder runs, and I'll investigate what else can be done to help. |
|
Specifically, it's build cache that is filling up the drive. My current workaround is to use |
|
I thought the buildlet wiped its work directory environment per run? |
|
Our buildlet doesn't do that. Maybe because it's an older version. |
|
No, I was just misremembering. But if that helps your environment & wrapper process that's managing keeping it alive, we could add it to the buildlet. |
|
Well, I think it would help us, yes. The build cache is consuming about 2GB per day, and the drive it's on is small enough that I am now cleaning it daily. (This is a new thing, as I never had to clean it before.) |
|
Change https://golang.org/cl/144637 mentions this issue: |
|
Okay, this the new s390x buildlet binary is pushed. I'll watch it for any new problems. It'll keep itself cleaned for new stuff, but you might have to clean legacy messes. |
|
Thank you! There may be an issue though: |
|
Hmm, weird. I'll debug in a few hours. |
|
Change https://golang.org/cl/144858 mentions this issue: |
Fixes regression from CL 144637 for s390x and arm5 (the only two deployed with that change). Fixes golang/go#27182 Change-Id: I7a4a92c7c97f3f6c738d757a05b9dfc755547bf0 Reviewed-on: https://go-review.googlesource.com/c/144858 Reviewed-by: Dmitri Shuralyov <[email protected]>
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version)?master tip (1.12)
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (
go env)?GOARCH="s390x"
GOBIN=""
GOEXE=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/data/golang/go"
GOTOOLDIR="/data/golang/go/pkg/tool/linux_s390x"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build637051903=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
What did you do?
loaded the build dashboard
What did you expect to see?
s390x building with all the new CL's
What did you see instead?
Since yesterday (Aug. 22) it's been silent. The buildbot is still running normally (as far as I can tell) but no builds are happening. Suggestions?
The text was updated successfully, but these errors were encountered: