x/build/cmd/tip: add _tipstatus error monitoring to farmer.golang.org#health #15266
Labels
Builders
x/build issues (builders, bots, dashboards)
FeatureRequest
FrozenDueToAge
NeedsFix
The path to resolution is known, but the work has not been done.
Milestone
Comments
|
/cc @cnoellekb @bradfitz @andybons This enhancement is something we might want to take into account as part of moving the website to the x/website repository (issue #29206). A similar issue happened in #29874, where tip.golang.org was out of date for 13 days before someone noticed. |
|
Well, we could add monitoring for tip's /_status and send emails when it's angry. |
FiloSottile
changed the title
|
We have https://farmer.golang.org/#health now, which is a great place to add a check for this. However, we'll likely need to resolve #32949 first. |
dmitshur
added
the
NeedsFix
dmitshur
changed the title
dmitshur
changed the title
|
Change https://golang.org/cl/185139 mentions this issue: |
gopherbot
pushed a commit
to golang/build
that referenced
this issue
Jul 9, 2019
We want the error to represent an ongoing problem. It's less useful to constantly show the last error; we have logs for that. This will help make the _tipstatus output more readable for humans, and more friendly to being monitored by farmer.golang.org#health. Updates golang/go#15266 Fixes golang/go#32949 Change-Id: I20e5f180209c54b31e81e29bd45af0e2d205d3cc Reviewed-on: https://go-review.googlesource.com/c/build/+/185139 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Emmanuel Odeke <[email protected]>
|
Change https://golang.org/cl/185981 mentions this issue: |
codebien
pushed a commit
to codebien/build
that referenced
this issue
Nov 13, 2019
We want the error to represent an ongoing problem. It's less useful to constantly show the last error; we have logs for that. This will help make the _tipstatus output more readable for humans, and more friendly to being monitored by farmer.golang.org#health. Updates golang/go#15266 Fixes golang/go#32949 Change-Id: I20e5f180209c54b31e81e29bd45af0e2d205d3cc Reviewed-on: https://go-review.googlesource.com/c/build/+/185139 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Emmanuel Odeke <[email protected]>
codebien
pushed a commit
to codebien/build
that referenced
this issue
Nov 13, 2019
This change adds health monitoring for the tip.golang.org website to farmer.golang.org#health, so we can know when there's an issue with it without having to manually check it separately from every other item already monitored at farmer.golang.org#health. This is possible now that golang/go#32949 is resolved and the _tipstatus page reports currently ongoing errors only. Tested by running coordinator locally in dev mode. Fixes golang/go#15266 Updates golang/go#32949 Change-Id: I3680a916e4b45336df837c72ac7bc555d37f3a9b Reviewed-on: https://go-review.googlesource.com/c/build/+/185981 Run-TryBot: Dmitri Shuralyov <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Alexander Rakoczy <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
A recent change to cmd/go (5b3e576) broke cmd/tip, but we didn't realize it was broken because the active instances just kept serving the last working version. Eventually all four instances were restarted and they all failed to start godoc, being unable to find it. By that stage it was hard to understand what was going on.
It would be better if cmd/tip could somehow report that it has been wedged for a long time. Ideally before it stops serving properly.
An additional, more radical idea is for cmd/tip to—on startup—try earlier versions of go and tools to see if they work. That seems like a lot of fiddly nonsense though.
The text was updated successfully, but these errors were encountered: