x/build: shard tests over N VMs to speed all.bash #10029
Labels
Milestone
Comments
bradfitz
added a commit
that referenced
this issue
Mar 3, 2015
This will enable test sharding over multiple VMs, to speed trybot answers. Update #10029 Change-Id: Ie277c6459bc38005e4d6af14d22effeaa0a4667e Reviewed-on: https://go-review.googlesource.com/6531 Reviewed-by: Russ Cox <[email protected]> Reviewed-by: Alex Brainman <[email protected]>
rsc
changed the title
bradfitz
added a commit
to golang/build
that referenced
this issue
May 14, 2015
Make all-compiler a trybot-only builder. This will be used for temporary sharded ARM trybots. Updates golang/go#10029 Change-Id: I6cc3188887c2f24edef97d0a9354e39553a15896 Reviewed-on: https://go-review.googlesource.com/10051 Reviewed-by: Andrew Gerrand <[email protected]>
bradfitz
added a commit
that referenced
this issue
May 14, 2015
For upcoming sharded ARM builders. Updates #10029 Change-Id: I3b1df9560be697c514a8ced0462814d406e23132 Reviewed-on: https://go-review.googlesource.com/10055 Reviewed-by: Ian Lance Taylor <[email protected]> Run-TryBot: Brad Fitzpatrick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>
|
CL https://golang.org/cl/10056 mentions this issue. |
bradfitz
added a commit
to golang/build
that referenced
this issue
May 14, 2015
This is a quick & hacky version of our eventual sharding plan. Later we want to run make.bash once, snapshot it, and then intelligently schedule all the tests out over N machines (starting with the built snapshot), giving each the next test to run as they become idle. And then we want to stitch together all of their output. But this CL just makes 6 ARM builders, each of which runs make.bash (4 minutes) and then each which runs a statically-configured subset of the tests. Based on measurements, each should run between 8.2 and 10.6 minutes. We can break up some of the larger test cases later into more shardable pieces (in particular: "test"). Update golang/go#10029 Change-Id: Ib3e008b61ce6f28d799ae298c8467bf62d89e5d4 Reviewed-on: https://go-review.googlesource.com/10056 Reviewed-by: Andrew Gerrand <[email protected]>
bradfitz
added a commit
that referenced
this issue
Jun 3, 2015
…fter test
Otherwise subsequent tests won't see any modified GOROOT.
With this CL I can move my GOROOT, set GOROOT to the new location, and
the runtime tests pass. Previously the crash_tests would instead look
for the GOROOT baked into the binary, instead of the env var:
--- FAIL: TestGcSys (0.01s)
crash_test.go:92: building source: exit status 2
go: cannot find GOROOT directory: /home/bradfitz/go
--- FAIL: TestGCFairness (0.01s)
crash_test.go:92: building source: exit status 2
go: cannot find GOROOT directory: /home/bradfitz/go
--- FAIL: TestGdbPython (0.07s)
runtime-gdb_test.go:64: building source exit status 2
go: cannot find GOROOT directory: /home/bradfitz/go
--- FAIL: TestLargeStringConcat (0.01s)
crash_test.go:92: building source: exit status 2
go: cannot find GOROOT directory: /home/bradfitz/go
Update #10029
Change-Id: If91be0f04d3acdcf39a9e773a4e7905a446bc477
Reviewed-on: https://go-review.googlesource.com/10685
Reviewed-by: Andrew Gerrand <[email protected]>
Run-TryBot: Brad Fitzpatrick <[email protected]>
|
CL https://golang.org/cl/10688 mentions this issue. |
bradfitz
added a commit
that referenced
this issue
Jun 4, 2015
Also modifies 'dist test' to use that sharding, and removes some old temporary stuff from dist test which are no longer required. 'dist test' now also supports running a list of tests given in arguments, mutually exclusive with the existing -run=REGEXP flag. The hacky fast paths for avoiding the 1 second "go list" latency are now removed and only apply to the case where partial tests are run via args, instead of regex. The build coordinator will use both styles for awhile. (the statically-sharded ARM builders on scaleway will continue to use regexps, but the dynamically-shared builders on GCE will use the list of tests) Updates #10029 Change-Id: I557800a54dfa6f3b5100ef4c26fe397ba5189813 Reviewed-on: https://go-review.googlesource.com/10688 Reviewed-by: Andrew Gerrand <[email protected]> Run-TryBot: Brad Fitzpatrick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>
|
This was done some time ago. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
As part of the rewrite of the coordinator and building subrepos (#8646 and #9506), we also want the coordinator to use N VMs to do an all.{bash,bat,rc} instead of 1, and then shard test execution over a bunch of VMs, to speed things up.
First step is to rewrite the bottom half of all.bash into Go and ditch the three variants (bash, bat, rc).
/cc @adg
The text was updated successfully, but these errors were encountered: