Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: all.bash takes far too long #10571

Closed
rsc opened this issue Apr 24, 2015 · 17 comments
Closed

build: all.bash takes far too long #10571

rsc opened this issue Apr 24, 2015 · 17 comments
Labels
FrozenDueToAge
Milestone
Unplanned

Comments

@rsc
Copy link
Contributor

rsc commented Apr 24, 2015

all.bash is taking too long. There are many reasons. This is a general tracking bug for work to fix this.
@rsc rsc self-assigned this Apr 24, 2015
@rsc rsc added this to the Go1.5 milestone Apr 24, 2015
@josharian
Copy link
Contributor

One regression: The os/exec race detector test in Go 1.4.2 went from ~0.6s on my machine to ~17.5s with tip. all.bash went from ~2.5m to ~6m, so this accounts for about 15% of the regression.

/cc @dvyukov

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/9350 mentions this issue.

josharian added a commit that referenced this issue Apr 27, 2015
@josharian
cmd/dist: consolidate runtime CPU tests
a9e50a6 
Instead of running:

go test -short runtime -cpu=1
go test -short runtime -cpu=2
go test -short runtime -cpu=4

Run just:

go test -short runtime -cpu=1,2,4

This is a return to the Go 1.4.2 behavior.

We lose incremental display of progress and
per-cpu timing information, but we don't have
to recompile and relink the runtime test,
which is slow.

This cuts about 10s off all.bash.

Updates #10571.

Change-Id: I6e8c7149780d47439f8bcfa888e6efc84290c60a
Reviewed-on: https://go-review.googlesource.com/9350
Reviewed-by: Dave Cheney <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Reviewed-by: Dmitry Vyukov <[email protected]>
This was referenced May 6, 2015
Closed
Closed
Closed
@robpike robpike mentioned this issue May 10, 2015
Closed
@gopherbot
Copy link
Contributor

CL https://golang.org/cl/9913 mentions this issue.

mk0x9 pushed a commit to mk0x9/go that referenced this issue May 11, 2015
@mwhudson @ianlancetaylor
cmd/internal/ld: change Cpos to not flush the output buffer
3475ec7 
DWARF generation appears to assume Cpos is cheap and this makes linking godoc
about 8% faster and linking the standard library into a single shared library
about 22% faster on my machine.

Updates golang#10571

Change-Id: I3f81efd0174e356716e7971c4f59810b72378177
Reviewed-on: https://go-review.googlesource.com/9913
Reviewed-by: Ian Lance Taylor <[email protected]>
Run-TryBot: Ian Lance Taylor <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
@gopherbot
Copy link
Contributor

CL https://golang.org/cl/10211 mentions this issue.

josharian added a commit to josharian/go that referenced this issue May 18, 2015
@josharian
test: re-enable rotate tests
d73e77d 
Memory usage has been reduced.
The tests are still slow,
but that is issue golang#10571.

/usr/bin/time shows significant variation
in the peak memory usage compiling with tip.
This is unsurprising, given GC.

Using Go 1.4.2, memory is stable at 410mb.
Using tip at a21cf5b,
which does not include some relevant CLs,
memory ranges from 524mb (+28%) to 593mb (+45%),
with a mean of 563mb (+37%), with n=43.

DO NOT SUBMIT -- trybot bait for the moment

Fixes golang#9933.

Change-Id: Id31f3ae086ec324abf70e8f1a8044c4a0c27e274
mwhudson added a commit to mwhudson/go that referenced this issue May 22, 2015
@mwhudson
cmd/internal/ld: change Cpos to not flush the output buffer
f25c512 
DWARF generation appears to assume Cpos is cheap and this makes linking godoc
about 8% faster and linking the standard library into a single shared library
about 22% faster on my machine.

Updates golang#10571

Change-Id: I3f81efd0174e356716e7971c4f59810b72378177
josharian added a commit to josharian/go that referenced this issue May 26, 2015
@josharian
test: re-enable rotate tests
256cd15 
Memory usage has been reduced.
The tests are still slow,
but that is issue golang#10571.

/usr/bin/time shows significant variation
in the peak memory usage compiling with tip.
This is unsurprising, given GC.

Using Go 1.4.2, memory is stable at 410mb.
Using tip at 791bb4f,
memory ranges from 506mb (+23%) to 571mb (+39%),
with a mean of 536mb (+31%), with n=50.

Fixes golang#9933.

Change-Id: Id31f3ae086ec324abf70e8f1a8044c4a0c27e274
josharian added a commit that referenced this issue May 28, 2015
@josharian
test: re-enable rotate tests
3ba6387 
Memory usage has been reduced.
The tests are still slow,
but that is issue #10571.

/usr/bin/time shows significant variation
in the peak memory usage compiling with tip.
This is unsurprising, given GC.

Using Go 1.4.2, memory is stable at 410mb.
Using tip at d2ee09298,
memory ranges from 470mb (+15%) to 534mb (+30%),
with a mean of 504mb (+23%), with n=50.

Fixes #9933.

Change-Id: Id31f3ae086ec324abf70e8f1a8044c4a0c27e274
Reviewed-on: https://go-review.googlesource.com/10211
Reviewed-by: Brad Fitzpatrick <[email protected]>
@gopherbot
Copy link
Contributor

CL https://golang.org/cl/10833 mentions this issue.

@rsc
Copy link
Contributor Author

rsc commented Jun 15, 2015

For the record, although I'm thrilled that the cmd/go tests now run by default, they take 55 seconds on my machine. More to do.

@rsc rsc modified the milestones: Go1.6, Go1.5 Jul 14, 2015
mwhudson added a commit to mwhudson/go that referenced this issue Sep 2, 2015
@mwhudson
cmd/internal/obj, cmd/link: change object files to have a symbol table
555baa7 
DO NOT SUBMIT

I had some time for some recreational hacking and I'd been wondering how much
using a symbol table in object files would speed up the linker. It turns out
that it was a lot more than I expected: this CL improves all.bash by about
25-30s on my laptop (~6.5 mins to ~6 mins). So even though it's late in the
release to do this, is that enough of an improvement to be worth it?

This is mostly just the simplest thing I could do, I didn't think about things
very hard.

Updates golang#10571

Change-Id: I24eb071425c77b1c611d8f1f4e63deee22887e76
@Thomasdezeeuw Thomasdezeeuw mentioned this issue Oct 1, 2015
Closed
@rsc rsc modified the milestones: Unplanned, Go1.6 Dec 5, 2015
@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18085 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18087 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18086 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18088 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18110 mentions this issue.

rsc added a commit that referenced this issue Dec 29, 2015
@rsc
build: shorten a few packages with long tests
04d732b 
Takes 3% off my all.bash run time.

For #10571.

Change-Id: I8f00f523d6919e87182d35722a669b0b96b8218b
Reviewed-on: https://go-review.googlesource.com/18087
Reviewed-by: Ian Lance Taylor <[email protected]>
Reviewed-by: Brad Fitzpatrick <[email protected]>
Run-TryBot: Russ Cox <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
rsc added a commit that referenced this issue Dec 29, 2015
@rsc
cmd/dist: run limited tests in race and libgcc=none modes
f41378e 
Mostly we just care that the test binaries link and start up.
No need to run the full test suites.

Takes 12% off my all.bash run time.

For #10571.

Change-Id: I01af618f3d51deb841ea638424e1389a2df7d746
Reviewed-on: https://go-review.googlesource.com/18086
Reviewed-by: Brad Fitzpatrick <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Run-TryBot: Russ Cox <[email protected]>
rsc added a commit that referenced this issue Dec 29, 2015
@rsc
cmd/dist: run various one-off tests in parallel
baa928a 
Takes 15% off my all.bash run time
(after this and earlier CLs, now down to 3½ from 5½ minutes).

For #10571.

Change-Id: Iac316ffb730c9ff0a0faa7cc3b82ed4f7e6d4361
Reviewed-on: https://go-review.googlesource.com/18088
Reviewed-by: Ian Lance Taylor <[email protected]>
@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18198 mentions this issue.

rsc added a commit that referenced this issue Dec 29, 2015
@rsc
cmd/dist: run wiki, shootout etc tests only on builders
b6473ff 
These find approximately nothing.
Takes 5% off my all.bash run time.

For #10571.

Change-Id: I21d3a844af756eb37f59bba0064f24995626da0d
Reviewed-on: https://go-review.googlesource.com/18198
Reviewed-by: Brad Fitzpatrick <[email protected]>
@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18205 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18206 mentions this issue.

rsc added a commit that referenced this issue Dec 29, 2015
@rsc
runtime: move test programs out of source code, coalesce
8d5ff2e 
Now there are just three programs to compile instead of many,
and repeated tests can reuse the compilation result instead of
rebuilding it.

Combined, these changes reduce the time spent testing runtime
during all.bash on my laptop from about 60 to about 30 seconds.
(All.bash itself runs in 5½ minutes.)

For #10571.

Change-Id: Ie2c1798b847f1a635a860d11dcdab14375319ae9
Reviewed-on: https://go-review.googlesource.com/18085
Reviewed-by: Austin Clements <[email protected]>
Run-TryBot: Austin Clements <[email protected]>
rsc added a commit that referenced this issue Jan 4, 2016
@rsc
runtime/pprof: shorten a few tests
9d549b5 
For #10571.

Change-Id: I4bdad64e2dfd692ef2adccf2e5e82e9b1996a8ea
Reviewed-on: https://go-review.googlesource.com/18206
Reviewed-by: Austin Clements <[email protected]>
Run-TryBot: Austin Clements <[email protected]>
rsc added a commit that referenced this issue Jan 5, 2016
@rsc
net: run all timeout tests in parallel
d731315 
For #10571.

Change-Id: I9a42226078b9c52dbe0c65cb101b5f452233e911
Reviewed-on: https://go-review.googlesource.com/18205
Reviewed-by: Ian Lance Taylor <[email protected]>
Reviewed-by: Mikio Hara <[email protected]>
Run-TryBot: Russ Cox <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
@bcmills
Copy link
Contributor

bcmills commented Oct 28, 2016

I notice that all.bash uses a tiny fraction of the CPU on my machine, and seems to only execute one process at a time. Would it be feasible to run more than one test process in parallel?

@rsc
Copy link
Contributor Author

rsc commented Oct 28, 2016

@bcmills, please open a separate issue with details about your machine, Go version, and what runtime.NumCPU() returns. Both the build half and the test half of all.bash should consume nearly all the CPUs for most of the time. The start and end of each half tend to get single-threaded due to dependency fanout or fanin (depending on how you look at it), and the initial bootstrap does run single-threaded. But most of all.bash should be very multithreaded.

@bcmills bcmills mentioned this issue Nov 2, 2016
Open
@rsc
Copy link
Contributor Author

rsc commented May 15, 2019

Closing in favor of #26473, which has most of the recent work.

@rsc rsc closed this as completed May 15, 2019
@golang golang locked and limited conversation to collaborators May 14, 2020
@rsc rsc removed their assignment Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@josharian @rsc @bcmills @gopherbot