x/build/builders: coordinator and buildlet need to build subrepos

[bradfitz]

buildlet probably needs environment variable support in HTTP headers. and returning a tarball of a directory. and extracting a tarball from a URL, rather than being fed it in a PUT.

coordinator needs to then get the subrepo lists from http://build.golang.org/?mode=json

[adg]

You can get the sub-repo list from
http://build.golang.org/packages?kind=subrepo

On 5 January 2015 at 16:11, Brad Fitzpatrick [email protected]
wrote:

buildlet probably needs environment variable support in HTTP headers. and
returning a tarball of a directory. and extracting a tarball from a URL,
rather than being fed it in a PUT.

coordinator needs to then get the subrepo lists from
http://build.golang.org/?mode=json

—
Reply to this email directly or view it on GitHub
#9506.

[bradfitz]

@adg, did you want to work on this?

We'll need to snapshot the "go" directory after make.{bash,bat,rc} and put it on GCS. Probably in a new bucket.

And then when we need to build a subrepo, we instead untar the pre-built make output from GCS onto the buildlet, and then run just the subrepo tests.

Doing the snapshot after make.bash means we need to actually run make.bash separately, and not all.bash. This ties in with the main repo test sharding later, wherein we'll snapshot after make.bash and then start N VMs over which to run all the tests. But for now you'll need to modify the coordinator to run make.bash, snapshot, and then run.bash

[adg]

Plans:

• Store builds in a GCS bucket called go-builder-tarballs
• Name the tarballs $HASH.$BUILDER.tar.gz
• Overwrite existing tarballs; shouldn't happen except in weird circumstances where we re-trigger builds
• In buildInVM run make.bash, then simultaneously push the tarball to GCS and run run.bash.
• checkpoint
• Add subrepo revision discovery to the findWork loop (requires changes on the dashboard side)
• Add new subrepo and subrepoRev fields to builderRev
• Pass a builderRev value through from the main loop to startBuilding, startBuildingInVM
• Add logic to buildInVM to do subrepo builds (fetch go tree tarball from gcs, fetch sub-repo tarball from gerrit, resolve any dependencies at master (the dashboard doesn't track transitive dependencies between subrepos unfortunately), run go test)

Issues to resolve:

• Garbage collection of old tarballs? Do we care? At, say, 50mb * 20 builders * 50 commits = 50gb/day. That's quite a lot.
  • How to manage gc? Hard to think of a means that doesn't track state somehow.

[nightlyone]

Garbage collection of tarballs could be done evaluating access logs. If old and not accessed for a long time, it is probably not interesting anymore (s. https://cloud.google.com/storage/docs/access-logs).

[adg]

Nice idea, but we aren't allowed to collect logs because "reasons" (am not
able to elaborate).

On 17 February 2015 at 06:47, Ingo Oeser [email protected] wrote:

Garbage collection of tarballs could be done evaluating access logs. If
not accessed for a long time, it is probably not interesting anymore (s.
https://cloud.google.com/storage/docs/access-logs).

—
Reply to this email directly or view it on GitHub
#9506 (comment).

[minux]

The Go 1.5 code freeze is about to come, we should prioritize fixing this as the more new builders we have, the less tested the sub-repos become.

[adg]

Working on it now.

[crawshaw]

@adg are you working on this? I'm interested in getting the mobile subrepository building. It needs a couple of custom hacks over the other subrepos to deal with cross-compilation, but they should be minimal on top of normal building, so I guess that should come first. If you don't have any code outstanding, I'm happy to take a crack at it.

[adg]

Yes I'm working on this with @bradfitz. We're close to getting the test
sharding code in; once that's in, I'll be in a good spot to tackle the
subrepo thing (and know pretty much how to do it).

What kind of hacks do you mean? You mean adding stuff to the buildlet base
images?

On 29 May 2015 at 13:26, David Crawshaw [email protected] wrote:

@adg https://github.com/adg are you working on this? I'm interested in
getting the mobile subrepository building. It needs a couple of custom
hacks over the other subrepos to deal with cross-compilation, but they
should be minimal on top of normal building, so I guess that should come
first. If you don't have any code outstanding, I'm happy to take a crack at
it.

—
Reply to this email directly or view it on GitHub
#9506 (comment).

[crawshaw]

The hacks for mobile will be minimal environmental changes. It will probably be easier to understand on a CL. I'll wait for you get basic subrepos working and then send you something.

(Basically the go test golang.org/x/mobile/... needs to run on the host with the android/iOS device attached.)