Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: migrate to using Go modules #26872

Closed
bcmills opened this issue Aug 8, 2018 · 21 comments
Closed

x/build: migrate to using Go modules #26872

bcmills opened this issue Aug 8, 2018 · 21 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Aug 8, 2018

On https://golang.org/cl/128636, @bradfitz notes:

I'd want to understand what we do & don't use [modules] for […].

For example, all of our "make deploy-prod" etc scripts right now build in a Docker container with an entirely different mechanism.

Initially, I think we should start with exactly what we're doing today, but with the ability to make changes in x/build with a known-reasonable set of dependencies. That's https://golang.org/cl/128636.

There are certainly more interesting steps we can take down the road: for example, we might be able to use go mod download in place of cmd/gitlock to wire in dependencies.

This issue is for tracking those steps.
@gopherbot gopherbot added this to the Unreleased milestone Aug 8, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 8, 2018
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed Builders x/build issues (builders, bots, dashboards) labels Aug 8, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 8, 2018
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/128636 mentions this issue: all: add module definitions

gopherbot pushed a commit to golang/build that referenced this issue Aug 8, 2018
all: add module definitions
0e768f6 
I computed these using a build of the Go toolchain from a working copy at
https://golang.org/cl/128136, patchset 12.

The versions selected by 'go mod tidy' needed two tweaks to pass tests:
go get golang.org/x/text@master
go get github.com/russross/blackfriday@v1

The x/text version bump is because the last tagged version (v0.3.0) is very old
(last December).

The blackfriday bump is because the repository made a breaking change (v2.0.0)
without changing the import path.

With those changes, 'go test ./...' passes at the repo root.
('go test all' fails, possibly due to golang/go#26279.)

Updates golang/go#26279.
Updates golang/go#26872.

Change-Id: Ieac5327e4bddd2b78b981d7683beb98608708a3a
Reviewed-on: https://go-review.googlesource.com/128636
Reviewed-by: Brad Fitzpatrick <[email protected]>
@gopherbot gopherbot mentioned this issue Aug 16, 2018
Closed
@bcmills
Copy link
Contributor Author

bcmills commented Aug 21, 2018

@FiloSottile notes: “there’s stuff in vendor/ that disagrees with the go.mod, we should fix that if we’re going all in on modules”

@bcmills
Copy link
Contributor Author

bcmills commented Aug 21, 2018 •
edited
Loading

(I'm guessing he's referring to the stuff in vendor/github.com/tarm/serial.)

@dmitshur
Copy link
Contributor

dmitshur commented Aug 21, 2018 •
edited
Loading

(I'm guessing he's referring to the stuff in vendor/github.com/tarm/serial.)

There's also https://golang.org/cl/129075, those 2 commands must be out of sync too by now.

Edit: Whoops, that's not related to stuff in vendor/, but rather a Dockerfile. Sorry.

@bradfitz bradfitz self-assigned this Nov 8, 2018
@bradfitz
Copy link
Contributor

bradfitz commented Nov 8, 2018

I think it's time to delete gitlock.

I've been using modules for my own projects for some time and prefer it over gitlock. (which was always meant as a medium-term solution)

/cc @dmitshur

@dmitshur
Copy link
Contributor

dmitshur commented Nov 8, 2018

Well, given this is x/build and its policy (which I can't find for some reason right now), we can experiment with modules as long as they work for our needs. See https://go-review.googlesource.com/c/build/+/128636/#message-82cc0747061a265a95ab44a9ef7debe6798a4392.

I'm in favor of using modules to build commands reproducibly for deployment (aka, replace what gitlock is used for now). 👍

As long as that doesn't interfere with keeping the libraries in this repo (e.g., maintner) working in the GOPATH world.

@bradfitz
Copy link
Contributor

bradfitz commented Nov 8, 2018

First speed bump:

bradfitz@gdev:~/src/golang.org/x/build/cmd/gopherbot$ GO111MODULE=on go mod init

bradfitz@gdev:~/src/golang.org/x/build/cmd/gopherbot$ GO111MODULE=on go list -json .
.....
can't load package: package golang.org/x/build/cmd/gopherbot: unknown import path "golang.org/x/build/cmd/gopherbot": ambiguous import: found golang.org/x/build/cmd/gopherbot in multiple modules:
        golang.org/x/build/cmd/gopherbot (/home/bradfitz/src/golang.org/x/build/cmd/gopherbot)
        golang.org/x/build v0.0.0-20181108190521-5df336cd7fdc (/home/bradfitz/pkg/mod/golang.org/x/[email protected]/cmd/gopherbot)

I suppose we could have one go.mod for all things in x/build but I'd prefer that each package main tool have its own, so we can bump dependencies independently as we deploy things.

Otherwise we'll be in a state where source code doesn't match deployed reality and we we'll bump go.mod deps affecting a dozen binaries and only deploy one of them, and have a few that haven't been deployed for years with untested deps.

What do I need to do to get past that error above, @bcmills?

@bcmills
Copy link
Contributor Author

bcmills commented Nov 8, 2018 •
edited
Loading

If you're splitting out a submodule, the new submodule needs to depend on a version of the parent that does not contain the newly-split packages.

(For strict go get -u compatibility it should be a cycle as described in #27858 (comment), but since nobody should be importing cmd/gopherbot from outside it probably doesn't matter in this case.)

If you want to actually test that change, you'll probably need a replace directive. (In general, testing changes to multiple modules in the same repo is difficult; see #27542.)

@myitcv
Copy link
Member

myitcv commented Nov 8, 2018

To add to @bcmills's comment, whilst the replace method works (gist, results), you can also do this via separate commits.

@bradfitz bradfitz mentioned this issue Nov 30, 2018
Closed
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/175985 mentions this issue: cmd/coordinator: stop using gitlock, use go modules

gopherbot pushed a commit to golang/build that referenced this issue May 9, 2019
@bradfitz
cmd/coordinator: stop using gitlock, use go modules
9a1c61c 
Also, along for the ride:

* update from jessie to stretch
* update from Go 1.10 to Go 1.12
* move to multi-stage Dockerfile, including drawterm, reducing image size
* remove the static linking which was resulting in build warnings
* clean up Makefile

Updates golang/go#26872
Updates golang/go#27719

Change-Id: Ic4dc9b8539fb8662c9621c113fa94b70bc7de061
Reviewed-on: https://go-review.googlesource.com/c/build/+/175985
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176257 mentions this issue: cmd/buildlet/stage0, cmd/scaleway: stop using gitlock, use go modules

gopherbot pushed a commit to golang/build that referenced this issue May 9, 2019
@bradfitz
cmd/{buildlet/stage0,scaleway,tip}, devapp: stop using gitlock, use g…
45de920 
…o modules

Updates golang/go#26872
Updates golang/go#27719

Change-Id: I4de6d4f157b349911362e02b1781abd8b813f87a
Reviewed-on: https://go-review.googlesource.com/c/build/+/176257
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176317 mentions this issue: playground: support third-party imports (off by default for now)

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176318 mentions this issue: http2/h2demo: stop using gitlock, use Go modules

gopherbot pushed a commit to golang/net that referenced this issue May 9, 2019
@bradfitz
http2/h2demo: stop using gitlock, use Go modules
a4d6f7f 
Updates golang/go#26872

Change-Id: If2a96708d1fcf43b7ce20a48fa5ae3492f970187
Reviewed-on: https://go-review.googlesource.com/c/net/+/176318
Reviewed-by: Dmitri Shuralyov <[email protected]>
gopherbot pushed a commit to golang/playground that referenced this issue May 10, 2019
@bradfitz
playground: support third-party imports (off by default for now)
046e863 
Also, modernize the Dockerfile while I'm at it, using multi-stage
builds more aggressively, and stop using gitlock (golang/go#26872) and
switch to using Go modules to build the playground. (This also turns
the playground into a module.)

Updates golang/go#31944

Change-Id: Ic6f6152469f1930fd04180a3d1e63ed92ea2cfbd
Reviewed-on: https://go-review.googlesource.com/c/playground/+/176317
Reviewed-by: Yury Smolsky <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176600 mentions this issue: cmd/gerritbot: stop using gitlock, use go modules

gopherbot pushed a commit to golang/build that referenced this issue May 10, 2019
@bradfitz
cmd/gerritbot: stop using gitlock, use go modules
626f5d4 
Updates golang/go#26872

Change-Id: I7cf1bc512f7e4ab6592c1d645b95c7d135bd6c59
Reviewed-on: https://go-review.googlesource.com/c/build/+/176600
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176602 mentions this issue: cmd/gopherbot: stop using gitlock, use go modules

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176603 mentions this issue: maintner/maintnerd: stop using gitlock, use go modules

gopherbot pushed a commit to golang/build that referenced this issue May 10, 2019
@bradfitz
cmd/gopherbot: stop using gitlock, use go modules
e06994e 
Updates golang/go#26872

Change-Id: If64b03c03f29ccff2ffcf255d94a2c736806af89
Reviewed-on: https://go-review.googlesource.com/c/build/+/176602
Reviewed-by: Dmitri Shuralyov <[email protected]>
gopherbot pushed a commit to golang/build that referenced this issue May 11, 2019
@bradfitz
maintner/maintnerd: stop using gitlock, use go modules
4420213 
And update to multi-stage Dockerfiles, simplifying the build and
making it more consistent with others.

Updates golang/go#26872

Change-Id: If26e19d73c9bdb996343468cd1cff8edc574cb17
Reviewed-on: https://go-review.googlesource.com/c/build/+/176603
Reviewed-by: Dmitri Shuralyov <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176604 mentions this issue: cmd/gitmirror: stop using gitlock, use go modules

gopherbot pushed a commit to golang/build that referenced this issue May 13, 2019
@bradfitz
cmd/{gitmirror,pubsubhelper,gitlock}: use go modules, delete gitlock
52dbde8 
This removes the last two gitlock users, so this also deletes gitlock.

And update to multi-stage Dockerfiles, simplifying the build and
making it more consistent with others.

Updates golang/go#26872

Change-Id: I1e38c65eccfe9b8b7cb2d9acd6eb02aa33f8b877
Reviewed-on: https://go-review.googlesource.com/c/build/+/176604
Run-TryBot: Brad Fitzpatrick <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
Reviewed-by: Andrew Bonventre <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177217 mentions this issue: cmd/tip: use proxy.golang.org when building x/website

gopherbot pushed a commit to golang/build that referenced this issue May 14, 2019
@dmitshur
cmd/tip: use proxy.golang.org when building x/website
8c74b73 
The new x/website is built in module mode. Now that proxy.golang.org
is available publicly, cmd/tip can start using it for fetching modules
when building x/website.

Also make a change to newServeMux so that httpsOnlyHandler is only used
when HTTPS is turned on. That stops redirecting to HTTPS when HTTPS is
not being served, thereby fixing HTTP-only local development mode.

Updates golang/go#26872
Updates golang/go#30232
Updates golang/go#29206

Change-Id: If0382c45b9c99540adce3a894054730f5a7a9fdb
Reviewed-on: https://go-review.googlesource.com/c/build/+/177217
Reviewed-by: Brad Fitzpatrick <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/179077 mentions this issue: devapp: add ca-certificates to Dockerfile

gopherbot pushed a commit to golang/build that referenced this issue May 27, 2019
@bradfitz
devapp: add ca-certificates to Dockerfile
78beebf 
CL 176257 refactored the Dockerfile to remove the use of gitlock but I
forgot to include ca-certificates here.

Updates golang/go#32272 (fixes maybe)
Updates golang/go#26872

Change-Id: I7b0e3a756bc9805e81e499b8b7d7c6ed0defb871
Reviewed-on: https://go-review.googlesource.com/c/build/+/179077
Reviewed-by: Dmitri Shuralyov <[email protected]>
@bradfitz
Copy link
Contributor

I think this is all done.

gitlock is dead.

@bradfitz bradfitz mentioned this issue Sep 9, 2019
Open
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/227440 mentions this issue: delete update-deps.sh

gopherbot pushed a commit to golang/build that referenced this issue Apr 7, 2020
@dmitshur
delete update-deps.sh
bce80ac 
This script was used when a custom gitlock tool was used for package
management. That approach has been superseded with the switch to the
module mode, and gitlock was deleted in CL 176604. By now, there are
no directories with an update-deps make target remaining.

For golang/go#26872.

Change-Id: Ia7f5d9e8b277be55010d5752523d1160b72115b2
Reviewed-on: https://go-review.googlesource.com/c/build/+/227440
Reviewed-by: Brad Fitzpatrick <[email protected]>
Reviewed-by: Carlos Amedee <[email protected]>
Reviewed-by: Alexander Rakoczy <[email protected]>
Run-TryBot: Alexander Rakoczy <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
@golang golang locked and limited conversation to collaborators Apr 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bradfitz @dmitshur @myitcv @bcmills @gopherbot