Migrate to Go modules #3418

axw · 2020-03-04T02:50:54Z

Motivation/summary

Use Go modules instead of vendoring. Modules are the future.

There are a bunch of changes here:

The vendor and _beats directories are removed. A couple of minor bash scripts, and the reviewdog.yml file, have been copied into our tree. They haven't been changed in a couple of years.
generate_notice.py has been forked to stop assuming vendoring. It now only considers modules that are used in the apm-server binary, and not those used for build/test purposes.
Updated beats import paths to use "beats/v7"
Updated Makefile and scripts to work with modules. A handful of scripts have been removed.
Updated to elastic/beats@bbf9d6697f4a, and adapted to some ES output-related breaking changes

Checklist

I have signed the Contributor License Agreement.
My code follows the style guidelines of this project (run make check-full for static code checks and linting)
I have rebased my changes on top of the latest master branch
I have made corresponding changes to the documentation
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
~~- [ ] New and existing unit tests pass locally with my changes~~
~~- [ ] I have updated CHANGELOG.asciidoc~~ (no user-facing changes)

How to test these changes

make
make docker-system-tests
make update-beats

Related issues

simitt

This is amazing work!

I tested the most important make cmds and they worked fine, except for running into the same issues as CI does for make docker-system-tests. Will do a final review once this is solved.

script/generate_notice.py

axw · 2020-03-04T09:50:16Z

I tested the most important make cmds and they worked fine, except for running into the same issues as CI does for make docker-system-tests. Will do a final review once this is solved.

@simitt thanks for taking a look. I'm working through CI issues one-by-one, I'll let you know when I think it's stable enough for reviewing again.

axw · 2020-03-05T07:00:02Z

crossbuild is failing to locate the github.com/elastic/beats/v7 module directory:

[2020-03-04T14:27:13.282Z] panic: failed determine libbeat dir location: expected 1 line, got 0

[2020-03-04T14:27:13.282Z] 

[2020-03-04T14:27:13.282Z] goroutine 1 [running]:

[2020-03-04T14:27:13.282Z] github.com/elastic/beats/v7/dev-tools/mage.LibbeatDir(0xc0001c20b0, 0x1, 0x1, 0xc0000b6f80, 0x34)

[2020-03-04T14:27:13.282Z] 	/var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-3418/pkg/mod/github.com/elastic/beats/[email protected]/dev-tools/mage/common.go:742 +0x1d3

[2020-03-04T14:27:13.282Z] github.com/elastic/beats/v7/dev-tools/mage.init()

[2020-03-04T14:27:13.282Z] 	/var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-3418/pkg/mod/github.com/elastic/beats/[email protected]/dev-

This is because in the fresh container, the module cache hasn't yet been populated. We'll need to either share the module cache from the host (might be a bit dangerous?), or otherwise seed the container's module cache before calling mage.

axw · 2020-03-06T03:07:03Z

elastic/beats#16837 is up to address the above issue. I went for mounting the module cache in, but made it read-only. Before starting the crossbuild containers, we run go mod download on the host to ensure the cache is up to date.

axw · 2020-03-06T13:08:40Z

I lost the race with elastic/beats#16150. Will need to update to the latest API in this PR.

- Add go.mod - Add tools.go for tracking build/test tools in go.mod - Update Makefile to use go.mod, instead of vendor - Update beats imports to use "beats/v7" - Update cespare/xxhash to use "xxhash/v2" - Copy reviewdog.yml from beats - Copy build scripts from beats/dev-tools

- Modify "update-beats" target to rsync libbeat/testing/environments into apm-server/testing/environments. We could alternatively just go our own way, but this is fairly easy to maintain for now. - Update tests/Dockerfile to remove GOPATH, and prime the container with the module dependencies by copying in "go.mod" and running "go mod download".

Handle libbeat/common/transport moves. Closes elastic#3416

Crossbuild fixes.

Closes elastic#3433

axw · 2020-03-07T03:05:07Z

@simitt sorry for the force push. I should have been clearer before, that I didn't intend for anyone to look at it while it was WIP. It was non-draft so that CI would run.

Should be good to review now, CI is passing. Every commit other than the first should be reviewable ;)

axw · 2020-03-07T03:10:11Z

TODO:

Update README to remove govendor bits, replace with modules.

simitt

Worked as expected locally.

jalvz

Great to have this!

docs/fields.asciidoc

testing/environments/5.x.yml

- Add go.mod - Add tools.go for tracking build/test tools in go.mod - Update Makefile to use go.mod, instead of vendor - Update beats imports to use "beats/v7" - Update cespare/xxhash to use "xxhash/v2" - Copy reviewdog.yml from beats - Copy build scripts from beats/dev-tools - Remove vendor and _beats - Fork generate_notice.py -- needed because we do not (can not) use vendoring - Update beats framework to bbf9d6697f4a; adapt to several breaking changes in libbeat - tests/system: use "go list" to find libbeat - Modify "update-beats" target to rsync libbeat/testing/environments into apm-server/testing/environments. We could alternatively just go our own way, but this is fairly easy to maintain for now. - Update tests/Dockerfile to remove GOPATH, and prime the container with the module dependencies by copying in "go.mod" and running "go mod download".

* Migrate to Go modules (#3418) - Add go.mod - Add tools.go for tracking build/test tools in go.mod - Update Makefile to use go.mod, instead of vendor - Update beats imports to use "beats/v7" - Update cespare/xxhash to use "xxhash/v2" - Copy reviewdog.yml from beats - Copy build scripts from beats/dev-tools - Remove vendor and _beats - Fork generate_notice.py -- needed because we do not (can not) use vendoring - Update beats framework to bbf9d6697f4a; adapt to several breaking changes in libbeat - tests/system: use "go list" to find libbeat - Modify "update-beats" target to rsync libbeat/testing/environments into apm-server/testing/environments. We could alternatively just go our own way, but this is fairly easy to maintain for now. - Update tests/Dockerfile to remove GOPATH, and prime the container with the module dependencies by copying in "go.mod" and running "go mod download". * Update to elastic/beats@9735c6d51540 The eslegclient changes have not been backported to 7.x (yet?), so this commit reverts those changes.

simitt reviewed Mar 4, 2020

View reviewed changes

script/generate_notice.py Show resolved Hide resolved

Remove vendor and _beats

9ff7364

axw force-pushed the go-modules branch from 1b4acd0 to d67f311 Compare March 7, 2020 02:54

axw changed the title ~~Migrate to Go modules (WIP)~~ Migrate to Go modules Mar 7, 2020

axw added 10 commits March 7, 2020 11:00

Fork generate_notice.py

bafb642

beater: adapt to upstream changes

7e115c0

validation: staticcheck tidy-up

88ad86a

tests/system: use "go list" to find libbeat

412ab09

elasticsearch: fix test

7c20a1a

Update beats framework to 9718fa246a26

b3f8886

Handle libbeat/common/transport moves. Closes elastic#3416

Update beats framework to bbf9d6697f4a

9d773c1

Crossbuild fixes.

Update for libbeat "eslegclient" changes

7b346d3

Closes elastic#3433

axw force-pushed the go-modules branch from d67f311 to 7b346d3 Compare March 7, 2020 03:01

axw requested a review from simitt March 7, 2020 03:03

axw requested a review from a team March 7, 2020 03:06

axw added 2 commits March 7, 2020 13:29

Update README

7d6abaf

fix-mod

7615d0b

simitt approved these changes Mar 9, 2020

View reviewed changes

jalvz approved these changes Mar 9, 2020

View reviewed changes

docs/fields.asciidoc Outdated Show resolved Hide resolved

testing/environments/5.x.yml Outdated Show resolved Hide resolved

axw added 2 commits March 10, 2020 11:40

Merge branch 'master' into go-modules

69485c8

magefile: remove autodiscover modules from fields

3f5f74d

axw merged commit 45e8825 into elastic:master Mar 10, 2020

axw mentioned this pull request Mar 12, 2020

[7.x] Migrate to Go modules (#3418) #3466

Merged

axw mentioned this pull request Mar 23, 2020

Heap usage increased #3520

Closed

bmorelli25 mentioned this pull request Apr 23, 2020

Improve requirements, developer and contribution section in README. #2118

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to Go modules #3418

Migrate to Go modules #3418

axw commented Mar 4, 2020 •

edited

Loading

simitt left a comment

axw commented Mar 4, 2020

axw commented Mar 5, 2020

axw commented Mar 6, 2020 •

edited

Loading

axw commented Mar 6, 2020

axw commented Mar 7, 2020

axw commented Mar 7, 2020 •

edited

Loading

simitt left a comment

jalvz left a comment

Migrate to Go modules #3418

Migrate to Go modules #3418

Conversation

axw commented Mar 4, 2020 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

simitt left a comment

Choose a reason for hiding this comment

axw commented Mar 4, 2020

axw commented Mar 5, 2020

axw commented Mar 6, 2020 • edited Loading

axw commented Mar 6, 2020

axw commented Mar 7, 2020

axw commented Mar 7, 2020 • edited Loading

simitt left a comment

Choose a reason for hiding this comment

jalvz left a comment

Choose a reason for hiding this comment

axw commented Mar 4, 2020 •

edited

Loading

axw commented Mar 6, 2020 •

edited

Loading

axw commented Mar 7, 2020 •

edited

Loading