Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow verifying vendored code #27348

Open
krancour opened this issue Aug 29, 2018 · 21 comments
Open

cmd/go: allow verifying vendored code #27348

krancour opened this issue Aug 29, 2018 · 21 comments
Labels
FeatureRequest modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@krancour
Copy link

go mod verify is extremely useful for validating the integrity of modules in the local cache.

It would be great if projects that choose to vendor their modules (then presumably building with go build -mod vendor ...) had a similar command to verify the integrity of modules in that directory.

This would satisfy a major requirement that many projects need to account for in their CI process-- ensuring that vendored code hasn't been tampered with.

@FiloSottile FiloSottile added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. FeatureRequest modules labels Aug 29, 2018
@FiloSottile FiloSottile added this to the Go1.12 milestone Aug 29, 2018
@FiloSottile FiloSottile changed the title cmd/go: feature request: verify vendored code cmd/go: allow verifying vendored code Aug 29, 2018
@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 25, 2018
@bcmills
Copy link
Contributor

bcmills commented Feb 15, 2019

This is still on our radar, but probably not happening for 1.13. (We have a lot to do this cycle!)

I'm hoping to get to it in 1.14, but we don't have a 1.14 milestone defined yet.

@bcmills bcmills modified the milestones: Go1.13, Unplanned Feb 15, 2019
stp-ip added a commit to stp-ip/caddy that referenced this issue Mar 5, 2019
amshinde added a commit to kata-containers/tests that referenced this issue Jul 31, 2019
While we move to go modules, perform the dep check for repos
that still use dep.
Run `go mod verify` instead for go modules.
Note, this just verifies the integrity of modules in the local
cache. We would have instead wanted to verify the vendored code
here, but that is still not supported.
golang/go#27348

Fixes #1879

Signed-off-by: Archana Shinde <[email protected]>
@bcmills bcmills modified the milestones: Unplanned, Go1.14 Aug 15, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
stevendanna added a commit to chef/automate that referenced this issue Nov 4, 2019
`go mod verify` does not verify the vendored copies of
dependencies:

golang/go#27348

As such, it seems that this change snuck in. This commit is the result
of commit the all changes produced by `make revendor` on master.

Signed-off-by: Steven Danna <[email protected]>
stevendanna added a commit to chef/automate that referenced this issue Nov 4, 2019
`go mod verify` does not verify the vendored copies of
dependencies:

golang/go#27348

As such, it seems that this change snuck in. This commit is the result
of commit the all changes produced by `make revendor` on master.

Signed-off-by: Steven Danna <[email protected]>
@bcmills bcmills added this to the Go1.16 milestone May 19, 2020
@bcmills
Copy link
Contributor

bcmills commented May 19, 2020

No, it's definitely not happening for 1.15.

@ianlancetaylor
Copy link
Member

Moving milestone to Backlog.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.17, Backlog Apr 19, 2021
@iholder101
Copy link

iholder101 commented May 26, 2021

This is indeed valuable! Projects that choose to vendor their modules usually do not want to count on every package to be available 100% of the time, therefore validation that enforces re-downloading the package is problematic.

Would be happy to see this implemented.

@ArsenArsen
Copy link

There seems to be a fundamental issue here, vendor directories exclude many files not used in the actual build. Verifying against go.sum partially becomes impossible.

I'd like to automatically generate vendored tarballs from sources, while being able to verify them, so this issue becomes relevant to me.

Is there any discussed solution that went unrecorded in this thread?

Thanks in advance

@bcmills
Copy link
Contributor

bcmills commented Feb 28, 2022

@ArsenArsen, the plan at the moment is to have go mod verify download the full module source code for each dependency.

Then we can demonstrate the integrity of the vendor tree by showing that, for each package in the vendor tree, the directory for that package contains exactly the expected subset of source files, and the contents of each of those files match the contents of the corresponding file in the module cache.

@bcmills
Copy link
Contributor

bcmills commented Feb 28, 2022

(However, note that go mod verify cannot be implemented for vendor trees in a way that is independent of the module cache.)

@ArsenArsen
Copy link

@bcmills Hm, while that works and will produce valid results, the thing we're implementing requires us to be able to do so without the network.

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment? I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Thanks in advance, and for the quick response.

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2022

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment?

In theory you could checksum each package, but the go.sum file (and sum.golang.org) only contains checksums for modules and I don't see that changing.

I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Actually, that would work (once we implement go mod verify for vendored code at all).
You can set GOPROXY to point to $GOMODCACHE/cache/download to treat a module cache as a module proxy. (See #43646.)

@petar-dochev-f3
Copy link

A problem we recently hit is that due to some recursive patterns in .gitignore some files under vendor were not committed to git, which resulted in some strange errors in CI which could not be reproduced locally. It would be very helpful if there is a command we could use use in our CI to verify vendor is intact. Also sometimes people make changes in vendor by accident.

@qmuntal
Copy link
Contributor

qmuntal commented May 17, 2023

@bcmills what if go mod vendor creates a vendor.sum file next to modules.txt and go mod verify checks the integrity of the vendored modules against vendor.sum?

The downside is that the verify command wouldn't have any external source (e.g., GOSUMDB) to validate vendor.sum checksums, but AFAIK that's not happening neither for non-vendored modules (see #47752). vendor.sum could be recreated from the module cache using go mod vendor, so I don't see this as a blocker.

@bcmills
Copy link
Contributor

bcmills commented May 24, 2023

I guess a vendor.sum could help defend against stray edits, at least. That's an interesting thought.

@jpbetz
Copy link

jpbetz commented May 26, 2023

Since this has been open for almost 5 years, maybe recommend a mitigation? Kubernetes performs this check with https://github.com/kubernetes/kubernetes/blob/master/hack/verify-vendor.sh. I'm looking to do the same for github.com/google/cel-go, and having a recipe would be a huge time saver.

@bcmills
Copy link
Contributor

bcmills commented May 30, 2023

@jpbetz, you can run the following:

  • go mod vendor (to redownload and checksum the dependencies in the vendor tree)
  • Then check the vendor directory for diffs (to make sure it matches what was downloaded).
  • Then run go mod verify -mod=readonly (to verify that the contents of downloaded modules match their checksums).

@jpbetz
Copy link

jpbetz commented May 30, 2023

Thanks @bcmills

In cel-go we ended up using this script: https://github.com/google/cel-go/blob/master/scripts/verify-vendor.sh

@aep-sunlife
Copy link

This is a security concern. An attacker lying about the contents of vendor/ in go.{mod,sum} can inject malware.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests