x/vuln/cmd/govulncheck: consuming 90%+ of memory and runs without an end

[ArnoSen]

govulncheck version

In my script I am using 'latest' which resolves to v1.1.1

Does this issue reproduce at the latest version of golang.org/x/vuln?

Yes

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY='git.private.repository'
GONOSUMDB='git.private.repository'
GOOS='linux'
GOPATH='/go'
GOPRIVATE='git.private.repository'
GOPROXY='https://p01-goproxy.private.repository:9020,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.10'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/builds/privae/project/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1979460228=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I using gitlab CI/CD with a runner that runs in Docker and then spins up a new image (docker in docker).
In this case govulncheck runs in a container with image "golang:1.21.10" .

During the build, the following command is executed:
$ go run golang.org/x/vuln/cmd/govulncheck@latest ./...

What did you see happen?

I noticed the vulnerability check takes very long and looks like it never finishes
Upon inspection of the container the memory usage is +90% . The build server has 4GB of mem and govulncheck takes over 3GB. It let it run for over 6 minutes but it did not produce any result.

What did you expect to see?

I expect it to finish it with an acceptable time and not consume over 90% of memory.

What makes this behavior particularly strange is that this seems to happen when a new Go (minor) version is released: when I run the pipeline with "golang:1.21.11", the vulnerability check finishes in less than 20s.
I suspect this has go to do with the fact that there is a finding.

When I run govulncheck from my local machine on the same project I do get a report of issues found:
$go version
go version go1.21.10 linux/amd64
$ $ go run golang.org/x/vuln/cmd/govulncheck@latest ./...

=== Symbol Results ===

Vulnerability #1: GO-2024-2887
    Unexpected behavior from Is methods for IPv4-mapped IPv6 addresses in
    net/netip
  More info: https://pkg.go.dev/vuln/GO-2024-2887
  Standard library
    Found in: net/[email protected]
    Fixed in: net/[email protected]
    Example traces found:
      #1: genutils/mailutils/goNativeMailer.go:40:33: mailutils.GoNativeMailer.SendMail calls net.Dialer.DialContext, which eventually calls netip.Addr.IsLoopback
      #2: genutils/mailutils/goNativeMailer.go:40:33: mailutils.GoNativeMailer.SendMail calls net.Dialer.DialContext, which eventually calls netip.Addr.IsMulticast

Your code is affected by 1 vulnerability from the Go standard library.
This scan also found 0 vulnerabilities in packages you import and 5
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
exit status 3

[seankhliao]

What makes this behavior particularly strange is that this seems to happen when a new Go (minor) version is released: when I run the pipeline with "golang:1.21.11", the vulnerability check finishes in less than 20s.
I suspect this has go to do with the fact that there is a finding.

Can you clarify this point?
Are you saying that it works (completes) with 1.21.11 but not with 1.21.10?

[timothy-king]

cc @zpavlinovic

[ArnoSen]

What makes this behavior particularly strange is that this seems to happen when a new Go (minor) version is released: when I run the pipeline with "golang:1.21.11", the vulnerability check finishes in less than 20s.
I suspect this has go to do with the fact that there is a finding.

Can you clarify this point? Are you saying that it works (completes) with 1.21.11 but not with 1.21.10?

Yes, that is correct. It completes in a reasonable timeframe with go 1.21.11 and with 1.21.10 it takes at least 30 minutes (after that the govulncheck was killed). I suspect it is because 1.21.11 is just out this week and has no vulnerabilities in the database.

[ArnoSen]

I tried to build a container image in which the issue can be replicated but so far without success.

[zpavlinovic]

What makes this behavior particularly strange is that this seems to happen when a new Go (minor) version is released: when I run the pipeline with "golang:1.21.11", the vulnerability check finishes in less than 20s.
I suspect this has go to do with the fact that there is a finding.

Can you clarify this point? Are you saying that it works (completes) with 1.21.11 but not with 1.21.10?

Yes, that is correct. It completes in a reasonable timeframe with go 1.21.11 and with 1.21.10 it takes at least 30 minutes (after that the govulncheck was killed). I suspect it is because 1.21.11 is just out this week and has no vulnerabilities in the database.

What results do you get with go1.21.11, No vulnerabilities found?

It seems you are able to get a result with go1.21.10 on your local machine. How much time does that take?

(Note that you are invoking govulncheck in its default mode where it tries to build a call graph and find a call stack to the vulnerable symbols. This will take time and memory for large projects with many dependencies)

[ArnoSen]

The results of govulncheck with go1.21.11 in the container on the build server are:

time go run golang.org/x/vuln/cmd/govulncheck@latest ./...
go: downloading golang.org/x/vuln v1.1.1
go: downloading golang.org/x/mod v0.17.0
go: downloading golang.org/x/tools v0.21.1-0.20240514024235-59d9797072e7
go: downloading golang.org/x/sync v0.7.0
=== Symbol Results ===
No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 4
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.

real    1m29.855s
user    0m52.493s
sys     0m9.099s

When I run it with 1.22.10 on my local machine (32GB RAM + intel i7-12800H multicore) it reports:

time go run golang.org/x/vuln/cmd/govulncheck@latest ./...
=== Symbol Results ===

Vulnerability #1: GO-2024-2887
    Unexpected behavior from Is methods for IPv4-mapped IPv6 addresses in
    net/netip
  More info: https://pkg.go.dev/vuln/GO-2024-2887
  Standard library
    Found in: net/[email protected]
    Fixed in: net/[email protected]
    Example traces found:
      #1: genutils/mailutils/goNativeMailer.go:40:33: mailutils.GoNativeMailer.SendMail calls net.Dialer.DialContext, which eventually calls netip.Addr.IsLoopback
      #2: genutils/mailutils/goNativeMailer.go:40:33: mailutils.GoNativeMailer.SendMail calls net.Dialer.DialContext, which eventually calls netip.Addr.IsMulticast

Your code is affected by 1 vulnerability from the Go standard library.
This scan also found 0 vulnerabilities in packages you import and 5
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
exit status 3

real    0m10,165s
user    0m28,072s
sys     0m1,664s

[zpavlinovic]

It looks that both of those runs finished within reasonable times. Could you also provide the results on running with go1.21.11 but on your local machine?

[ArnoSen]

Sure. See below:

$ go version
go version go1.21.11 linux/amd64
$ time go run golang.org/x/vuln/cmd/govulncheck@latest ./...
=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 4
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.

real    0m6,951s
user    0m39,196s
sys     0m4,693s

[zpavlinovic]

On your local machine, both go1.21.10 and go1.21.11 finish rather quickly. go1.21.11 is somewhat faster. My hunch is that in this case there are no vulnerabilities that apply to your code at all and govulncheck can take a fast path out. (A signal for this is that GO-2024-2887 is fixed in 1.21.11.)

What is interesting is that it takes ~13x more time for go1.21.11 to finish in the container compared to the local machine.

govulncheck integration tests (cmd/govulncheck/integration) run govulncheck in a container on k8s source code and on a binary built from a certain repo. Building the container, cloning the k8s and the other repo, building govulncheck and the test binary, and then finally running govulncheck takes in total 3.5 minutes. (We've observed similar times with different go versions in the container).

That suggests that the issue you are seeing is related to your container setup.

[ArnoSen]

Can it be that the resources (RAM+CPU) are insufficient for the buildserver? I think that when it starts swapping, it does not finish the job in a reasonable timeframe.
On the other hand, if that makes the govulncheck finish, it would indicate that when a vulnerability is found the cost of getting the details is relatively high in terms of performance.

[zpavlinovic]

That could be the case for really very large programs with plenty dependencies, but I am still a little bit surprised by the numbers. You can play with this by running govulncheck in the container for a specific package of your module (instead of ./...) and see if things are faster.

If increasing RAM on your builder is not an option, note that you can also run govulncheck with -scan package and -scan module modes. These should run much faster but will also be less precise as they don't inspect program call graph.

[ArnoSen]

This application has quite a few dependencies.
I think increasing my RAM is the only option to get a good insight of potential security vulnerabilities.
How does govulncheck internally work with regards to creating a call graph? Does it create one after there is a match between an import part and a entry in the vuln database or does it need a call graph always before it starts checking the vulnerability database?
Ideally there is a warning/error from the govulncheck when it has not enough resources to finish in a reasonable time but that maybe not that easy to develop.

[zpavlinovic]

govulncheck will first get vulnerabilities for your dependencies from the database. If there are no vulnerabilities for your dependencies or you are not depending on them at a vulnerable version, govulncheck will exit without creating a call graph.

[ArnoSen]

I did not know that. Thanks for explaining. Tomorrow I will add more resources to the buildserver and I would expect it to run fine. I will close the issue unless I see something strange. Thanks for the great support. It was very helpful.

[ArnoSen]

After doubling the RAM from 4 to 8 GB (still with 1 CPU) the vulncheck finished just under 2 minutes:

time go run golang.org/x/vuln/cmd/govulncheck@latest ./...
=== Symbol Results ===

Vulnerability #1: GO-2024-2887
    Unexpected behavior from Is methods for IPv4-mapped IPv6 addresses in
    net/netip
  More info: https://pkg.go.dev/vuln/GO-2024-2887
  Standard library
    Found in: net/[email protected]
    Fixed in: net/[email protected]
    Example traces found:
      #1: genutils/mailutils/goNativeMailer.go:40:33: mailutils.GoNativeMailer.SendMail calls net.Dialer.DialContext, which eventually calls netip.Addr.IsLoopback
      #2: genutils/mailutils/goNativeMailer.go:40:33: mailutils.GoNativeMailer.SendMail calls net.Dialer.DialContext, which eventually calls netip.Addr.IsMulticast

Your code is affected by 1 vulnerability from the Go standard library.
This scan also found 0 vulnerabilities in packages you import and 5
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
exit status 3

real    1m55.524s
user    1m19.019s
sys     0m9.860s

I will close the issue. It would be nice to have the check consumes less memory. Before this issue popped up, the build server only had 2GB of RAM.
Thanks again for the support! Much appreciated.

[timothy-king]

@ArnoSen 8Gb is a bit higher than we want folks to need to set limits on CI/CD pipelines. If you have precise reproducer instructions, we could use that to see if there are any improvements we can make for memory consumption. No guarantees on whether we will do this or when it might happen. But if we have a reproducer that is a much clearer starting point if this gets off the back burner.

[ArnoSen]

I cannot share the code so I am afraid that limits your ability to exactly reproduce the vulnerability checks memory consumption. Is there anything I can help you with in that case? Maybe some profiling data?