Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/vulndb: refactor and test database generation logic #56417

Closed
tatianab opened this issue Oct 25, 2022 · 16 comments
Closed

x/vulndb: refactor and test database generation logic #56417

tatianab opened this issue Oct 25, 2022 · 16 comments
Assignees
@tatianab
Labels
FrozenDueToAge vulncheck or vulndb Issues for the x/vuln or x/vulndb repo
Milestone
vuln/unplanned

Comments

@tatianab
Copy link

No description provided.

@tatianab tatianab added the vulncheck or vulndb Issues for the x/vuln or x/vulndb repo label Oct 25, 2022
@tatianab tatianab self-assigned this Oct 25, 2022
@gopherbot gopherbot modified the milestones: Unreleased, vuln/unplanned Oct 25, 2022
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/445338 mentions this issue: internal/database: (no-op) move loadDB and Diff

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/445339 mentions this issue: internal/database: add validate.go and db test files

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/448840 mentions this issue: internal/database: (no-op) split load and generate files

gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 8, 2022
@tatianab
internal/database: (no-op) split load and generate files
0161bd7 
Move Diff function into its own file, and move code and tests related
to OSV generation into their own file.

For golang/go#56417

Change-Id: Ia41b3f9068efe543c1a603c34738810c404e6caf
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/448840
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
Reviewed-by: Jonathan Amsterdam <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/448841 mentions this issue: internal/database: update and test Load

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/449135 mentions this issue: internal/database: add integration test for Generate

gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 15, 2022
@tatianab
internal/database: update and test Load
f5fee51 
Add a new struct, Database, which is an in-memory representation of
a Go vulnerability database. Update Load to return this struct.

Also adds a test database folder which is used by tests for Load, and
can be used in tests for Generate.

For golang/go#56417

Change-Id: I62d882722186ee96846b646cf88d4320a4f3a307
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/448841
Reviewed-by: Tatiana Bradley <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/450976 mentions this issue: internal/database: add New function

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/450975 mentions this issue: internal/database: add Database.Write function

gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 16, 2022
@tatianab
internal/database: add Database.Write function
28636db 
Adds a function, Write, which writes a Database struct to files. This
will be used by Generate.

Also updates test data to be indented.

For golang/go#56417

Change-Id: I639c36b081611a3d1e90d60d3224e2b40bd101ff
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/450975
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Joedian Reid <[email protected]>
gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 17, 2022
@tatianab
internal/database: add New function
1a5fdb8 
Add a function, New, which generates a new Database struct from a git
repo. The git repo must contain a folder "data/osv" with OSV files.

Adds integration tests to ensure that the current Generate logic is the same
as running New then Write. (Generate will eventually be replaced by these
functions.)

Test data is updated to allow for testing with respect to a git repo,
and to test timestamp logic.

For golang/go#56417

Change-Id: Iae88c5bb8d788bcf025af6d9fb700d87b1834455
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/450976
Reviewed-by: Damien Neil <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Tatiana Bradley <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/448842 mentions this issue: internal/database: add Validate function

gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 22, 2022
@tatianab
internal/database: add Validate function
0397c86 
Adds a function, Validate, which checks a Go vulnerability for
internal consistency. Also adds a command line tool, "checkdb" which
can be used to validate databases.

This tool will be used in the deploy script for vulndb.

For golang/go#56417

Change-Id: I427eab6b5385d3c858d4a371d90e6e5f54f10812
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/448842
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Tatiana Bradley <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/452443 mentions this issue: internal/database: add ValidateDeploy function

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/451400 mentions this issue: internal/database: replace Generate logic with New and Write

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/452771 mentions this issue: deploy: add Validate step to deploy script

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/452772 mentions this issue: internal/database: add more tests for Validate

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/453175 mentions this issue: internal/database: add more tests for Load

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/453176 mentions this issue: internal/database: replace Generate logic with New and Write

gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 28, 2022
@tatianab
internal/database: add logic to validate a new deploy
5d80fbd 
Adds a function, Validate, which checks a candidate Go vulnerability
database against an existing one, to ensure that both databases are
valid, timestamps are consistent and no OSV entries would be deleted.

Moves single-database validation logic (previously called Validate) to
the Load function, so that Load now loads and checks a database.

Also adds a command line tool, "checkdeploy" which calls the new
Validate function. This tool will be used in the deploy script for vulndb.

For golang/go#56417

Change-Id: Ifa12234376f2a3fd577d96978919b167fcb25f64
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/452443
Reviewed-by: Damien Neil <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Jonathan Amsterdam <[email protected]>
gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 28, 2022
@tatianab
internal/database: add more tests for Load
f983c1a 
Adds tests for Load failure cases, adds more context to error message,
and adds an additional failure case.

For golang/go#56417

Change-Id: If4927c11f433c931827b262ee65a04f3594a125a
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/453175
Reviewed-by: Damien Neil <[email protected]>
Reviewed-by: Jonathan Amsterdam <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
gopherbot pushed a commit to golang/vulndb that referenced this issue Nov 28, 2022
@tatianab
deploy: add Validate step to deploy script
8622df3 
Adds two steps, CopyExisting and Validate to the vulndb deploy script,
to ensure that the existing and new databases are valid internally
and with respect to each other. Deploy will not proceed if validation
fails.

For golang/go#56417

Change-Id: I9c522cfb9e3f66f3538d9bc9c89f927692f2c96e
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/452771
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Jonathan Amsterdam <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/455315 mentions this issue: deploy: add Validate step to deploy script

gopherbot pushed a commit to golang/vulndb that referenced this issue Dec 6, 2022
@tatianab
deploy: add Validate step to deploy script
65654d5 
Adds two steps, CopyExisting and Validate to the vulndb deploy script,
to ensure that the existing and new databases are valid internally
and with respect to each other. Deploy will not proceed if validation
fails.

(Reinstates https://go-review.git.corp.google.com/c/vulndb/+/452771
with fix. The bug was a missing "-c" flag in CopyExisting).

For golang/go#56417

Change-Id: I0ef8e38a6679225e8b7b02a9b4b39c18a975ba9a
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/455315
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Jonathan Amsterdam <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
gopherbot pushed a commit to golang/vulndb that referenced this issue Dec 12, 2022
@tatianab
internal/database: replace Generate logic with New and Write
f9476be 
Replaces existing Generate logic with refactored New and Write functions,
which have the equivalent behavior.

For golang/go#56417

Change-Id: Ie01c0c77e93c779c717e89acecb81fc00dd4cfbe
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/453176
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Jonathan Amsterdam <[email protected]>
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
Reviewed-by: Jenny Rakoczy <[email protected]>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/457017 mentions this issue: internal/database, cmd/gendb: refactor Generate for unit testing

gopherbot pushed a commit to golang/vulndb that referenced this issue Dec 13, 2022
@tatianab
internal/database, cmd/gendb: refactor Generate for unit testing
ffba2fd 
Modify Generate to take in a *git.Repository instead of a directory
string, so it can be more easily unit tested. Add a unit test.

For golang/go#56417

Change-Id: I3eaa84b41568e9582ac1f16be8c979d7b71d5ad3
Reviewed-on: https://go-review.googlesource.com/c/vulndb/+/457017
Run-TryBot: Tatiana Bradley <[email protected]>
Reviewed-by: Damien Neil <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>
@tatianab tatianab closed this as completed Jan 9, 2023
@golang golang locked and limited conversation to collaborators Jan 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tatianab tatianab

Labels
FrozenDueToAge vulncheck or vulndb Issues for the x/vuln or x/vulndb repo
Projects
None yet
Milestone
vuln/unplanned
Development

No branches or pull requests
2 participants
@tatianab @gopherbot