-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Separate import and improve operations #525
Conversation
331f70c
to
925d0ef
Compare
Feel free to review now. It's not perfect yet, so please consider these points
|
@Hritik14 re:
If you have a default of 100% this becomes a non-issue? I kinda like to avoid near-duplicated data structures. |
Note that using a common data structure does not mean that we have only importers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
... see some review comments inline.
@@ -0,0 +1,37 @@ | |||
import dataclasses |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For consistence, please add a license header. We still need to update the license everywhere per #277 , but best is to have whatever we have now consistently across all code files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noted. How about I add all the license in the last commit ? Might make things clearer. Let me know
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
A typo could lead to inferences to have 100% confidence in that case...
Importers and improvers look similar for now mostly because improvers don't really contain anything yet. There are various subclasess for importers like GitDataSource or OvalDataSource with corresponding helper functions to parse advisory data from an actual data source, whereas in case of improvers, they only get the entries from the existing data thus no advisory parsing would be required. On the other hand, improvers could benefit with functions enabling them to access data from the database in a better manner (than done here, I'll be removing direct database interactions in improvers). Other than that an
If it's a factual data then then by definition it's a fact thus 100% confidence, right?
Inserting into the database (be it importer or improver) is essentially the same in our structure. Wouldn't we finally need a common data structure anyway ? |
Current commit brings breaking changes. Feel free to review but do ignore the TODO's and pointers mentioned in the latest commit message. This is a step towards https://github.com/nexB/vulnerablecode/wiki/WeeklyMeetings#meeting-on-tuesday-2021-08-17-at-1400-utc |
6b706c0
to
1834ef0
Compare
1834ef0
to
7f9fbb4
Compare
Trying out adding a real improver other than the default one now. It is not ready yet. Suggestions are welcome for the WIP commit but please avoid it in any strict review currently. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider using to_dict
rather than directly to_json
... a to_json
becomes then a thin wrapper on the serialization with to_dict
See also my other comments inline.
@@ -0,0 +1,37 @@ | |||
import dataclasses |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
The refactors are based on aboutcode-org#525 (review) - class Inference: order should be the logic order from most important to least important fields - class Improver: docs - data_source.py: Use to_dict() than json() - models: Advisory: docs - improvers/nginx.py: Removed, relevant improvers will now reside inside importer module. In this case, importers/nginx.py - black formatting Signed-off-by: Hritik Vijay <[email protected]>
7f9fbb4
to
fbe993b
Compare
Using the OSV design has some caveats. Given data like:
It is impossible to directly get one fixed version for a range of affected versions. |
@Hritik14 re:
Is this a real case rather than some made up example? Now things would be different if there were more than one range of course... but then that would be something entirely different? |
Here is the gist of our discussion:
|
Turns out we can parse nginx versions smoothly. Here's more on it #553 |
I do not like the batch processing of multiple Advisory at once in a single transaction... this may be an optimization we will need in the future, but not now. Here is a suggested updated way:
|
fbe993b
to
8897af9
Compare
The refactors are based on aboutcode-org#525 (review) - class Inference: order should be the logic order from most important to least important fields - class Improver: docs - data_source.py: Use to_dict() than json() - models: Advisory: docs - improvers/nginx.py: Removed, relevant improvers will now reside inside importer module. In this case, importers/nginx.py - black formatting Signed-off-by: Hritik Vijay <[email protected]>
d2a2167
to
56866f7
Compare
Signed-off-by: Hritik Vijay <[email protected]>
Signed-off-by: Hritik Vijay <[email protected]>
Signed-off-by: Hritik Vijay <[email protected]>
Signed-off-by: Hritik Vijay <[email protected]>
Signed-off-by: Hritik Vijay <[email protected]>
Signed-off-by: Hritik Vijay <[email protected]>
e25b437
to
318663e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ready to merge! ... see a few nits for your review.
affected_version_range = VersionRange.from_string(affected_pkg["affected_version_range"]) | ||
fixed_version = affected_pkg["fixed_version"] | ||
if fixed_version: | ||
# TODO: revisit after https://github.com/nexB/univers/issues/10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fixed now.
H: No..... aboutcode-org/univers#10 (comment)
761fdab
to
61fea9e
Compare
Signed-off-by: Hritik Vijay <[email protected]>
61fea9e
to
06c793c
Compare
Fixes are in accordance with the final review for this PR done at : aboutcode-org#525 (review) Signed-off-by: Hritik Vijay <[email protected]> Co-authored-by: Philippe Ombredanne <[email protected]>
06c793c
to
ce87783
Compare
from_dict is a factory, should be a classmethod Signed-off-by: Hritik Vijay <[email protected]>
The entire view is messed up. Will probably need a rewrite Co-authored-by: Philippe Ombredanne <[email protected]> Signed-off-by: Hritik Vijay <[email protected]>
Fixes are in accordance with the final review for this PR done at : aboutcode-org#525 (review) Signed-off-by: Hritik Vijay <[email protected]> Co-authored-by: Philippe Ombredanne <[email protected]>
d01ebc7
to
33d0c0d
Compare
Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
!! Breaking Changes !!
Entire database model is restructured.
improvers
Note: All changes in the importers are only reflected in the nginx importer while this PR is in draft stage.
Knows defects (in current PR):
AdvisoryData
has no bodyKnows defects (to be solved in different PR):
Signed-off-by: Hritik Vijay [email protected]