Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New plugin architecture #885

Merged
merged 122 commits into from
Feb 7, 2018
Merged

New plugin architecture #885

merged 122 commits into from
Feb 7, 2018

Conversation

pombredanne
Copy link
Member

This is for #787 and a few more tickets.

There are a few changes on what is reported:

  • scanning a single file or empty dir will always return a value even with strtip-root
  • the root directory has a proper entry with its file info
  • the counts are correct and do not need to load the whole set of scan results in memory, using instead the lighter weight Codebase and Resources tree
  • the scan cache has been folded in the Codebase and Resource code. This will enable a simpler implementation for Allow cache location configuration  #357 and Allow cache location configuration #878

I want to close #856 in favor of this heavily updated plugin and CLI processing architecture.

@yashdsaraf and @haikoschol ping!

yashdsaraf and others added 30 commits January 17, 2018 22:59
@yashdsaraf @pombredanne
Support multiple options for pre-scan plugins #787
065d0a3 
Signed-off-by: Yash D. Saraf <[email protected]>
@SaravananOffl @pombredanne
Added MakeHuman Exception License
6095fb1 
Signed-off-by: Saravanan G <[email protected]>
@SaravananOffl @pombredanne
formatted LICENSE file and removed rules
4f0475c 
Signed-off-by: Saravanan G <[email protected]>
@pombredanne
Refactor post scan plugins so they can have command line options
5a304fd 
Signed-off-by: Haiko Schol <[email protected]>
@pombredanne
Improve documentation of PostScanPlugin.process_results()
f41bca6 
Signed-off-by: Haiko Schol <[email protected]>
@pombredanne
Rename get_click_options() to get_options()
403b1b2 
Signed-off-by: Haiko Schol <[email protected]>
@pombredanne
Allow post-scan plugins to take options other than boolean flags
5f4c626 
Signed-off-by: Haiko Schol <[email protected]>
@pombredanne
Use a common BasePlugin class for all plugins #787
612cf68 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Inline fileutils imports #787
b765432 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use proper import in prep for Python 3 #787
7f9b74c 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Update help text test to match latest code #787
cff89ad 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use class method for get_plugin_options #787
75908e7 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use scans_cache_class not scan_cache_class #787
586d392 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Extract function to get_cache_dir #787
66ec6dc 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Ensure all tests pass #787
81780ea 
 * more ScanOption to scancode.__init__.py
 * add active_scans to BasePlugin initializer
 * ensure plugins work consistently
 * try to not make ignores a special case

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Remove junk print statements
d9f8ec1 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Streamline core CLI and plugins processing
4f143b7 
 * use structured namedtuple to track Scanners and CommandOptions
 * rework how default scan and selected options are handled using these
   tuples to streamline the corresponding code and the way selected
   options are reported in the scan results (at least for JSON)
 * add is_enabled method to Plugin
 * ensure the help of options is used for each option, and not a global
   help for all the options of a given plugin
 * add new plural option for each scan (such as --licenses) that maps
   exactly to the key used in the scan results. Use plural variables
   throughout to simplify the code
 * use kwargs for the main cli.scancode function for all all options and
   no longer use named arguments: args can come from any plugins and
   are not known until runtime.

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use resource_iter everywhere #787
7ef1cba 
 * fileutils.py: remove file_iter and dir_iter as they were not really
   used/useful

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Do not use on-disk file log. Improve pre-scan plugins #787
a6446c9 
 * refactor pre-scan plugins to use the same approach as all other
   plugins and not make ignoring a special case
 * cli.py and cache.py: remove the on-disk file log and use an in-memory
   list instead

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add new extract_zip_raw test function
0c3625b 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Bump attrs and add typing library #787
9884c9f 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Make info a regular scan and other CLI loop reorg #787
f8eabb3 
 * move and other file info code such as get_file_infos and Resource to
   new resource.py module. Make infos a proper, regular scan
 * Do use content checksum for scan cache, instead use plain path
   checksum. This enabled making info a proper scan, since content
   checksums are no longer needed for cache handling.
 * remove info cache and use only one cache file per file for its
   details
 * Reorganize scan steps to make them co-located and explicit: inventory
   pre-scan, scan and post-scan are not a set of clean loops
 * move most of the CLI output in the main scancode function
 * Add name class attribute to plugins to support UI reporting

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Remove diag arg from get_resources() #787
40835fd 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add function to skip first or last iterable item #787
3be4577 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Change interruptible returned values #787
b5e627c 
 * now return (error string or None, value or None)
 * this is cleaner than (success, value) where the value could be
   either the callble result or the error string 

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Ensure api function return a list #787
e9cd4e5 
 * and not a generator, so they can serialize OK without wrapping

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Use latest attrs and typing #787
59bef89 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Add file base_name to info scans #787
683ca26 
Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Replace path_to_bytes/unicode by fsen/decode #787
0279764 
 * This was a useless wrapper
 * Also improve imports locality and other minor refactorings

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Improve base plugins design #787
4ba3b77 
 * use generic get_plugins for all plugin types rather than
   a function with a unique name
 * use Resource objects throughout (except for format plugins)
 * enhance BasePlugin base class: the methods are now setup() and
   teardown() to be executed at init/exit time, process_resource() to
   process a single Resource and process_codebase() to process a whole
   Codebase

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Copy link
Member Author

ok, so I am merging now. We can take it from develop for any other quirks.
@haikoschol @yashdsaraf thanks you for starting this in the first place. It took a while but I feel we are in a better place now!

@pombredanne pombredanne merged commit d5c583a into develop Feb 7, 2018
This was referenced Feb 7, 2018
Closed
Closed
Closed
Closed
Closed
Open
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Feb 16, 2018
@sschuberth
Upgrade ScanCode to the latest development version
762d5aa 
This includes a massive refactoring with lots of performance
improvements. For details see

aboutcode-org/scancode-toolkit#885
@sschuberth sschuberth mentioned this pull request Feb 16, 2018
Merged
sschuberth added a commit to oss-review-toolkit/ort that referenced this pull request Feb 16, 2018
@sschuberth
Upgrade ScanCode to the latest development version
4519ccf 
This includes a massive refactoring with lots of performance
improvements. For details see

aboutcode-org/scancode-toolkit#885
@pombredanne pombredanne deleted the 787-new-plugin-architecture branch February 19, 2018 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@haikoschol haikoschol haikoschol left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@pombredanne @JonoYang @haikoschol @yashdsaraf @SaravananOffl