Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plugins #552

Closed
4 tasks done
pombredanne opened this issue Mar 10, 2017 · 2 comments
Closed
4 tasks done

Plugins #552

pombredanne opened this issue Mar 10, 2017 · 2 comments
Assignees
@yashdsaraf
Labels
core and api new feature plugin

Comments

@pombredanne
Copy link
Member

pombredanne commented Mar 10, 2017 •
edited
Loading

We need a plugin architecture such that it is easier to add new scanners, new formats and other features in a decoupled way.

At a high level, this is similar to Firefox, Chromium or Eclipse plugins.

The goal is to create a decoupled architecture such that output formatters and scanners (and in the future deduction tools) can:

  1. be their own project/library and be plugged in ScanCode at installation time

  2. for scanners and scan "pipeline processors" , they can possibly be coded in any language beside Python with Python being a special privileged language in any case. And accept an ABC data doc in and spit an ABC data out.

See for ABCD:
https://github.com/nexB/aboutcode/blob/a84e207bbe579dd9b419cc52bbfd00239f5beeb9/aboutcode-data/README.rst

For instance for packages scanners, a plugin approach such that new package manifests parsers are implemented as external libraries, eventually using different languages than Python to reuse the many parsers natively available (Ruby, Js, Java, etc) would make a lot of sense. .

Formatter/output should be the first to be implemented as plugins for a start, then eventually all scanners and more should be implemented as plugins. See #381 for format plugins.

There stages where we can plugin things are, in this order of implementation priority:

Other points:

  • A plugin should be able to add new command line options even if minimally.

  • The ability to have plugins in other languages than Python would be great: they would be spawned in their process and receive a path to scan and/or an ABC data packet and return an ABC data packet after having done their stuff.

  • Some Python tools on Pypi such as pluggy, yapsy, stevedore and the likes as well as possible click extensions may be candidate for this.
@pombredanne pombredanne mentioned this issue Jun 15, 2017
Merged
@yashdsaraf yashdsaraf mentioned this issue Jun 20, 2017
Closed
pombredanne added a commit that referenced this issue Jul 19, 2017
@pombredanne
Remove writers module. Not needed #552
186abd2 
Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 19, 2017
@pombredanne
Use shared code for JSON. Separate tests #552
9568bb5 
 * Use a shared function fro both JSON plugins. Also rename functions
   and add some documentation.
 * Create a separate test module with its own independent test files. 

Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 19, 2017
@pombredanne
Ensure that a custom template file is allowed #552
9a73886 
Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 19, 2017
@pombredanne
Refactor html format plugins #552
9f8ef71 
 * merge html and html-app in one module
 * rename functions and entry points
 * create a separate test module for these
 * rename format to temaplted for utilities shared with html and
   custom templates

Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 19, 2017
@pombredanne
Add doc and simplify CSV and SPDX plugins #552
278dbfa 
 * Rename entry points
 * Refactor tests in separate modules
 * Ensure all tests pass
 * cli.py: ensure output file is opened 'wb' as unicodecsv chokes
   otherwise. Clean up imports nd refine code formatting

Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 20, 2017
@pombredanne
Move all template-related code to format_html #552
ea1e44a 
Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 20, 2017
@pombredanne
Rename format_html to format_templated #552
5f14d15 
 * also move cli.py code that creates a custom output to this module
   and use a common function with the html plugin

Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 20, 2017
@pombredanne
Refactor plugincode #552
0c82355 
 * rename the output module, hookspecs and entrypoint
 * expose a get_format_plugins() function as the man API for cli.py
   to use
 * remove the unused scan_proper* code
 * move all output plugin initialization code to plugincode instead of
   globals in cli.py
 * refine the --format CLI option help text 
 
Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 20, 2017
@pombredanne
Add message telling that json2csv is deprecated #552
ceeb164 
Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jul 20, 2017
@pombredanne
Wrap loading setup entrypoints in a function #552
fb29d6c 
 * otherwise there is a circular import and conflicts between imports
   and setuptools entrypoints loading

Signed-off-by: Philippe Ombredanne <[email protected]>
@yashdsaraf yashdsaraf mentioned this issue Jul 21, 2017
Closed
@pombredanne
Copy link
Member Author

with #639 we now have support for pluggable output formats. This ticket should be used to track the high level feature completion. @yashdsaraf please create separate ticket for each pluggable features (e.g. pre-scan, scan, post-scan)

This was referenced Jul 22, 2017
Closed
Closed
Closed
@yashdsaraf yashdsaraf mentioned this issue Jul 27, 2017
Merged
pombredanne added a commit that referenced this issue Jan 17, 2018
@pombredanne
Add new scan stage and plugins #787 #552 #698
861e9a1 
 * A scan plugin handles the scan of one resource at a time
 * Existing scans have been reworked as plugins

Signed-off-by: Philippe Ombredanne <[email protected]>
pombredanne added a commit that referenced this issue Jan 17, 2018
@pombredanne
New plugin and CLi architecture #787 #552
60bdb24 
This is a major update of the code scancode processing. 
The scancode CLI has been reworked such that:

   * each plugin can have as many options as needed
   * plugins are organizaed by stages. New stages have been added
     for output_filter, scan and housekeeping
   * plugins work on a Codebase or a Resource which is an in-memory
     representation of a scanned filesystem

Signed-off-by: Philippe Ombredanne <[email protected]>
@pombredanne
Copy link
Member Author

With the latest merge of #885 this concludes the creation of plugins! Closing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yashdsaraf yashdsaraf

Labels
core and api new feature plugin
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pombredanne @yashdsaraf