Feature: purl2sym: Expand support for multiple tools

[pombredanne]

This is to extend the ScanCode.io pipeline in #312 to support multiple tools, including two pure Python tools, two native tools, as well as collecting symbols from OpenSSL, Busybox, and uCLibc. This includes updating the scanning architecture of PurlDB to accommodate multiple ScanCode.io worker systems (whole machines) and expose a queue API where ScanCode.io instances can pick a scanning job to run symbol collections. A new API endpoint will accept a PURL and return a list of all known PURLs and download URL for all versions of the PURL, and another will provide status on the symbols collections that are pending.

• ScanCode.io: Support multiple tools adding gettext for strings scancode.io#1108
  • Collect symbols with ctags source-inspector#2
  • ScanCode.io: ScanCode.io: Collect symbols in scan (from source code and store in scan results) scancode.io#1106
  • Extract strings from source code with xgettext source-inspector#3
  • ScanCode.io: Support multiple tools adding gettext for strings scancode.io#1108
• ScanCode.io: Support multiple scan workers systems scancode.io#1107
• PurlDB: Provide a list of all known package versions #333
  • Batch 0: Collect symbols and strings from OpenSSL, Busybox, and uCLibc #366
• PurlDB: Monitor scanning jobs #334
• Add support for Python symbol tools
  • Extract source symbols and strings using Pygments #399
  • Extract source symbols and strings using https://tree-sitter.github.io/ #400

[keshav-space]

We now support native tools like Universal-ctag and xgettext for symbol and string collection. We also support Python tool Pygments and tree-sitter.

The simple way to test this is to follow the below steps after installing PurlDB (https://github.com/nexB/purldb?tab=readme-ov-file#installation):

1. Go to /api/collect/ add PURL for indexing. ex: /api/collect/?purl=pkg:generic/[email protected]&addon_pipelines={TOOL_PIPELINE}
2. Once the indexing has completed go to /api/resources/and filter out resource based on PURL to see the symbols and strings for each resource in extra_data field. ex: /api/resources/?purl=pkg:generic/[email protected]

Replace the {TOOL_PIPELINE} in above request with your desired Pipeline from the table shown below.

Pipeline	Tool	Output
collect_symbols_ctags	Universal Ctags	symbols
collect_strings_gettext	Gettext	string literals
collect_symbols_pygments	Pygments	symbols, string literals and comments
collect_symbols_tree_sitter	Tree-Sitter	symbols and string literals

Tip

Follow this tutorial to index the symbols and strings for a PURL/Package using various different addon pipelines. https://aboutcode.readthedocs.io/projects/PURLdb/en/latest/how-to-guides/tutorial_symbol_and_string_collection.html