Scan queue updates

[JonoYang]

We are revamping the scan queue. The new process will have scancode.io workers get jobs from the purldb by visiting an API endpoint that returns the next available scan queue entry to be worked on. (api/scan_queue/get_next_download_url). When the worker is done with the scan, it will post the results back to api/scan_queue/submit_scan_results.

TODO:

• Guard indexing endpoints with special API key
• Update scan queue to create new scan queue entries whenever we reindex a package, rather than reusing existing entry
• Create scancode.io management command that gets jobs from api/scan_queue/get_next_download_url and sends results back
  • Get download url
  • Create project, get input, run pipeline
  • Return results

Consider:

• POST scancode io worker info before getting download url so we can tell which worker took what job

[404-geek]

Hi @JonoYang , I would like to work on this specific issue if not completed, and would like your guidance to understand some parts of it.

[JonoYang]

@404-geek Thanks for your interest in the issue, but I'm currently working on this.

[404-geek]

Hi @JonoYang Is the above issue related to

https://github.com/nexB/aboutcode/wiki/GSOC-2024-Project-Ideas/#purldb-add-improved-scan-queue-with-multiple-scio-instances

because I was thinking to have a actual queue system implemented using message brokers or Redis Queue and keeping the API pipeline separate which can bring scale in between the communication of the services.

[JonoYang]

This has been completed. The scan queue has now been updated such that the ScanCode.io workers get the next package and pipelines to be run on the package from the new API endpoint /api/scan_queue/get_next_download_url/. These are the default pipelines run on a Package: https://github.com/nexB/purldb/blob/main/minecode/model_utils.py#L30 . The ScanCode.io worker then fetches the package, runs the specified pipelines, and then send the results back to PurlDB to the /api/scan_queue/update_status/ endpoint.

Relevant PRs:

• Expose new scan queue to many workers #49 #290
• Create purldb scan worker  scancode.io#1078

To test this feature:

• Deploy the main branch of PurlDB
• Create a PurlDB API key for the ScanCode.io worker using the management command create-scan-queue-worker-user
  • docker compose -f docker-compose_purldb. exec web bash
  • python manage_purldb.py create-scan-queue-worker-user <username>
• Deploy this branch of ScanCode.io (https://github.com/nexB/scancode.io/tree/update-docker-compose). In the .env file, set the PurlDB integration environment variables to the PurlDB instance deployed earlier, using the API key generated.
• Run the ScanCode.io scan queue worker by running docker compose -f docker-compose.purldb-scan-queue-worker.yml up
• Add some Packages to the PurlDB using the /api/packages/index_packages/ endpoint
• The scan queue worker should be fetching and scanning the packages from the scan queue