Continuously watch a PURL of interest, aka. purlwatch

[pombredanne]

Given a PURL, I would like to have a new API endpoint to register "interest" in this PURL. Once this is "registered", this PURL should continuously be "watched" for updates to this Package URL (all versions), polling for new versions on schedule, like on daily, or weekly basis.
When a new version is discovered, we should run the steps to collect metadata and trigger new scancode pipeline runs on each update.
Optionally we could also work from a PURL and have a flag to expand to either the previous and next versions, or to all versions of a PURL, or all the future versions.

The solution could include these elements:

• Create the models to store the registration for interest in a PURL. This is tracked in Watch for packages (model and implementation) #244 #271
• Create a REST API end-point that accepts a PURL input with a version and registers/stores interest in this PURL. Also in Watch for packages (model and implementation) #244 #271
• Create a mechanism to schedule jobs that run periodically in the context of the PurlDB using a task scheduler (such as RQ) in order to eventually watch for new versions This is in Watch for packages (model and implementation) #244 #271
• Create function(s) to collect the new package versions of registered PURLs on a regular basis (avoiding collisions on DB duplicates that are already collected). This is in Watch for packages (model and implementation) #244 #271 and in FetchCode
• Create function(s) to trigger new background runs to collect collect metadata and trigger new scancode runs on new versions if requested. Watch for packages (model and implementation) #244 #271
• Extend to more versions of a PURL Create function to expand data collection to more versions of a PURL.  #368
• Create management command to watch for PURLs Add management command to watch all PURLs #268

[pombredanne]

Carried over from #88 closed in favor of this one:

I would like to ensure that PurlDB has up-to-date packages for my projects.
For this I would like to somehow register interest for some package version, and purlDB would go fetch newer version of these packages on a regular basis and also would fetch the newer versions of the whole dependencies set of these packages all the way down. 🐢🐢🐢🐢🐢🐢🐢

See also #87

Other things to consider carried over from #64 :

• Trigger mining, matching, scanning of the dependency trees, possibly with priority for runtime dependencies over others
• Trigger mining, matching, scanning for newer versions of packages that were matched
• Trigger mining, matching, scanning for older versions of packages that were matched
• Create a watcher that looks periodically for new versions of a package that was requested, may be by registering interest in this package

We have ways to trigger the indexing of a package version range now, but nothing specifically to do anything periodically.
Some inspiration:

• Debian watch files
• Fedora Bohdi

[pombredanne]

In a the future, we could refine the watch to only look for versions after a version or after a release date to avoid collecting a badzillion of old, historical, unused versions.

This could take the form of:

• a date field that would would be the earliest_release_date to filter out older versions
• a minimum version field that would would be the minimum_version to filter out older versions, with the caveat that popular can have v3.1 released after v4.0, e.g., multiple release "lines" active at a time.
• only consider versions created since last watch and always set the last watch to a default value that could today or in a recent past.

[pombredanne]

There is a design that needs some thinking: How do we select which PackageWatch to process?

1. We could store the next watch date. This is would need to be done in save() and is problematic if there is any exception. But this is simple afterwards to filter only the records that have next watch date less or equal to today

2. We could use a DatetimeField and DurationField and do a date computation in the queryset filter as an expression but this is not portable beyond PostgreSQL per https://docs.djangoproject.com/en/5.0/ref/models/fields/#durationfield

3. We could not filter on dates but instead compute if a watch is eligible in a run in a loop. The watch interval would stored as plain integer in days. We would loop this way:

• we do a simple filter on active watches.
• then we iterate and for each we compute the next watch date as: last watch date + watch interval and skip a record if this is not less or equal to now.

This would be portable beyond PostgreSQL but requires more processing, as most watches would need to be look at once on each run and would not use a queryset expression.

Here the last watch date is a DatetimeField and the watch_interval would be a number of days between watches.

4. we could store instead a POSIX timestamp as an integer for the last watch date and store a number of seconds in an integer as the watch interval (but no less than one hours or 3600 seconds) and use a an arithmetic expression in the queryset. This would be like 2. but portable beyond PostgreSQL. But display would need to convert the timestamp valud back to a datetime.

Let's go with 3. for start as we can easily migrate to other options afterwards: this is the most expressive and easiest to test:

[pombredanne]

Another design point is the processing:

The thing could start with either a cron-like task in RQ or a command line management command.
This could be a function that can be re-used in both. A command line would be helpful for testing, and a task would be fine for production. This could run daily for a start (but it could run more than daily too).

In all cases, this would select packages eligible for watch per #244 (comment) and then could either:

1. create one RQ task for each package to watch
2. OR, create one PriorityQueue entry for each package to watch
3. OR, in the future we could decide to batch check for the new versions of a whole ecosystem at once, possibly excluding the things not watched.

Using 2. is probably best for now to avoid duplicated entries.
We should also ensure that there is no pre-existing queue entry for this watch. This could be done either by querying the queue first OR/AND during the watch run checking that we did not watched already in the watch interval.

[pombredanne]

This is completed now.

We did extend PurlDB with a new “watch” API endpoint. Given a PURL, you can register "interest" in this PURL and continuously "watch" for updates to this Package URL (ignoring versions), polling for new versions on scheduled based on a per-watched defined interval.

To back this, we integrate a queue and scheduler (using RQ). When the watch runs on schedule and a new package version is available, we further collect metadata and trigger new indexing scans in the background.

• Working system with a REST API end-point that accepts a PURL input with a version and register interest in this PURL.

  • We designed a new model to track package watching and its backing functions: https://github.com/nexB/purldb/blob/21c24f4a47a03c2f47a8661c23f5330a0ecf10ab/packagedb/models.py#L1226
  • We exposed a new API endpoint at api/watch/ https://github.com/nexB/purldb/blob/21c24f4a47a03c2f47a8661c23f5330a0ecf10ab/packagedb/serializers.py#L335
  • We also created a watch_packages command to enable exercising package watch from the CLI
    https://github.com/nexB/purldb/blob/21c24f4a47a03c2f47a8661c23f5330a0ecf10ab/packagedb/management/commands/watch_packages.py

• A mechanism to schedule jobs that run periodically in the context of the PurlDB using a task scheduler (such as RQ).

  • We implemented using RQ (as already used in ScanCode.io) with https://github.com/nexB/purldb/blob/e317b56be475716913ab2688573438e8d18ca6c2/packagedb/schedules.py

• Functions to collect the new package versions of registered PURLs on a regular basis (avoiding collisions on DB duplicates already collected)

  • We designed a way to schedule the watch intervals and run periodic watch tasks
    https://github.com/nexB/purldb/blob/e317b56be475716913ab2688573438e8d18ca6c2/packagedb/tasks.py . We also enabled the ability to merge data and the models have constraints to avoid duplication.

• We now have functions to trigger new background runs for tasks to collect metadata and run indexing

• We designed a way to schedule the watch intervals and run periodic watch tasks https://github.com/nexB/purldb/blob/e317b56be475716913ab2688573438e8d18ca6c2/packagedb/schedules.py

• We have functions to expand data collection to all versions of a PURL.

  • To support watching package, a key feature is to collect new versions of packages. We added support for this in FetchCode and in PurlDB and can expand collection to newer or all versions of a PURL.
  • https://github.com/nexB/fetchcode/blob/d0a3fa9bb56dc3a77f7d3d7bd5b8d0e40c7a8612/src/fetchcode/package_versions.py
  • https://github.com/nexB/purldb/blob/main/packagedb/package_managers.py

To test this feature:

• Locally:

1. Install PurlDB
   or request access to a private demo instance at https://private.purldb.io/api

2. CLI-based watch: From the command line, in the activated virtualenv, run the command to create a new watch for a PURL such as
   ./manage_manage_purldb.py watch_packages --purl pkg:maven/org.apache.logging.log4j/log4j

3. Go to https:///api/watch/ to view existing watched packages and also see the results of indexing this

• Remotely:
  Alternatively, since we also have an endpoint for watch, you can simply add the new PURL here https://private.purldb.io/api/watch. (Access on request).
  Once a new PURL is added, a watch is scheduled immediately, and thereafter, it will be watched periodically as per the specified watch interval or every 7 days by default. You can view the details of a watch by going to https://private.purldb.io/api/watch/{PURL} there we can find the last watch date, errors and so on.
  Note that for every new watch, an immediate watch is automatically triggered, for already existing watch we can only change the watch interval but it can not be less than 1 day. And If a PURL doesn't already exists in PurlDB then it will index all the version of that PURL as part of the watch feature.
  Specifically the API behaves this way:
  • a GET request on the endpoint https://private.purldb.io/api/watch/ will give the list of existing watch.
  • a POST request on the endpoint https://private.purldb.io/api/watch/ with the below given data will create a new watch for requested package_url. For instance, posting this to the end point:

{
    "package_url": "pkg:pypi/fetchcode",
    "depth": 3,
    "watch_interval": 7,
    "is_active": true
}

You may want to use another PURL as this may already been watched