Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Make W3C script use the W3C API as source
This is a complete re-write of the W3C update script to switch from the still-maintained-but-deprecated `tr.rdf` file to the more complete and current W3C API instead. What changes? Essentially nothing substantial in terms of data, but: - When an entry is updated, the `source` property will target the API endpoint from which the data was pulled, such as: `https://api.w3.org/specifications/_shortname_` - The W3C API has a few additional statuses that were not reported in `tr.rdf` such as `DNOTE`, `FPWD`, `LCWD`, and the registry statuses. - The script fills out properties more systematically for versions - The order of the properties for each entry is not always exactly the same as the order generated by the previous script. The first time the script runs, it will: - Fix a few entries of very old specs in Specref, for which the title is not the title of the actual spec. - Add entries for the draft registries published by a couple of groups. - Complete a few entries with additional versions that did not exist in `tr.rdf` for some reason. - Create consistent `obsoletes` properties, as Specref contains a few `obsoletes` properties that don't have a matching `obsoletedBy` property. The script only updates recently published entries by default. That is, it does not attempt to refresh the whole list. That's needed because the W3C API follows the HAL convention: https://en.wikipedia.org/wiki/Hypertext_Application_Language One consequence is that each API request returns only a minimal amount of information, and re-generating the entire `w3c.json` file requires sending ~30000 requests, which would be at best impractical to do on an hourly basis, all the more so because the W3C API server has some rate limits rules in place (6000 requests every 10 minutes). More importantly, that would be a waste of resources as data essentially never changes once published. Thus the script takes an incremental approach instead and only refreshes: 1. Specifications recently published... where recently published means specifications published since the newest publication date known to Specref minus 2 weeks by default. The "minus 2 weeks" is meant to catch data fixes that are sometimes made shortly after publication. 2. Specifications for which the base info (title, URL) is not aligned with the W3C API. That's meant to fix the data in Specref during the transition, and to catch further updates that could be made to the W3C API once in a while. All in all, a typical update should send ~500 requests to the W3C API. The code throttles requests to 1 every 100ms. Running the script should take ~1-2 minutes. A more thorough refresh may be forced by calling the script with a date as parameter (format YYYY-MM-DD, with month and day optional). The date gets interpreted as the synchronization point. For example, to refresh all specs published since 2023, run: ``` node scripts/w3c.js 2023 ``` To force a "full" refresh (any year prior to 1995 would work): ``` node scripts/w3c.js 1995 ``` A full refresh sends ~30k requests to the W3C API and may take >2h. I suggest to run a full refresh manually once, shortly after this script starts being used, and then to run it again every few months to capture potential fixes that might have been made to the data in the meantime. Running that full refresh will also be useful to fix the few `obsoletedBy` properties that are not fully correct, and to move a few `hasErrata` links to the right spec version, as some entries have these links at the root level of the entry in Specref, whereas the latest version is no longer the REC that linked to the errata. I worked with @deniak to fix and complete the data in the W3C API where Specref had more correct info. I also updated entries that contained incorrect info in Specref. The script contains a number of comments to explain the different cases that need to be handled to be able to fully map the data in the W3C API with the data in Specref. There will remain a few entries where the mapping is somewhat imperfect, notably when the shortname of a spec evolved from a level-less shortname to shortname with level, and sometimes back to a level-less shortname (examples include `user-timing`, `performance-timeline`). There are also a few entries for old specs that are flagged as retired in Specref (`isRetired: true`) but not in the W3C API. Mismatches are reported to the console as warnings. These should be addressed over time. The script preserves the information in Specref in any case. The script also preserves the information in Specref in case of transient network errors while fetching info from the W3C API. The new overwrites rules are needed during the transition (the changes need to be made as the same time as the data gets updated), but can be dropped afterwards. They affect specifications that switched from a shortname without a level to a shortname with a level. Longer term, these should be handled through the notion of specification series (see tobie#811).
- Loading branch information
