Add/update cocoapods and pypi download_url (and other) support to fetchcode/package.py

[johnmhoran]

In connection with a purl2url issue in packageurl-python, we cannot add download URL support for cocoapods or pypi in purl2url.py because the process involves network calls, and consequently will need to add/update that support in fetchcode/package.py.

[johnmhoran]

@pombredanne I have a rather detailed question for you re metadata queries (i.e., using fetchcode/package.py) for a PURL with no version. The issue: missing podspec.json files, which we use to populate the metadata object for each version.

Sometimes the default URL we use, the raw.githubusercontent.com/CocoaPods/Specs... URL from cocoapods.py (with /blob deleted), returns a 404 for one or more of the tags I retrieve for that PURL from its GitHub repo. I currently print a statement to that effect in the console and add it to a new .log file in package.py, which I can access from the metadata command in purlcli.py and include in the headers/errors section of the metadata JSON output. Simple and clean, but can result in cocoapod tags for which we report no data.

SCTK's packagedcode/cocoapods.py also has a "backup" podspec URL: https://cdn.cocoapods.org/Specs/{hashed_path}/{name}/{version}/{name}.podspec.json. I don't currently use this as a backup in my code but could. FWIW, using sqlite as an example, both the default and the backup URLs return a 404 for the tags I've tried. BTW, these are the tags for sqlite: ['0.1.23', '0.1.22', '0.1.20', '0.1.18', '0.1.16', '0.1.15', '0.1.14', '0.1.13', '0.1.12', '0.1.11', '0.1.10', '0.1.9', '0.1.8', '0.1.7', '0.1.6', '0.1.5', '0.1.4', '0.1.3', '0.1.2', '0.1.1', '0.1.0'].

Finally, a 3rd podspec URL we might have available: sometimes, the cocoapods.org page has a podspec link to a location in the pod's GitHub repo. sqlite is an example: https://github.com/CocoaPods/Specs/blob/master/Specs/4/9/7/SQLite/0.1.23/SQLite.podspec.json. And sometimes that GitHub URL will also return a podspec.json for some of the other tags for that pod -- but not always.

How would you like this search, analysis and reporting process to be handled? For each tag, exhaust all 3 URLs if needed in case one of them has a podspec.json for the tag in the list, and do that for each tag in the list of tags from GitHub (which can be numerous)?

[johnmhoran]

Note that the 3rd podspec URL example uses a different hashed_path from that used in the 2 cocoapods.py podspec URLs, e.g.

• https://raw.githubusercontent.com/CocoaPods/Specs/master/Specs/7/8/6/sqlite/0.1.23/sqlite.podspec.json
• https://cdn.cocoapods.org/Specs/7/8/6/sqlite/0.1.23/sqlite.podspec.json
• https://github.com/CocoaPods/Specs/blob/master/Specs/4/9/7/SQLite/0.1.23/SQLite.podspec.json

I don't (yet) know how that latter hashed_path is calculated....

[johnmhoran]

^ It's the familiar MD5 hash we already use, applied to a cocoapod name that does not reflect the "official" uppercase/lowercase name as it exists in the cocoapods.org page for that pod, e.g., incorrect upper/lowercase of the input PURL in the PURL CLI metadata command (which of course leads to a different MD5 result ;-)

We now use the cocoapods.org case structure for the hashing and (via the .log file I've added to fetchcode/package.py) disclose that in the headers/warnings section of the JSON output (e.g., "Input PURL name 'afnetworking' processed as 'AFNetworking' per https://cocoapods.org/pods/afnetworking").