Figure out how to access the content of local PyPI mirror

[s117]

No description provided.

[s117]

web/json folder contains 216757 files, and it looks the json metadata for all 216734 mirrored packages are located there, though 23 unknown extra files existed.

➜  json ls                   
0
0-._.-._.-._.-._.-._.-._.-0
00000a
0.0.1
007
00print_lol
00SMALINUX
....
zzz
zzzeeksphinx
zzzfs
zzzutils

➜  json ls |wc -l                        
216757

➜  json cat requests
{
    "info": {
        "author": "Kenneth Reitz",
        "author_email": "[email protected]",
        "bugtrack_url": null,
        ......
        "requires_dist": [
            "chardet (<4,>=3.0.2)",
            "idna (<3,>=2.5)",
            "urllib3 (!=1.25.0,!=1.25.1,<1.26,>=1.21.1)",
            "certifi (>=2017.4.17)",
            "pyOpenSSL (>=0.14) ; extra == 'security'",
            "cryptography (>=1.3.4) ; extra == 'security'",
            "PySocks (!=1.5.7,>=1.5.6) ; extra == 'socks'",
            "win-inet-pton ; (sys_platform == \"win32\" and python_version == \"2.7\") and extra == 'socks'"
        ],
        "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
        "summary": "Python HTTP for Humans.",
        "version": "2.23.0"
    },
    ......
}

[s117]

Btw the size of web/json folder is 3.5 GB.

[werwty]

@s117 Would you mind uploading a zipped copy of the web/json directory to gdrive and sharing with me?

[s117]

@werwty Here you go:

https://drive.google.com/file/d/13udwIGgPUNdfUE-Rb6_NBw25ASj1CVHd/view?usp=sharing

[s117]

Since the JSON API cannot provide dependency information accurately (pypi/warehouse#474), reopen this for exploring how to index arbitrary package inside the mirror.

[s117]

Solved, see https://github.com/werwty/774-project/blob/8099c9be92d93c0bb69611db3d7ce60007907b90/code/app/dependency_graph_builder/lib/RepoProxy.py#L118