Adds a script for updating vendored code in sunpy/extern

[akash5100]

PR Description

This PR is an attempt to close #6094

This PR will add a script in the sunpy/tools that will be used to update vendor code automatically, by the person doing a release. Executing this script will update all python files in sunpy/sunpy/extern.

[ConorMacBride]

Hi @akash5100, thank you for working on this PR! Looking through your script, it seems that you are attempting to update the dependencies of the packages in sunpy/sunpy/extern rather than the packages (sunpy/sunpy/extern/*.py) themselves.

As an example, updating sunpy/sunpy/extern/appdirs.py should download the latest released version of the upstream ActiveState/appdirs appdirs.py. (Either using the GitHub API or pip download appdirs for example.) Then the script should replace the old appdirs.py file with the upstream file (and possibly run SunPy's precommit hooks to fix any PEP8 issues etc. not enforced upstream).

[akash5100]

Oh sorry! I misunderstood, thank you for the example @ConorMacBride.

[akash5100]

I think this script will fail in 2 cases (maybe more):

1. If you've ever had to use the GitHub REST APIs, you may be familiar with GitHub's API Rate Limits:
   This means that if you call a GitHub API more than 5,000 times within a 60-minute window, even if you are authenticated, the GitHub APIs will return 403 FORBIDDEN.

2. If the packages are released without a version tag.

edit: I don't know much about release of a package, but If pip download <package> always gives us the latest version, it would be the solution.

[nabobalis]

I think this script will fail in 2 cases (maybe more):

1. If you've ever had to use the GitHub REST APIs, you may be familiar with GitHub's API Rate Limits:
   This means that if you call a GitHub API more than 5,000 times within a 60-minute window, even if you are authenticated, the GitHub APIs will return 403 FORBIDDEN.

I don't think we have to worry about that.

2. If the packages are released without a version tag.

This might be an issue.

edit: I don't know much about release of a package, but If pip download <package> always gives us the latest version, it would be the solution.

What does pip download do?

[akash5100]

What does pip download do?

This command will download the .whl file of the package in the given directory.

For example, pip download distro will download distro-1.7.0-py3-none-any.whl (python wheel file) in the same directory, we then rename the extension from .whl to .zip, after that same procedure. (extracting, moving)

[akash5100]

I think we also should copy over the latest license file for each package, which should be stored in the dist.

If we have a list of files that we need to extract, we can do this in the same function, but I need to figure out how we going to rename the files.

If we have a list of files that we need to extract

Something like this:

PACKAGES = {
    "appdirs": ["ActiveState", "appdirs", ["appdirs.py", "LICENSE.txt" ],
}

Otherwise, without any change in the current function, we can make a new dictionary of LICENSE and call it once again? (But this will repeat downloading, extracting, deleting)

[nabobalis]

We just rename it to <app_name>_licence?

[akash5100]

We just rename it to <app_name>_licence?

Adding this does the work without changing the function, does this looks good?

LICENSES = {
    "appdirs_license.txt": ["ActiveState", "appdirs", "LICENSE.txt"],
    "distro_license.rst": ["python-distro", "distro", "LICENSE"],
    "inflect_license.txt": ["jaraco", "inflect", "LICENSE"],
    "parse_license.txt": ["r1chardj0n3s", "parse", "LICENSE"],
}

for package, (user, repo, src) in LICENSES.items():
    download_github_file(user, repo, src, SUNPY_EXTERN_DIR / package)

[ConorMacBride]

We just rename it to <app_name>_licence?

Might need to update the dictionary keys to be appdirs.py etc, and then add the licences as new "packages" appdirs_licence.txt etc to the dictionary.

[ConorMacBride]

Just seeing your new comment now. I think the separate LICENSES dictionary looks much neater! And you can leave PACKAGES as-is.

[nabobalis]

But wont it download the archive twice?

[akash5100]

Actually, we can add .py in the keys instead of formatting the string while calling the function, still works the same.

PACKAGES = {
    "appdirs.py": ["ActiveState", "appdirs", "appdirs.py"],
    "distro.py": ["python-distro", "distro", "src/distro/distro.py"],
    "inflect.py": ["jaraco", "inflect", "inflect/__init__.py"],
    "parse.py": ["r1chardj0n3s", "parse", "parse.py"],
}

and then

    for package, (user, repo, src) in PACKAGES.items():
        download_github_file(user, repo, src, SUNPY_EXTERN_DIR / package)

If we have the LICENSES dict, we can just replace PACKAGES.items() to LICENSES.item()

[ConorMacBride]

Oh! We don't have to download the whole repo! We should be able to just download the single file: https://raw.githubusercontent.com/{user}/{repo}/{version}/{src} (e.g. https://raw.githubusercontent.com/ActiveState/appdirs/1.4.4/appdirs.py)

Then we can keep download_github_file as a function which only downloads one file.

[akash5100]

Nice, no more zips

edit: Sorry, I missed this, all these days downloading and extracting zips. But, I wonder, if this endpoint is not in the documentation how do you find it?

[ConorMacBride]

In GitHub if you navigate to a file with a particular tag selected e.g. https://github.com/sunpy/sunpy/blob/v4.0.0/setup.py you can see this URL by clicking the "Raw" button. I think we should be able to rely on this? I have seen this type of URL used directly before. What do you think @nabobalis?

[ConorMacBride]

Actually, that URL is documented so this should be fine. https://docs.github.com/en/repositories/creating-and-managing-repositories/about-repositories#text-limits

[nabobalis]

In GitHub if you navigate to a file with a particular tag selected e.g. https://github.com/sunpy/sunpy/blob/v4.0.0/setup.py you can see this URL by clicking the "Raw" button. I think we should be able to rely on this? I have seen this type of URL used directly before. What do you think @nabobalis?

Sounds good

Lots of new changes!

[akash5100]

Then we can keep download_github_file as a function which only downloads one file.

We get bytes as a response, Should I write the byte in the required file?
for example: write the response directly in sunpy/extern/appdirs.py

[nabobalis]

Then we can keep download_github_file as a function which only downloads one file.

We get bytes as a response, Should I write the byte in the required file? for example: write the response directly in sunpy/extern/appdirs.py

Sure.

[Cadair]

Seems sane to me. Thanks!

[nabobalis]

SQUASH MERGE

[nabobalis]

Thanks for the PR @akash5100