Add GraalPy

[mayeut]

In order to install GraalPy in the image, one must call manylinux-interpreters ensure graalpy310-graalpy230_310_native.

closes #1509

[mayeut]

@timfel,
I tried to backport most of the comments from #1518 in this PR.
We'll able to move forward once the "skip the simple project build on graalpy" is dropped.

[mayeut]

timfel wrote:

FYI, I added something to get the versions of graalpy to my earlier cibuildwheel PR: timfel/cibuildwheel@ace30fb#diff-2fc4993e0fc3281ad7dfa0e696c6791c9339e16180d62b6fdd05df72a217ced5R112. Not sure if we want to do something like this here to more easily update graalpy, too. I can also do that in a separate PR if/when this one goes in.

[mayeut]

That would be a good thing to add overall but this can until this one's merged.

[timfel]

@mayeut we have released graalpy 23.1, this should now produce the correct wheel tags and pass the tests. Can we update this PR?

[mayeut]

@timfel, I updated the PR but the tests are not passing. pip 23.3 should fix the issue.

[timfel]

@timfel, I updated the PR but the tests are not passing. pip 23.3 should fix the issue.

Ah yes :( We backported the fix into pip 22.2 whl that we ship, but of course on manylinux we use the latest pip

[timfel]

@timfel, I updated the PR but the tests are not passing. pip 23.3 should fix the issue.

23.3 is out, should we rerun the ci?

[mayeut]

The update of packaging has been moved to 24.1 by the pip team.

[timfel]

Thank you for the update Matthieu 🙏 Let me know if there's anything I need to help with from the GraalPy side 🙂

[mayeut]

@timfel,

It seems the latest beta of pip now works properly with GraalPy so this will most likely be ready in a couple weeks.

Let me know if there's anything I need to help with from the GraalPy side

If you can have a look at how to get automated updates in tools/update_interpreters_download.py, it would be nice to have it. Thanks.

[henryiii]

pip 24.1 is out!

[msimacek]

Hi @mayeut,
Here's the changes for tools/update_interpreters_download.py:

index 094653e..ce28e25 100644
--- a/tools/update_interpreters_download.py
+++ b/tools/update_interpreters_download.py
@@ -2,6 +2,7 @@ from __future__ import annotations
 
 import argparse
 import json
+import re
 import subprocess
 from hashlib import sha256
 from pathlib import Path
@@ -82,6 +83,69 @@ def update_pypy_versions(versions, updates):
             )
 
 
+def update_graalpy_version(releases, graalpy_spec, tag, arch, version_dict, updates):
+    graalpy_arch = {"x86_64": "amd64"}.get(arch, arch)
+    current_version = None
+    if "version" in version_dict:
+        current_version = Version(version_dict["version"])
+    for r in releases:
+        version = Version(r['tag_name'].split('-')[1])
+        if current_version is not None and current_version >= version:
+            continue
+        if not graalpy_spec.contains(version):
+            continue
+        message = f"updating {tag} {arch} to {version}"
+        print(message)
+        for asset in r['assets']:
+            if asset['name'] == f'graalpy-{version}-linux-{graalpy_arch}.tar.gz':
+                break
+        response = requests.get(asset["browser_download_url"], stream=True)
+        response.raise_for_status()
+        sha256sum = sha256()
+        for chunk in response.iter_content(chunk_size=1024 * 4):
+            sha256sum.update(chunk)
+        version_dict["version"] = str(version)
+        version_dict["download_url"] = asset["browser_download_url"]
+        version_dict["sha256"] = sha256sum.hexdigest()
+        updates.append(message)
+        break
+
+
+def get_next_page_link(response):
+    link = response.headers.get('link')
+    if link:
+        for part in re.split(r'\s*,\s*', link):
+            split = re.split(r'\s*;\s*', part)
+            url = split[0][1:-1]
+            for param in split[1:]:
+                if re.match(r'rel="?next"?', param):
+                    return url
+
+
+def update_graalpy_versions(versions, updates):
+    releases = []
+    url = "https://api.github.com/repos/oracle/graalpython/releases"
+    while url:
+        response = requests.get(url)
+        response.raise_for_status()
+        releases += response.json()
+        url = get_next_page_link(response)
+    for tag in versions:
+        if not tag.startswith("graalpy"):
+            continue
+        _, abi_tag = tag.split("-")
+        graalpy_ver, _, _ = abi_tag.split("_")
+        assert graalpy_ver.startswith("graalpy")
+        graalpy_ver = graalpy_ver[len("graalpy"):]
+        graalpy_major = int(graalpy_ver[:2])
+        graalpy_minor = int(graalpy_ver[2:])
+        graalpy_spec = Specifier(f"=={graalpy_major}.{graalpy_minor}.*")
+        for arch in versions[tag]:
+            update_graalpy_version(
+                releases, graalpy_spec, tag, arch, versions[tag][arch], updates
+            )
+
+
 def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("--dry-run", dest="dry_run", action="store_true", help="dry run")
@@ -89,6 +153,7 @@ def main():
     versions = json.loads(PYTHON_VERSIONS.read_text())
     updates = []
     update_pypy_versions(versions, updates)
+    update_graalpy_versions(versions, updates)
     if not args.dry_run:
         PYTHON_VERSIONS.write_text(json.dumps(versions, indent=2))
         if updates:

[mayeut]

This is now good to merge !
Thanks everyone.