Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move 2021 conda packages from robotology to robotology-2021 channel #1585

Closed
traversaro opened this issue Jan 29, 2024 · 3 comments · Fixed by #1587
Closed

Move 2021 conda packages from robotology to robotology-2021 channel #1585

traversaro opened this issue Jan 29, 2024 · 3 comments · Fixed by #1587

Comments

@traversaro
Copy link
Member

In #1572 (comment) to unblock the generation of packages in the robotology channel, we decided to move the 2021 packages from robotology to robotology-2021 . I started last week with some manually download packages, we should complete this and document it.

@traversaro
Copy link
Member Author

As we always increment the CONDA_BUILD_NUMBER at every build, we have a way to quickly spot 2021 builds: all builds till 37 has been done in 2021, while build 38 was the first in 2022.

@traversaro traversaro changed the title Move 2021 packages from robotology to robotology-2021 channel Move 2021 conda packages from robotology to robotology-2021 channel Jan 29, 2024
@traversaro
Copy link
Member Author

I am downloading all files with build_number <= 37 with:

import requests
import os
from tqdm import tqdm
from urllib.parse import urljoin

def download_packages(base_url):
    # Fetch package information from the URL
    repo_url = urljoin(base_url, "repodata.json")
    repo_data = requests.get(repo_url).json()
    packages_info = repo_data["packages"]

    # Download packages with build_number smaller than 37
    for package_name, package_info in packages_info.items():
        build_number = package_info.get("build_number", 0)
        if build_number <= 37:
            download_url = urljoin(base_url, package_name)
            file_size = package_info.get("size", 0)
            sha256_checksum = package_info.get("sha256", "")

            # Create a directory to store downloaded packages
            download_directory = "downloaded_packages"
            os.makedirs(download_directory, exist_ok=True)

            # Download the file
            response = requests.get(download_url, stream=True)
            file_path = os.path.join(download_directory, package_name)

            with open(file_path, "wb") as file, tqdm(
                desc=package_name,
                total=file_size,
                unit="B",
                unit_scale=True,
                unit_divisor=1024,
            ) as bar:
                for data in response.iter_content(chunk_size=1024):
                    bar.update(len(data))
                    file.write(data)

            # Verify SHA256 checksum if available
            if sha256_checksum:
                import hashlib

                sha256_hash = hashlib.sha256()
                with open(file_path, "rb") as file:
                    for chunk in iter(lambda: file.read(4096), b""):
                        sha256_hash.update(chunk)
                if sha256_hash.hexdigest() != sha256_checksum:
                    print(f"SHA256 checksum does not match for {package_name}.")
                else:
                    print(f"{package_name} downloaded successfully.")
            else:
                print(f"{package_name} downloaded successfully.")

# List of repository URLs
repository_urls = [
    "https://conda.anaconda.org/robotology/win-64/",
    "https://conda.anaconda.org/robotology/linux-64/",
    "https://conda.anaconda.org/robotology/osx-64/",
    "https://conda.anaconda.org/robotology/noarch/",
]

# Download packages from each repository
for repo_url in repository_urls:
    download_packages(repo_url)

@traversaro
Copy link
Member Author

Probably we should migrate this packages to robotology on prefix.dev now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant