Supporting non standard package sources

[ericlchen1]

Hey, I ran into some issues when trying to use the nvidia repo as a source url https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64. This is due to the way urls are parsed in rules_distroless and are used to find the Packages file. Also there doesn't seem to be much support when the packages are not multiarch. In this case, the nvidia packages only support amd64 or i386.

Currently I've put together a pretty nonstandard solution with a patch to work around this which supports my use case. However, I was wondering if this would be supported in the future. Thanks!

rules_distroless.patch

diff --git a/apt/private/package_index.bzl b/apt/private/package_index.bzl
index 620a021..c441d4d 100644
--- a/apt/private/package_index.bzl
+++ b/apt/private/package_index.bzl
@@ -2,19 +2,24 @@
 
 load(":util.bzl", "util")
 
-def _fetch_package_index(rctx, url, dist, comp, arch, integrity):
+def _fetch_package_index(rctx, url, dist, comp, arch, integrity, override_url = None):
     target_triple = "{dist}/{comp}/{arch}".format(dist = dist, comp = comp, arch = arch)
     output = "{}/Packages.xz".format(target_triple)
+    url = "{}/dists/{}/{}/binary-{}/Packages.xz".format(url, dist, comp, arch)
+    if override_url:
+        url = override_url
+        output = "{}/Packages".format(target_triple)
     r = rctx.download(
-        url = "{}/dists/{}/{}/binary-{}/Packages.xz".format(url, dist, comp, arch),
+        url = url,
         output = output,
         integrity = integrity,
     )
-    rctx.execute([
-        "xz",
-        "--decompress",
-        output,
-    ])
+    if not override_url:
+        rctx.execute([
+            "xz",
+            "--decompress",
+            output,
+        ])
     return ("{}/Packages".format(target_triple), r.integrity)
 
 def _parse_package_index(state, contents, arch, root):
@@ -59,9 +64,11 @@ def _create(rctx, sources, archs):
     )
 
     for arch in archs:
-        for (url, dist, comp) in sources:
+        for (url, dist, comp, override_url, override_arch) in sources:
+            if override_arch and override_arch != arch:
+                continue
             rctx.report_progress("Fetching package index: {}/{}".format(dist, arch))
-            (output, _) = _fetch_package_index(rctx, url, dist, comp, arch, "")
+            (output, _) = _fetch_package_index(rctx, url, dist, comp, arch, "", override_url)
 
             # TODO: this is expensive to perform.
             rctx.report_progress("Parsing package index: {}/{}".format(dist, arch))
diff --git a/apt/private/resolve.bzl b/apt/private/resolve.bzl
index ea443d2..b14b4ff 100644
--- a/apt/private/resolve.bzl
+++ b/apt/private/resolve.bzl
@@ -66,9 +66,11 @@ def _deb_resolve_impl(rctx):
         if not dist:
             dist = distr
         sources.append((
-            src["url"],
+            src.get("url"),
             distr,
             comp,
+            src.get("override_url"),
+            src.get("override_arch"),
         ))
 
     pkgindex = package_index.new(rctx, sources = sources, archs = manifest["archs"])
@@ -76,7 +78,11 @@ def _deb_resolve_impl(rctx):
     lockf = lockfile.empty(rctx)
 
     for arch in manifest["archs"]:
-        for dep_constraint in manifest["packages"]:
+        packages = manifest["packages"]
+        arch_packages = manifest.get("arch_packages", {})
+        if arch in arch_packages:
+            packages = packages + arch_packages[arch]
+        for dep_constraint in packages:
             constraint = package_resolution.parse_depends(dep_constraint).pop()
 
             rctx.report_progress("Resolving %s" % dep_constraint)

diff --git a/apt/private/package_index.bzl b/apt/private/package_index.bzl
index c441d4d..c344442 100644
--- a/apt/private/package_index.bzl
+++ b/apt/private/package_index.bzl
@@ -42,6 +42,11 @@ def _parse_package_index(state, contents, arch, root):
 
         if len(pkg.keys()) != 0:
             pkg["Root"] = root
+            if "Filename" in pkg:
+                pkg["Filename"] = pkg["Filename"].strip("./")
+            if "Architecture" in pkg and pkg["Architecture"] != "all" and pkg["Architecture"] != arch:
+                pkg = {}
+                continue
             util.set_dict(state.packages, value = pkg, keys = (arch, pkg["Package"], pkg["Version"]))
             last_key = ""
             pkg = {}
@@ -72,7 +74,7 @@ def _create(rctx, sources, archs):
 
             # TODO: this is expensive to perform.
             rctx.report_progress("Parsing package index: {}/{}".format(dist, arch))
-            _parse_package_index(state, rctx.read(output), arch, url)
+            _parse_package_index(state, rctx.read(output), arch, override_url.split("/Packages")[0] if override_url else url)
 
     return struct(
         package_versions = lambda **kwargs: _package_versions(state, **kwargs),

debian12.yaml

version: 1
sources:
- channel: bookworm main
  url: https://snapshot-cloudflare.debian.org/archive/debian/20240612T080215Z
- channel: bookworm-security main
  url: https://snapshot-cloudflare.debian.org/archive/debian-security/20240612T080215Z
- channel: bookworm-updates main
  url: https://snapshot-cloudflare.debian.org/archive/debian/20240612T080215Z
- channel: nvidia main
  override_url: https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/Packages
  override_arch: amd64
archs:
- amd64
- arm64
...
bunch of packages here
...
arch_packages:
  amd64:
  - libcuda1

[ericlchen1]

Seems to be related to #55 but the question of how architecture is handled is still there.

[thesayyn]

the question of how architecture is handled is still there.

My instinct is to do what apt does when you mix non-flat and flat repos. However, i don't have enough time to dig in myself.

[ericlchen1]

I'm a bit hesitant to remove arch_packages in the PR as you'd need to restrict your archs to a single arch if there's only one architecture available. It would throw a Error in fail: Unable to locate package error when running the lock instead of allowing it and having an error when trying to use the package in an invalid architecture. I'm not sure what the best way to handle this case is. The arch portion in sources is completely optional and probably could be refactored.

[jjmaestro]

the question of how architecture is handled is still there.

My instinct is to do what apt does when you mix non-flat and flat repos. However, i don't have enough time to dig in myself.

I think it's totally OK to mix them, IMHO the issue is more with the architectures supported by the repos, regardless of flat VS canonical. For example, you could have a canonical repo with amd64 and arm64 that has (pkg1, pkg2) in amd64 but e.g. only pkg1 in arm64 and you would have the same problem as mixing a canonical repo and a flat repo that have packages missing for certain architectures.

[jjmaestro]

I'm a bit hesitant to remove arch_packages in the PR as you'd need to restrict your archs to a single arch if there's only one architecture available. It would throw a Error in fail: Unable to locate package error when running the lock instead of allowing it and having an error when trying to use the package in an invalid architecture. I'm not sure what the best way to handle this case is. The arch portion in sources is completely optional and probably could be refactored.

As I mentioned in #64 (comment) I think this issue and @thesayyn concerns about architectures are "better solved" (at least for now) by separating the problematic packages and repos into their own manifest.

Also, this is (at least to me) the best way to do it because a manifest is "just the way to get the packages you want from the sources that have them" so if there are problematic packages-arch pairs, they should have their own manifest.

What do you think? Does #86 look like it solves all these concerns? :-?

[ericlchen1]

As I mentioned in #64 (comment) I think this issue and @thesayyn concerns about architectures are "better solved" (at least for now) by separating the problematic packages and repos into their own manifest.

Also, this is (at least to me) the best way to do it because a manifest is "just the way to get the packages you want from the sources that have them" so if there are problematic packages-arch pairs, they should have their own manifest.

What do you think? Does #86 look like it solves all these concerns? :-?

Looks good and I agree with the architecture design with separate manifests :). I haven't had the time recently to work on this. It's great that someone can follow up and hopefully push this through. Happy to close this when #86 goes through.