Skip to content

Commit

Permalink
Support zstd-compressed repositories, including streaming compression
Browse files Browse the repository at this point in the history
Sometimes a zstd-compressed repository will have used streaming
compression. The simpler zstd package does not support it, so use the
full zstandard package instead.
  • Loading branch information
nicois committed Jul 18, 2024
1 parent 201825c commit 7a7c7b3
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 5 deletions.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ dependencies = [
"python-dateutil>=2.8.1,<3",
"botocore>=1.23.50,<2",
"lxml>=4.6.5,<5",
"zstd>=1.5.5.1",
"zstandard>=0.21.0",
]

[project.optional-dependencies]
Expand Down
2 changes: 1 addition & 1 deletion rpm_s3_mirror.spec
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Requires: python3-requests
Requires: python3-dateutil
Requires: python3-botocore
Requires: python3-lxml
Requires: python3-zstd
Requires: python3-zstandard
Requires: systemd
Requires: zchunk

Expand Down
7 changes: 4 additions & 3 deletions rpm_s3_mirror/repository.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright (c) 2020 Aiven, Helsinki, Finland. https://aiven.io/
import dataclasses
import lzma
import zstd
import zstandard
import re
import subprocess
from abc import abstractmethod
Expand Down Expand Up @@ -214,10 +214,11 @@ def _compress(self, root, open_size, open_checksum):


def decompress(filename: Path | str) -> bytes:
decompressor = zstandard.ZstdDecompressor()
try:
with open(filename, "rb") as f:
return zstd.decompress(f.read())
except zstd.Error:
return decompressor.decompress(f.read())
except zstandard.ZstdError:
with gzip.open(filename) as f:
return f.read()

Expand Down

0 comments on commit 7a7c7b3

Please sign in to comment.