Unable to use apt-file with an aptly repository

[mtmr]

apt-file is a tool that gives information about packages that provide a specific file

Detailed Description

aptly generates a Contents-arch.tgz, but it is not being recognized by apt-file.

[smira]

@mtmr could you please provide more details or some steps to reproduce?

[mtmr]

Hey, Sorry for the delay in responding. You just need to create a repo/mirror with aptly, publish it and use that in any instance. In that instance, install apt-file and try to search for any package in apt-file. It will not work, and apt itself does not fetch Contents-arch.tgz unlike the official debian repos. If you need any more information, you can ping me. Thanks.

…

On Thu, Nov 9, 2017 at 1:59 AM, Andrey Smirnov ***@***.***> wrote: @mtmr <https://github.com/mtmr> could you please provide more details or some steps to reproduce? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/smira/aptly/issues/667#issuecomment-342949202>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ5vmQmYpyv0nB3RV_Jq7SOII2gyXIkIks5s0g8igaJpZM4QWPTb> .

[nordri]

Hello there,
We are struggling with something similar.
The Contents-amd64.gz is generated in http://S3_bucket/dists/xenial/component/Contents-amd64.gz but apt-file tries to download it from http://S3_bucket/dists/xenial/Contents-amd64.gz_ so it returns 403 forbidden.
If I upload manually the file to that specific location it works fine.
Any idea how to change this behavior?

aptly version 1.2.0

Thanks in advance.

[smira]

Official documentation:

https://wiki.debian.org/DebianRepository/Format#A.22Contents.22_indices

The files dists/$DIST/$COMP/Contents-$SARCH.gz (and dists/$DIST/$COMP/Contents-udeb-$SARCH.gz for udebs) are so called Contents indices. The variable $SARCH means either a binary architecture or the pseudo-architecture "source" that represents source packages. They are optional indices describing which files can be found in which packages. Prior to Debian wheezy, the files were located below "dists/$DIST/Contents-$SARCH.gz".

Which version of Debian/Ubuntu are you using?

[mtmr]

We are using Debian stretch, but for some reason it doesn't work as expected. Copying the file to "dists/$DIST/Contents-$SARCH.gz" fixes the issue however. This might be because of old versions of apt itself.

…

On Fri, Feb 23, 2018 at 5:02 AM, Andrey Smirnov ***@***.***> wrote: Official documentation: https://wiki.debian.org/DebianRepository/Format#A.22Contents.22_indices The files dists/$DIST/$COMP/Contents-$SARCH.gz (and dists/$DIST/$COMP/Contents-udeb-$SARCH.gz for udebs) are so called Contents indices. The variable $SARCH means either a binary architecture or the pseudo-architecture "source" that represents source packages. They are optional indices describing which files can be found in which packages. *Prior to Debian wheezy*, the files were located below "dists/$DIST/Contents-$SARCH.gz". Which version of Debian/Ubuntu are you using? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/smira/aptly/issues/667#issuecomment-367859087>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ5vmZyrXehuXCYLDeo-S-j_V09OkuRmks5tXfj-gaJpZM4QWPTb> .

[smira]

Even official Debian mirror doesn't have Contents indices in distribution folder, only per-component ones: http://ftp.us.debian.org/debian/dists/stretch/

[smira]

I'm closing this one unless there's some additional info on that, if there's anything else we should do on aptly side to support it better.

[j1elo]

Hi @smira, I'm working with @nordri and we faced this issue. After doing my research, it turns out that Ubuntu does still use on this day the traditional mirror layout that Debian modified some time ago. It is easy to see by checking on their repos:

Ubuntu: http://archive.ubuntu.com/ubuntu/dists/xenial/ -> Contents-* file located in the root dir.
Debian: http://ftp.debian.org/debian/dists/stretch/ -> Contents-* file(s) located in $COMPONENT subdirs (which in case of Debian, are "main", "contrib", "non-free").

On each system, apt-file is configured to work with their corresponding (incompatible) repo layouts. In Ubuntu at least, it is also possible to configure the download URL, in /etc/apt/apt-file.conf.

I have empirically tested this, on Ubuntu 16.04 (Xenial):

• With the default settings in /etc/apt/apt-file.conf, it fails accessing a Debian-style layout:

http = diffindex-download -i "<uri>/dists/<dist>/Contents-<arch>.gz" <cache>/<dest>

# apt-file --verbose --sources-list /etc/apt/sources.list.d/debian.list update
D: Using cache directory /var/cache/apt/apt-file
D: reading sources file /etc/apt/sources.list.d/debian.list
D: got 'deb http://deb.debian.org/debian stretch main contrib non-free'
D: kept 'deb http://deb.debian.org/debian stretch main contrib non-free'
D: diffindex-download -i "http://deb.debian.org/debian/dists/stretch/Contents-amd64.gz" /var/cache/apt/apt-file/deb.debian.org_debian_dists_stretch_Contents-amd64.gz
+ diffindex-download -i http://deb.debian.org/debian/dists/stretch/Contents-amd64.gz /var/cache/apt/apt-file/deb.debian.org_debian_dists_stretch_Contents-amd64.gz
Downloading Index http://deb.debian.org/debian/dists/stretch/Contents-amd64.diff/Index:
No Index available.
Ignoring source without Contents File:
  http://deb.debian.org/debian/dists/stretch/Contents-amd64.gz

• With the modified settings in /etc/apt/apt-file.conf (manually adding the component "contrib", as an example), access is successful:

http = diffindex-download -i "<uri>/dists/<dist>/contrib/Contents-<arch>.gz" <cache>/<dest>

# apt-file --verbose --sources-list /etc/apt/sources.list.d/debian.list update
D: Using cache directory /var/cache/apt/apt-file
D: reading sources file /etc/apt/sources.list.d/debian.list
D: got 'deb http://deb.debian.org/debian stretch main contrib non-free'
D: kept 'deb http://deb.debian.org/debian stretch main contrib non-free'
D: diffindex-download -i "http://deb.debian.org/debian/dists/stretch/contrib/Contents-amd64.gz" /var/cache/apt/apt-file/deb.debian.org_debian_dists_stretch_Contents-amd64.gz
+ diffindex-download -i http://deb.debian.org/debian/dists/stretch/contrib/Contents-amd64.gz /var/cache/apt/apt-file/deb.debian.org_debian_dists_stretch_Contents-amd64.gz
Downloading Index http://deb.debian.org/debian/dists/stretch/contrib/Contents-amd64.diff/Index:
No Index available.
Downloading complete file http://deb.debian.org/debian/dists/stretch/contrib/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   327  100   327    0     0   4691      0 --:--:-- --:--:-- --:--:--  4739
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
File is up-to-date.

Regarding what can be done in the aptly side to better support this situation, I would guess that supporting both traditional (Ubuntu) and modern (Debian) repo layout styles would do it. I don't know if that would introduce too much complexity to aptly, though.

Related:

• Debian bug 755086.

[smira]

@j1elo thanks for researching it, it makes sense to add support for Ubuntu (legacy) layout

[j1elo]

Yes I think it would make sense. There is a bit more to know: apt-file has been reworked lately and in the latest version (to be included in the still unreleased Ubuntu 18.04 LTS) it is now more "intelligent" and able to use the Apt subsystem in order to detect what is the repo layout. Not sure about Debian, but it probably does the same in the most up-to-date versions.

However for aptly I think this would be solving the issue by staying too much on the tip of the latest releases; the current Ubuntu 16.04 is officially supported until year 2021, and even the older 14.04 will still be around for one whole year until April 2019 (see the Ubuntu Releases page).

(I'm talking here only about Long Term Support releases, because in my opinion those are the only ones for which it makes sense the effort of building tools and providing support)

[smira]

I've got the implementation ready, now I need to update tests for the changes, and we should be good to go

[smira]

Nightly build aptly_1.2.0+36+g4d3b42e contains the fix