Lazy loading or loading and downloading FILELISTS

[j-mracek]

The feature would be cool to implement but there are many obstacles on the way. There are additional problems in DNF4 code base.

Trigger for lazy downloading

There might be different triggers, but when the condition is satisfied then it requires to download filelists for all repositories.

Transaction resolve fails

When transaction resolve fails due to missing file provider. Additionally there must be a repository that has filelists in repomd file defined but filelist was not downloaded or loaded. In DNF4 the trigger must be implemented 2 times - for DNF and second one for microdnf and Packagekit. This approach is not capable to detect all cases when filests are required. If I want to upgrade all packages with no-best (Fedora default) the transaction may bee created, but installed file provide will not get upgraded because available upgrade does not provide required file, when filelists are not loaded

When a package in transaction requires file pattern not in /etc or /usr/(s)bin

This approach might lead in situation when filelist gets loaded even when it is not required. If provider have an explicit Provides: <filepath> then filelists are not required.

How to implement

1. Lazy loading of FILELISTS
   It requires that metadata for filelists will be always downloaded, but only lazy loaded. This approach is stable, but it requires additional downloads. After loading of filelist to Sack/Pool it will require to make provides ready otherwise filelists might be not available for solver. The consequence of that operation is that all temporal ID in libsolv get erased. DNF and DNF5 uses in Goal jobs oneof that uses temporal ID to specify what is supposed to do. DNF5 has one advantage here, it does not store oneof jobs in Goal class. In DNF4 it will result that all instances of Goal class might have corrupted information. dnf.base() does contain one instance of Goal but there might be other instances and we don't have a tracker for them. Even worst situation might be with PackageKit. As a solution it would be possible to decode the information in Goal class before makes provides ready is performed and encoded after.

2. Lazy downloading and loading of FILELISTS
   The approach is more efficient but less stable because FILELISTS might be unavailable to download when they are required. It also has all disadvantage from the first approach. FILELISTS might be not available on any mirror or dnf would be not allowed to download them. The situation might be resolve by two ways:

   • Fail with a description
   • Reset Sack and reload and refresh all metadata from beginning
     • This approach is NO-GO, because it also resets excludes, all changes performed by plugins in Sack/pool, makes all instances that requires pool invalid (Package class, Goal Class) and the current protection mechanism in DNF5 does not catch it. In DNF4 there is a missing protection mechanism for mixing data from different pools. The list of problem is much longer.

Link to a discussion: https://lists.fedoraproject.org/archives/list/[email protected]/thread/BRX7BAAFUS4HCRDN4J243FFBSECEFNTV/