Refactor the caching code and partition cache into sections

The new cache is now backed by xmlb instead of an LMDB database, which
allows us to perform a lot more complex queries with low effort.
The cache is also now shared between all applications by default (by
popular request), for every grouping of metadata.
In addition to that, what AsPool understands as "cache" is now a
collection of partitions, called sections, which represent AppStream
metadata from one domain, e.g. one Flatpak repository, the OS'
collection metadata, the combined metainfo/desktop-entry data of the
system, etc. This permits updating those sections independently, which
means that if a MetaInfo file changes, we will not have to rebuild the
whole cache, but only a small section of it.

This is a prerequisite for efficient monitoring of metadata directories,
and a lot of other neat optimizations. Since AppStream is used in
desktop shells nowadays, support for this is a needed addition to not
keep the system busy with needless work and cause lag.

In addition to that, a lot of cruft and complex code has also been
cleaned up. The current code runs about 60% slower than the previous
cache on cache rebuilds, query time is about 10% slower. There is a lot
of room for improvements though, and we will likely get to the previous
times before release.

Caution! The new code is not yet fully threadsafe and has various rough
edges, but it compiles and passes the testsuite. Further improvements
are located in smaller, easier to manage follow-up patches.

CC: #337

docs/API-TODO.md

@@ -23,3 +23,7 @@ libappstream API break for the AppStream 1.0 release.
    (there are likely some performance improvements to be found there)
 
  * Make UNKNOWN the first entry in AsFormatVersion enum
+
+ * Rename AsPoolFlags: e.g. AS_POOL_FLAG_READ_COLLECTION -> AS_POOL_FLAG_USE_OS_COLLECTION
+
+ * Cleanup AsPool API, only keep sensible functions (maybe make the pool read-only?)

meson.build

@@ -93,14 +93,7 @@ gio_unix_dep = dependency('gio-unix-2.0', version: '>=2.58')
 curl_dep = dependency('libcurl', version : '>= 7.62')
 xml2_dep = dependency('libxml-2.0')
 yaml_dep = dependency('yaml-0.1')
-lmdb_dep = dependency('lmdb', required: false)
-
-if not lmdb_dep.found()
-    lmdb_lib = cc.find_library('lmdb', required: true)
-    if not cc.has_header('lmdb.h')
-        error('Headers for dependency "lmdb" not found')
-    endif
-endif
+xmlb_dep = dependency('xmlb', version : '>= 0.3.2')
 
 if get_option ('gir')
     # ensure we have a version of GIR that isn't broken with Meson

qt/pool.cpp

@@ -18,6 +18,7 @@
  */
 
 #include <appstream.h>
+#include "as-pool-private.h"
 #include "pool.h"
 
 #include <QStringList>
@@ -151,12 +152,16 @@ QList<AppStream::Component> Pool::search(const QString& term) const
 
 void Pool::clearMetadataLocations()
 {
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
     as_pool_clear_metadata_locations(d->pool);
+#pragma GCC diagnostic pop
 }
 
 void Pool::addMetadataLocation(const QString& directory)
 {
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
     as_pool_add_metadata_location (d->pool, qPrintable(directory));
+#pragma GCC diagnostic pop
 }
 
 void Pool::setLocale(const QString& locale)
@@ -174,22 +179,30 @@ void Pool::setFlags(uint flags)
     as_pool_set_flags (d->pool, (AsPoolFlags) flags);
 }
 
+void Pool::overrideCacheLocations(const QString &sysDir, const QString &userDir)
+{
+    as_pool_override_cache_locations (d->pool, qPrintable(sysDir), qPrintable(userDir));
+}
+
 uint Pool::cacheFlags() const
 {
-    return (uint) as_pool_get_cache_flags(d->pool);
+    return 0;
 }
 
 void Pool::setCacheFlags(uint flags)
 {
-    as_pool_set_cache_flags (d->pool, (AsCacheFlags) flags);
 }
 
 QString AppStream::Pool::cacheLocation() const
 {
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
     return QString::fromUtf8(as_pool_get_cache_location(d->pool));
+#pragma GCC diagnostic pop
 }
 
 void Pool::setCacheLocation(const QString &location)
 {
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
     as_pool_set_cache_location(d->pool, qPrintable(location));
+#pragma GCC diagnostic pop
 }

qt/pool.h

@@ -46,7 +46,9 @@ Q_OBJECT
          * FlagNone:             No flags.
          * FlagReadCollection:   Add AppStream collection metadata to the pool.
          * FlagReadMetainfo:     Add data from AppStream metainfo files to the pool.
-         * FlagReadDesktopFiles: Add metadata from .desktop files to the pool.
+         * FlagReadDesktopFiles: Add metadata from desktop-entry files to the pool.
+         * FlagLoadFlatpak:      Add AppStream metadata from Flatpak to the pool.
+         * FlagIgnoreCacheAge:   Ignore cache age and always load data from scratch.
          *
          * Flags on how caching should be used.
          **/
@@ -55,6 +57,8 @@ Q_OBJECT
             FlagReadCollection   = 1 << 0,
             FlagReadMetainfo     = 1 << 1,
             FlagReadDesktopFiles = 1 << 2,
+            FlagLoadFlatpak      = 1 << 3,
+            FlagIgnoreCacheAge   = 1 << 4,
         };
 
         /**
@@ -123,11 +127,14 @@ Q_OBJECT
         uint flags() const;
         void setFlags(uint flags);
 
-        uint cacheFlags() const;
-        void setCacheFlags(uint flags);
+        void overrideCacheLocations(const QString &sysDir,
+                                    const QString &userDir);
 
-        void setCacheLocation(const QString &path);
-        QString cacheLocation() const;
+        Q_DECL_DEPRECATED uint cacheFlags() const;
+        Q_DECL_DEPRECATED void setCacheFlags(uint flags);
+
+        Q_DECL_DEPRECATED void setCacheLocation(const QString &path);
+        Q_DECL_DEPRECATED QString cacheLocation() const;
 
     private:
         Q_DISABLE_COPY(Pool);

qt/tests/asqt-pool-test.cpp

@@ -44,10 +44,11 @@ void PoolReadTest::testRead01()
     auto flags = pool->flags();
     flags &= ~Pool::FlagReadDesktopFiles;
     flags &= ~Pool::FlagReadMetainfo;
+    flags &= ~Pool::FlagIgnoreCacheAge;
     pool->setFlags(flags);
 
-    // don't use caches
-    pool->setCacheFlags(Pool::CacheFlagNone);
+    // use clean caches
+    pool->overrideCacheLocations("/tmp", nullptr);
 
     // read metadata
     QVERIFY(pool->load());