Automatically reload cache when metadata changes

[agx]

I'm using AsPool like

  self->as_pool = as_pool_new ();
  as_pool_set_flags (self->as_pool, AS_POOL_FLAG_READ_METAINFO);

  as_pool_load_async (self->as_pool, self->cancel,
                      on_as_pool_load_ready,
                      data);

so only bother about metainfo of installed applicaticions (/usr/share/metainfo, etc) however i need to refresh the pool when new metainfo gets added there and for that as_pool_add_metadata_locations () would be nice so i can set up appropriate file monitors.

The alternative would be to explicitly add all metainfo locations but that looks odd given AsPools built-in knowledge. But maybe i'm using it incorrectly?

Or would adding file watches via g_file_monitor_directory in AsPool itself be an option so caches stay up to date automatically for these?

[ximion]

There is as_pool_add_metadata_location, but I don't think that method will work for metainfo files (its purpose is for metadata collections, like Flatpak and asgen produce). Doesn't mean that the scope of that function couldn't be expaned though...

Or would adding file watches via g_file_monitor_directory in AsPool itself be an option so caches stay up to date automatically for these?

I would definitely prefer that above all other options. I did even want to write this feature, but so far it didn't have a high enough priority. Annoyingly, due to the way caching works in AsPool currently, this means we would need to reload everything once there is a change detected, but that will probably be okay for most cases (and in future, things can be made smarter when the new caching backend gets implemented).

[agx]

Thanks @ximion . That looks preferable to me as well. I'm not sure i get to this right away but will put it in my list.

[ximion]

This will likely be much easier to add once libappstream was ported to use xmlb as caching backend, and some cache refactoring has been done. AsPool comes from a time where AppStream wasn't so ubiquitous and accepted, so it contains a lot of now unneeded hacks and workarounds and doesn't handle cases with many metadata sources that well, while AsCache & Co. come from a time where Flatpak didn't exist and applications were mainly shipped by distribution packages. So a lot of assumptions there don't hold true anymore, and this needs refactoring (and the resulting code will likely be much simpler).
There are some pretty major dragons hiding in that code at the moment though, so I don't know yet when I can get to it.

[agx]

@ximion gentle ping. The flatpak side for being able to filter apps on form factor in shells is now merged (flatpak/flatpak#4350 (comment)) and if we had this I could finish up the phosh side so we don't need to rely on X-Purism-* in desktop files but use the app metadata.

[ximion]

If this is a blocker, I can actually make it a priority instead of the archive work that I'm currently doing.
The thing is that this is not a small change, but actually a huge piece of work - but has to be done eventually, so I might as well start now...

[ximion]

This was an insane week-long effort... But also necessary maintenance work for the future of the project. After the essential cache refactoring was done, implementing cache auto-reload was rather easy. Currently, queries are a lot slower (at least twice as slow, but depending on the results returned) than before, but speeding that up will require some API additions in libxmlb (planned for later). Aside from that, everything else works better and is more robust.
The cache reloads can on Debian at least also be automatically done in the background via a dpkg trigger on /usr/share/metainfo, for that to work efficiently we will need to monitor the cache file itself as well though, which is not yet implemented.

The tests should cover all the common cases, but please test the code to see if everything (still) works fine! A lot of API has also been deprecated, so clients will need to adjust to that in the long run.

[agx]

I checked that adding/removing items triggeres the signal as expected, thanks!

I'll open seprate bugs should I hit any issues.

[ximion]

@agx I will likely also add a helper function (probably not in the next release but the one after that) that will check the dependency relations (requires/recommends/supports) for the client tool, so that not every client has to reimplement the "has enough memory" check etc. Anything that can't be checked in libappstream would be passed to the client to verify in a callback (because I can't depend on a GUI toolkit to check screen resolutions, for example).
This combined with a filter to only check specific subsets of relations may turn out to be useful for your usecase as well.