Support for PMTiles updates

[HarelM]

I have the following scenario currently, which I'm looking to improve, hopefully with PMTiles.
In order to have offline support in my app (which uses maplibre-gl-js and Capacitor) I'm downloading mbtiles to a folder of the device and serve tiles from there (reading it with a sqlite native support).
This works great and is very fast, but doesn't support incremental updates - i.e. if I want to update the mbtiles I'll need to update specific tiles in it, when we looked at this, we saw that the mbtiles file gets bigger and bigger and the vaccum opperation doesn't solve the problem.
So our current solution is to generate and download the entire file every day.
This can work on our situation, which is only the country of Israel, which is small, but this doesn't scale.
Since PMTiles is a zip archive, I'm almost sure this can be done well without issues.
The problem is that I don't think the JS library supports writing single tiles (I only saw getZxy method).
I'm still looking into what other solutions we have in general, but this direction seems promising I hope, assuming I don't need to load all the file into memory to do these operations.
Any feedback is welcome.

[bdon]

Since PMTiles is a zip archive, I'm almost sure this can be done well without issues.

PMTiles is not a zip archive and does not support in-place or transactional updates.

I would recommend generating two PMTiles in clustered mode (the go-pmtiles and tippecanoe tools do this) and then using tools that perform RSync binary diffs to only send over a patch. I haven't investigated this deeply but it should be efficient in theory. Let me know what you learn.

[bdon]

@HarelM xdelta, rsync or unison might be one of the tools for this

[HarelM]

Ahh, I got things wrong then, sorry...
I'm still not in POC mode, still in design phase...

[bdon]

no problem, actually this has overlap with #74 and I'm interested in this application, but I haven't had the time to prototype the diffing.

[HarelM]

Yes, it does overlap. Just a different perspective on the same issue I guess - the need to be able to do incremental updates.

[bdon]

December 2023 update: this on the roadmap for 2024, but no PoC yet.

[neodescis]

Also interested in this! We are currently shipping two planet-wide pmtiles archives, and moving around the entire files with any frequency is a non-starter. I will look into the binary diff tools, but it would be great to have built-in support.

[bdon]

A binary diffing method like rsync should work as-is, but it really depends on the nature of your data and what parts of it are changing. changing a string key in a vector tile could change the majority of the archive.

most of the binary diff programs like rsync require server-side support, so I've been looking into a way to make diffing work with static HTTP files. It would require HTTP multi-range bytes=0-1,3-4 support though, which doesn't work on most S3-like services, but does work with nginx and Caddy.

[neodescis]

In theory, our tiles should not have changed much structurally. One of the archives is created using the protomaps/basemaps Planetiler profile. The other archive is generated from Overture data, also using Planetiler. I had not created new archives in around 9 months though.

I did look into using xdelta3, and also hdiffpatch. Unfortunately they each produced diffs 95% the size of the new tile archives. I'll try again in a month when new Overture data is available, but I'm not very hopeful. I did try xdelta3 with an archive covering a much smaller area with only a week's worth of differences, and that seemed to work better.

[bdon]

for protomaps/basemaps I was seeing roughly a 2-3% diff size for the planet between days.

It does take several minutes to calculate the diff though, possibly longer than it takes to download that 2-3% of data.

[neodescis]

I circled back on this recently and I think I've come to the conclusion that running a diff on a month's worth of changes just isn't really feasible, at least at the archive level. I'm seeing diffs that are almost as large as the archives.

That said, I would be willing to try diffing individual tiles, but I'm not really sure how to get them out of an archive en masse. Does the CLI have anything that would do that? If that path were to prove fruitful, I could come up with a script or something to extract, diff and re-archive.

[bdon]

No, there's no CLI for that because that doesn't correspond to any common operation, you could try using the go or python implementations as libraries.