Optimizing PMTiles on Cloudflare R2: chunking and batch requesting strategies

[tourtiere]

Hi,

I've been following the recent discussions on PMTiles in Cloudflare that emerged from a post on Hacker News yesterday (https://news.ycombinator.com/item?id=41635592). These discussions have raised a couple of questions about optimization strategies in the Cloudflare context, particularly regarding latency issues and data retrieval costs.

Firstly, considering the latency challenges with large files on Cloudflare R2, would it be advisable to segment planet tile datasets into smaller files, each under 500 MB? If this approach has been previously attempted in similar contexts, what were the results?

Secondly, since tiles are packed using Hilbert curves—which suggest that neighboring tiles are also physically close in the storage blob—and given that Cloudflare offers free bandwidth but charges per request, has there been any exploration into the feasibility of batch requesting multiple tiles in a single query, tailored to the user's map view?

Thank you for your insights

[bdon]

would it be advisable to segment planet tile datasets into smaller files, each under 500 MB? If this approach has been previously attempted in similar contexts, what were the results?

I believe the file size on Cloudflare R2 has a pretty small effect on overall latency. Here is an example of a small file on R2:

curl -H "Range: bytes=0-100000" https://r2-public.protomaps.com/protomaps-sample-datasets/cb_2018_us_zcta510_500k.pmtiles --output tmp -w "%{time_total}"

Here is a large file on R2 (beginning):

curl -H "Range: bytes=0-100000" https://r2-public.protomaps.com/protomaps-sample-datasets/terrarium-z12.pmtiles --output tmp -w "%{time_total}"

Here is a large file on R2 (@ 100gb mark):

curl -H "Range: bytes=100000000000-100000100000" https://r2-public.protomaps.com/protomaps-sample-datasets/terrarium-z12.pmtiles --output tmp -w "%{time_total}"

So the size of the file or where you ask for the range doesn't have much effect on the latency. Cloudflare R2 in general is pretty slow, possibly because it's quite a new platform. But I use if for the distribution of the daily builds at http://maps.protomaps.com/builds/ and Zsolt is using it for distributing OpenFreeMap datasets because there are few alternatives with no egress fees.

If latency is a high priority I would recommend looking at Google Cloud, DigitalOcean and Azure as storage options.

Here is an example of a way to split a large file into multiple chunks by modifying the client.
https://thomasgauvin.com/blog/static-protomaps-on-cloudflare
Others have also done this to work around limits on file size like on GitHub pages.

I don't plan on supporting splitting up a large file into multiple chunks as part of the reference implementations. Storage platforms are almost always atomic, if you upload test.pmtiles and then overwrite it with a different file, client programs either see the entire old file or entire new file, never any partial state. If you split the file in 2, then there will be a point in time where the 2 files are an inconsistent version. I don't know of any static storage platform that has transactions across upload operations, for that you need a real database like SQLite or PostgreSQL.

The consequence of inconsistency across files means that if you have a program caching the PMTiles directories, like the web browser client, or a ZXY tile proxy, if one file changes then the offsets you have cached are garbage and you have no way to detect this situation.

The HTTP specification is designed specifically to handle this case for single files, with the interaction of the If-Match request header and the ETag response header. All major platforms support these and calculate ETag (usually as as md5sum of multipart upload md5sums with a suffix), and all the PMTiles tooling in the GitHub org - pmtiles.js, go-pmtiles, Cloudflare Workers/Lambda - support this too.

So all the tooling around PMTiles relies on the single-file design and the HTTP standard for client programs. You might be able to get away with splitting the file in specific situations but it is going to lead to potential failure the data is ever modified or you could do something like assign a UUID to every version and make the data on storage immutable.

You could design a different multi-file format outside of PMTiles that handles these situations but I believe it would be much more complex to have correct atomic behavior (you are kinda writing a database). Lennart Poettering's casync I think is quite interesting in this space.

(Will continue in next comment...)

[bdon]

Secondly, since tiles are packed using Hilbert curves—which suggest that neighboring tiles are also physically close in the storage blob—and given that Cloudflare offers free bandwidth but charges per request, has there been any exploration into the feasibility of batch requesting multiple tiles in a single query, tailored to the user's map view?

Yes, in the case of a clustered archive neighboring tiles are adjacent to each other, with the exception of de-duplicated tiles, for example if there is 1 million water vector tiles then that water tile might appear at an earlier part of the archive.

You could modify the pmtiles.js or Leaflet, MapLibre etc to batch together requests within a few milliseconds, combine their ranges, and then resolve each tile to that byte range. Markus worked on this in the design of COM-Tiles. That would lead to fewer requests and lower request-based usage fees.

The reason I have not done this is because it is a trade-off between user experience and cost. In general the per-request fees of storage platforms are very low (< 0.4 USD per million requests, etc). The map rendering libraries responsible for the user experience such as Leaflet, MapLibre and OpenLayers settle on a standard tile size of 256x256 or 512x512 screen pixels, where a full-page map could load 10-20 tiles and a small locator map loads 1-4.

If you loaded 4 tiles at a time instead of 1, you might have lower total latency but it will "feel" higher to the end user because it takes longer for individual tiles to appear. This is the same concept as progressive loading with PNG interlacing - it is better to show progress with something small fast than taking longer to show everything all at once.

So I don't plan on changing the pmtiles.js web browser rendering integration to take advantage of requesting multiple tiles, because the default behavior conforms to the established UX conventions of web map renderers.

Open to discussion or corrections on any of the above.

[tourtiere]

Thank you for the insights. I've run your tests and the results have convinced me to maintain the current setup for now.

I'm intrigued by the potential of using a clustered archive for batch requests and plan to experiment with this approach soon. I will share the results if they prove significant. I was thinking about using readable streams and displaying tiles as they are downloaded. Although this might lead to users loading tiles that are out of view, I believe it's a worthwhile compromise to explore. However, I agree that this kind of optimization may not be a good fit for pmtiles.js.
Let's see how it goes! Have a good one.

[hyperknot]

Interesting discussion here! So what I was trying to say to Brandon is that a simple curl is not an adequate test to check real world user experience. When I was evaluating the PMTiles approach, I've set up, configured a full screen map in a full screen browser and started scrolling around. There were some tiles which loaded instantly, some tiles which loaded somewhat slower, and some tiles which felt like taking forever, they were still not loaded after 10 seconds.

What I'm saying is that even if 90% of the tiles load OK, the 5-10% which is slow destroys the experience.

Now, I see from your link that there is a PM (Thomas Gauvin) who works for Cloudflare and is an enthusiast about PMTiles, so we have a chance that Cloudflare does address those problems in the long term. Maybe you can even invite him to this discussion, it's basically our only chance of raising this problem at Cloudflare.

---

So for example, on the https://maps.protomaps.com/builds/ page, on a random map, I see these requests regularly. This is actually not so bad, but definitely not a smooth experience.

[tourtiere]

For additional data, I've just performed 1,000 random range requests at https://r2-public.protomaps.com/protomaps-sample-datasets/terrarium-z12.pmtiles. Here are the latency results for September 26, 2024, in Montreal, Canada:
50th percentile latency: 0.51 seconds
95th percentile latency: 0.75 seconds
99th percentile latency: 1.01 seconds
100th percentile (maximum) latency: 1.98 seconds.