IPFS support

[Martinligabue]

Because it's a static file, IPFS would be a very useful way of hosting the file and sharing the load between users.
IPFS distributes the load among nodes: users with IPFS or Brave installed become nodes and help to distribute data. Users without IPFS won't notice any difference, but for those who have it, content is accessed more efficiently and could load faster because it's accessible from more nodes.
In addition, IPFS improves file resiliency, ensuring that content is available even during disruptions or outages, as one individual node can suffice to maintain access to the data, like in the event of server failure.
I'm not sure if you also need some censorship resistance, but that's also something that it can help with.

[wipfli]

@hfu made something like this I think

[wipfli]

https://twitter.com/_hfu_/status/1716072474159567025

[hfu]

Wow, thank you for mentioning on my small project.

It looks to me IPFS is a great way to share very large dataset at the scale of tens or hundreds of gigabytes. PMTiles has a really good scalability. Therefore, the combination of PMTiles and IPFS is really promising!

Because I would like to pursue possibilities of IPFS on geospatial information, I am working on elevation tiles (elevation encoded in PNG or WEBP) and 3D Tiles (.pnts tiles not is ZXY). I am just wondering if PMTiles technology can be applied on those formats in the future.

I am often using PMTiles + IPFS for sharing vector tile data. I really appreciate the PMTiles community for sharing this highly scalable and thus promising technology!

[bdon]

Nice suggestion! I'm going to convert this issue to a discussion, because it sounds like it should just work with non-HTTP protocols.

[hfu]

Thank you so much, @wipfli, for mentioning. Let me introduce what we are doing!

What are we (@hfu's group) doing?

We are using PMTiles (and 3D Tiles, among other web map data) in combination with IPFS (InterPlanetary File System) hoping that it will help more web maps to available on the web.

Who are we?

We are a group of engineers and educators who want to test new technologies for future geospatial operations.

We name our group UN Smart Maps Group and the group is under UN Open GIS Initiative. We name our IPFS project "UN Smart Maps Bazaar". We have a web site of the group.

Any demos?

How about these?

Map of Japan in PMTiles, fully on IPFS

https://smb.optgeo.org/ipfs/QmTU8chHFkydjc5XZoNVT3TQ1xEHP5zpJ4wzYktrPd1aWD/

Japan's mapping agency publishes PMTiles of its web map. Smart Maps Group (SMG) copied the PMTiles file on IPFS with site files. Because everything is hosted on IPFS, Brave browser can read the site as ipfs://bafybeicmgug7hzkt7ahlebm2hkuiygfz2sz77dwbchh272gwm6okwuwjai/. (my Brave browser switches to ipfs automatically and show this ipfs:// URL.)

Repository: https://github.com/optgeo/c2

Protomaps planet base map on Smart Maps Bazaar

https://smb.optgeo.org/ipfs/QmdPqXGTh1hLdkkPqjLdaJFG1rA8c3UEtUbx1e95pLvtTG/#12.06/-1.45158/-48.46963

Repository: https://github.com/optgeo/pmb

Others

We also have hosted Cesium 3D Tiles for both 3D City Models and 3D Point Cloud, on IPFS. Although 3D Tiles would be off topic for this discussion, let me share https://observablehq.com/@hfu/nagasaki-point-cloud-visualization. This uses 400GB of 3D Tiles hosted on IPFS. There are tons of space for performance optimization, but I see it is feasible to host large geospatial cloud-native data on IPFS.

What can we do for PMTiles?

Maybe we can promote the use of IPFS with PMTiles. We can also share what we know about how to use kubo to host PMTiles files.

Let's discuss how we can combine efforts for more open, free, and distributed web maps!

[hfu]

I have slightly updated README.md of my IPFS-related project at https://github.com/UNopenGIS/smb/blob/main/README.md.

My observation about the current status of IPFS support for geospatial is as below:

Kubo has become an invaluable tool for the UN Smart Maps Group due to its unlimited data storage capacity and the ability to transport data anywhere. By utilizing Kubo, it is possible to send data to the internet from a local environment without the need for setting up a server. However, on the other hand, IPFS cannot fully replace Content Delivery Networks (CDNs).

Given the decentralized nature of IPFS, supporting large-scale access on its own is challenging. This is because data is distributed across the entire network, and in the case of high traffic, there may be challenges in ensuring sufficient bandwidth and node availability.

Moreover, caution is necessary regarding tamper resistance. While IPFS protocol ensures that the Content Identifier (CID) changes when the content changes, in actual implementations, it is theoretically possible to create nodes or gateways that return inappropriate content.

Currently, the UN Smart Maps Group leverages Kubo as an excellent tool for the ingestion and movement of data. This means that we can handle our data flexibly and deploy it rapidly to necessary locations. This flexibility is particularly crucial in emergency situations or disaster response. However, at the same time, we are mindful of the limitations and considerations of IPFS, using it in appropriate contexts.

[hfu]

To demonstrate our method, I converted "Addresses, postal codes and WGS84-coordinates of Finnish buildings" by Digital and Population Data Services Agency (Finnish Digital Agency), accessed 2023-11-12, into PMTiles and hosted it on IPFS.

https://twitter.com/_hfu_/status/1723573832240496917

• Demo site, thanks to PMTiles Viewer
• Repository

After generating PMTiles file, we just needed to push to IPFS by "ipfs add --progress a.pmtiles" and then open it using the returned CID.

[shawaj]

You can also serve IPFS stuff via Cloudflare IPFS gateway as well if you want. There's a nice explainer here...
https://medium.com/pinata/how-to-easily-host-a-website-on-ipfs-9d842b5d6a01

[hfu]

Thank you! I was able to test these gateway servers. This reminded me the importance of proper node / gateway management.