getTile(x,y,z) method for zoom-able maps

[gszep]

We would like a getTile(x,y,z) type method that will eventually be called by some sort of front end such as OpenLayers that will allow us to zoom in and out of the beautiful SOM Embeddings. Making this work with the Makie plotting ecosystem with both GLMakie and WGLMakie backends would be ideal. Otherwise JSServe.jl can be used to connect to OpenLayers. Things to consider:

• Inplace Rasterisation vs Caching Do we keep all results in ram (presumably always a the size of the viewing window) or do we cache results on disk, creating a z/x/y.png folder structure?
• Triggering re-reasterisation for updating colours, marker sizes etc

[exaexa]

Hello!

I'm just adding the links from the e-mails we exchanged, for future reference:

• current functions: https://makie.juliaplots.org/dev/abstractplotting_api.html#AbstractPlotting.scatter-Tuple and https://github.com/JuliaPlots/WGLMakie.jl
• ideal starting point: GeoMaike, xyz tiles MakieOrg/GeoMakie.jl#67
• example with getTile: https://openlayers.org/en/latest/examples/xyz.html

[gszep]

probably need to store embeddingCoordinates = embedGigaSOM(som,data) as a quadtree so that we can use the tile index embeddingCoordinates[x,y,z] to retrieve the coordinates in that tile.. maybe this package is useful? https://github.com/rdeits/RegionTrees.jl

[exaexa]

Yeah, that may help, not sure about the package performance and tree storage overhead though.

For 2D point indexing, the simplest&fastest solution (by far) is usually to have the dataset pre-sorted by z-order integerized coordinates, and stored in full tree in array (like heaps, see e.g. treaps) for cache efficiency -- that's literally few lines of code, and doesn't have any memory overhead, which is a grave concern if we want to render 1G points.

[gszep]

useful resource for implementing tile and image coordinates
https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames

A python script that tiles images for you
https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/scripts/gdal2tiles.py
command line usage

gdal2tiles.py -p raster path/to/image.png

[exaexa]

Hi Greg! Sorry for the longer delay, stuff has been quite dense.

I've put together a demo (really primitive for now) that does the zoomable embedding trick with the javascript demo you sent:
https://github.com/exaexa/gigascatter-tiles
It runs pretty smoothly for me; there's an example generated "spiral" dataset. let me know if it seems like a step in the "right direction" for you. There are still certainly some TODOs to solve, but nothing really complicated.

[exaexa]

PS.
Regarding the original questions:

• Inplace Rasterisation vs Caching -- certainly in-place rasterization, it should be fast enough and the browser does the caching for itself.
• Triggering re-reasterisation -- if you change the data on the back-end, you'll start getting new colors (or whatever else). One possible problem now is the browser cache (cf. above :] ), but that can be "solved" using some very dirty trick, such as prepending a random nonce to the URL.

[gszep]

the above example is amazing! I think this will be very powerful. OpenLayers has all the functionality we need for gating and annotations too 😉 I cannot overstate how useful this is going to be. Its funny how human cell atlas projects will end up leveraging existing atlas tools!

Caching Is there any way we can see which tiles are cached and for how long they are stored? I can see you have a nice info message in backend telling us which tile is being requested. Is there a similar message we can have in the front-end to tell us whether we are using a cached response or not?

Space Partitioning I'm surprised how fast it is despite the loop through 1M points per tile request 😅 I think storing the points as a quad-tree is the perfect solution. I would again suggest https://github.com/rdeits/RegionTrees.jl for the sake of code re-use, its only one additional dependency 😉