Custom sources

[mourner]

The custom source API (addType, etc.) as implemented in #2667 is an undocumented / experimental feature. There's some private documentation in

mapbox-gl-js/src/ui/map.js

Lines 950 to 960 in 6a64798

	/**
	* Adds a [custom source type](#Custom Sources), making it available for use with
	* {@link Map#addSource}.
	* @private
	* @param {string} name The name of the source type; source definition objects use this name in the `{type: ...}` field.
	* @param {Function} SourceType A {@link Source} constructor.
	* @param {Function} callback Called when the source type is ready or with an error argument if there is an error.
	*/
	addSourceType(name, SourceType, callback) {
	return this.style.addSourceType(name, SourceType, callback);
	}

and https://github.com/mapbox/mapbox-gl-js/blob/6a6479884e08385998354fc845d4193d6bdb4336/src/source/source.js, but no guarantees that it is / will stay accurate!

The true custom source API is something we're hoping to work on pretty soon ( 🤞 ), and you can track that work here: https://github.com/mapbox/mapbox-gl-js/projects/2

[anandthakker]

Just a note for anyone tracking this: a fair amount of the custom source type API is documented under the hood, just marked private for now:

We should validate the custom source architecture and build some demo sources before documenting the internals publicly.

So, with that in mind, some specific steps before going public w/ this API:

• Fixup this topojson source example, including a nice readme. This is an example of a fairly minimal custom source type that leverages much of the existing geojson source.
• Complete the "dynamic vector source" type (proof of concept here, relies on Bucket enhancements to allow property-only update #2812 )
• Actually use ^ in production (Dev Seed has a project that will definitely be doing this)
• Wait X weeks

@lucaswoj @mourner any additional suggestions, or thoughts on the right value of X?

[lucaswoj]

That plan looks good to me @anandthakker 😄

X could be 0 weeks. Building the "dynamic vector source" is what I'm after.

[pwilczynski]

We also are interested in exploring some of the custom source ideas for our map but we're worried about the instability of the existing API. Do you have a sense of how "beta" it is at this point? @anandthakker, how did it work for your project?

[anandthakker]

@pwilczynski it's working well in our project, but I do think that there are very likely changes on the horizon, esp relating to some architectural changes in how the WebWorker code works (tracked here: #3034 ). I am not sure what the likely timeline is for that, but I've been holding off on the 'dynamic vector source' example that's blocking this ticket until after that change.

[lucaswoj]

@pwilczynski There are some big changes on the horizon for this API. I recommend looking at designs that don't require creating custom sources for the time being.

[nickpeihl]

Ah, this is very interesting to me. I've been messing around with creating and storing vector tiles in IndexedDB in this repo. I have hacked together a Leaflet module that loads tiles from IndexedDB, but haven't quite figured out how to get a custom mapbox-gl source. Looking forward to seeing more of this!

[thadk]

Any progress on this Omnivore example?

[amirkhan81]

If you're just looking for CSV go GeoJSON, this is worth looking at: https://bl.ocks.org/danswick/effa94375f9ed24cea9f

…

On Jun 14, 2017, at 5:17 PM, Thad Kerosky ***@***.***> wrote: Any progress on this Omnivore example? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[thadk]

That approach had some inconsistent async problems when I last tested it. For instance, in Chrome, the CSV data does not appear the second time you visit the link.

[thadk]

I made a slightly hacky version of the mapbox-gl-topojson into mapbox-gl-csv which seems to work, see:
https://github.com/thadk/mapbox-gl-csv
and the demo at
http://thadk.net/mapbox-gl-csv/?access_token=
(where access_token is filled with your Mapbox-GL token)

[thadk]

I am not sure why, but public versions of Mapbox-GL-JS, even the version matching package.json that is public, 0.22.0 do not seem to have the required mapboxjs.Source to addType API. Is this what changed above and is there any alternative?

Edit: mapboxgl.Map.prototype.addSourceType seems like it might be helpful -- any idea how to adapt?

[thadk]

Here is another reference in the live codebase to the mapbox-gl-topojson though it does not show to use addSourceType: https://github.com/mapbox/mapbox-gl-js/blame/4fcbf38531be1a7a22ebf9211eb433663dc929a3/src/source/geojson_worker_source.js#L17

[anandthakker]

@thadk the custom source API (addType, etc.) in its present form is an undocumented / experimental feature. (Even that might be an overstatement.) There's some private documentation in

mapbox-gl-js/src/ui/map.js

Lines 950 to 960 in 6a64798

	/**
	* Adds a [custom source type](#Custom Sources), making it available for use with
	* {@link Map#addSource}.
	* @private
	* @param {string} name The name of the source type; source definition objects use this name in the `{type: ...}` field.
	* @param {Function} SourceType A {@link Source} constructor.
	* @param {Function} callback Called when the source type is ready or with an error argument if there is an error.
	*/
	addSourceType(name, SourceType, callback) {
	return this.style.addSourceType(name, SourceType, callback);
	}

and https://github.com/mapbox/mapbox-gl-js/blob/6a6479884e08385998354fc845d4193d6bdb4336/src/source/source.js, but no guarantees that it is / will stay accurate!

The true custom source API is something we're hoping to work on pretty soon ( 🤞 ), and you can track that work here: https://github.com/mapbox/mapbox-gl-js/projects/2

[a1gemmel]

Is this custom source API still under active development?

[HarelM]

I'm looking into using this feature to load vector tiles from websql rigged with mbtiles content. Can someone share a working example of a custom source? I'm having a hard time understanding what I need to implement and how to inherit...

[HarelM]

Ok, Two days later I found some examples and I can wrap my head around how to implement this. Unfortunately, all the examples I found are basically creating a custom build of mapbox-gl-js and adding the relevant custom source.
I'm guessing that this happen since adding a custom source is usually done while inheriting an exiting source and all the sources are not exported and so one can't really benefit from the exiting code in order to write a new custom source.
Is there a chance to export the sources as part of this request so one can inherit from them and add the relevant functionality? I would hate to create a custom mapbox-gl-js version only because some classes are not exported...
I can create a pull request just for the export, but I'm guessing it's an overkill...

[nerik]

Any news on that @mourner @anandthakker ? We are trying to find an efficient way of presenting spatiotemporal data to Mapbox GL and so far have been very limited by:
a) lack of support in MVT tiles for complex array/object data structures;
b) performance constraints of the renderer (only very simple expressions on flat structures are efficient - see #8268)

We would like to avoid forking or doing custom build at all costs.

Thanks :)

[HarelM]

In case anyone is interested, I have created a fork of mapbox-gl-js here.
This fork kinda uses the concept of openlayers to custom source - it exports a global loadTilesFunction property to facilitate for a hook when trying to load tiles, it only applies for the custom:// protocol in the styles json file.
It is a very simple implementation and an elegant concept, IMO. it's still very rough though...
If this is something that can work for mabpox I'll be more than happy to create a pull request for this, let me know...

[michalfapso]

@HarelM, thanks for your patch! It works great! This is exactly what I was looking for. I hope it will be merged to the main mapbox-gl-js repository, especially the "custom://" handler. It is extremely useful.

[HarelM]

Thanks @michalfapso. If anyone likes my solution above please up vote it, maybe if it gets enough up votes someone from mapbox would consider it. It is a very simple solution that doesn't involve a lot of code changes.
@mourner @anandthakker any thoughts?

[nerik]

In case anyone's following up on that:
We are experimenting with a service worker based solution. The idea is to have a SW intercept fetches mad by Mapbox GL, reading a PBF tile optimized for space from the server, then returning to the client a MVT tile optimized for rendering/animation.
We're not 100% sure yet but it looks promising and allows for some cool stuff like generating geometries on the fly with the same data already queried from the server.

[HarelM]

@nerik Can you share the relevant code? Service worker seems like the right solution to intercept mapbox tile requests - I have a problem with it since I write my app using Angular and I use dependency injection and all the goodies Angular provides that are not available when using a service worker, which makes the development with it much more complex...

@mourner @anandthakker Any update regarding my question above? I would very much like to add my code to mapbox and reduce the need to manually updating my fork and creating a patch every time I want to update mapbox version. Also I think others can benefit from it (8 developers on this thread have found this useful :-)).

[nerik]

@HarelM We have a PoC on a private repo, just invited you with read access. It's a react/CRA setup, but with not a lot added actually so you should get your bearings easily (Rollup compiles the sw at src/sw/index.js, client code is at src/App.js)

[lasterra]

@HarelM Thanks Harel for your idea. It's simple and clever, and solve our problem. I was trying to implement an extension Class of VectorTileSource and asociate it with a new sourcetype.

But VectorTileSource is not public available so i also need to fork mapbox code.

I hope this kind of extension will be available on mapbox in the future.

[HarelM]

@lasterra no problem :-)
In my initial solution I tried to do what you said and encountered this exact issue. Later when I figured out the "custom" solution I found out it also works for terrainRGB and raster so as a general solution it's better :-)
I hope this will get merged someday so I won't need to rebase my fork every once in a while...

[rustygreen]

Any update on this?

[neave]

+1 on a solution for this. The ability to intercept loading tiles and access their imagery/data is essential for my mapping app.

I'm currently using OpenLayers and their tileLoadFunction to load tiles and apply effects to them before they are rendered, or in some cases load the tile and access the header information per tile. But there doesn't seem to be a way to do this in Mapbox GL JS without resorting to hacks.

[nerik]

For anyone interested, we ended up opting for the solution proposed by @HarelM (#2920 (comment)).
Turns out forking MGL is the most elegant solution with the least custom code. We added a custom source which runs some logic on its own worker, all within MGL infrastructure, it's neat. We have a bot that regularly opens PR to merge the upstream so that we can stay up-to-date with the mothership relatively painlessly.

Cheers.

[HarelM]

@nerik I'm not sure I fully understand your comment.
I looked in the fork you have and I don't see the solution I did in my fork, only a solution for something called temporal grid, which I'm not sure what it means.
Can you elaborate a bit on what you did? or you meant to say that you used the idea from my solution to develop something new called temproal grid and not a general solution to a custom source?
If your fork has the capabilities that my fork has and is kept up-to-date using a bot I would very much like to use it instead of using mine, as I need to manually update my every time I want to get the latest changes from master :-)

[nerik]

@HarelM Sorry, what I meant was that we opted to fork Mapbox GL JS, as you did, instead of trying to hijack tile calls with a separated service worker.
For the rest you are correct, our solution is different, in that instead of supplying a generic hook to apply a transformation to any tile, we created a distinct temporalgrid source type for our use case (gridded/heatmap data that can be filtered by time).
We keep a PR opened to compare to upstream but in a nutshell:

• we registered a new temporalgrid source type that can be used as a style source.type
• we extended both VectorTileSource and VectorTileWorkerSource
• the worker source picks up a URL with parameters via loadVectorData. These URL parameters are actually used to configure what happens in the worker. It's a bit of a hack, but it avoids hooking too deep into Mapbox GL code by having a fully configurable source
• the worker forms the final tile URL, loads the tile, decodes it (we use a custom PBF tile format that is more efficient for spatiotemporal data), perform a bunch of aggregation steps, and finally build a standard MVT tile on the fly, so that Mapbox GL can render it (the worker will generate square or point geometries depending on what's needed, which can be then rendered with your standard circle/heatmap/fill/etc layers).

Hope this helps

[j8seangel]

If your fork has the capabilities that my fork has

Hi @HarelM we didn't develop a way to add custom sources but created a specific type instead to fit our needs, sadly I guess this won't work for you.

A generic solution to add custom sources like you proposed didn't work for use because we wanted to reuse as much logic as possible from the VectorTileSource renderer and we didn't find a way to extend it from outside but any ideas are welcome 😄

as I need to manually update my every time I want to get the latest changes from master

To explain a bit more about the auto updates, we mainly have two branches in the forked repo:

1. The main branch, without any change and kept updated automatically with GitHub bot tool called pull see config
2. Temporalgrid branch, which contains our custom code

This simplifies us a lot the release process:

1. Merge master to temporalgrid
2. Fix conflicts when needed
3. Publish to npm from temporalgrid branch

I'd recommend you to follow this workflow to keep your fork updated easily

[bman654]

FYI I was able to create a custom Source that re-uses existing Source logic. The trick is to use undocumented Style.getSourceType() to get access to the private/existing sources.

I've only tested this with 3.0 beta, but probably it works on earlier versions?

Here is an example that debounces the tile requests so you don't spam the tile server when the user uses scroll wheel flick to change zoom levels.

import { Style, RasterSource } from "mapbox-gl";
import { Source } from "react-map-gl";

// available type definitions do not include a definition for the Style export
declare module "mapbox-gl" {
  export const Style: any
}

const RasterTileSourceImpl = Style.getSourceType("raster")
const SOURCE_TYPE = "debounced_raster"
const DEBOUNCE_TIME = 100

class DebouncedRasterSourceImpl extends RasterTileSourceImpl {
  constructor(...args: unknown[]) {
    super(...args)
    this.type = SOURCE_TYPE
  }

  loadTile(tile: any, callback: (e: null) => void) {
    const timer = setTimeout(() => {
      if (!tile.aborted) {
        super.loadTile(tile, callback)
      }
    }, DEBOUNCE_TIME)

    tile.request = {
      cancel: () => clearTimeout(timer)
    }
  }
}

Style.setSourceType(SOURCE_TYPE, DebouncedRasterSourceImpl)

export type DebouncedRasterSourceProps = Omit<RasterSource, "type"> & {
  id?: string
  children?: any
}

export const DebouncedRasterTileSource = (props: DebouncedRasterSourceProps) => {
  return (
    <Source
      {...props}
      // @ts-ignore
      type={SOURCE_TYPE}
      />
  )
}

// example usage:
  return (
    <DebouncedRasterTileSource tiles={tiles} tileSize={tileSize}>
      <Layer
        type={"raster"}
        paint={paint}
        minzoom={minzoom}
        maxzoom={maxzoom}
      />
    </DebouncedRasterTileSource>
  );