Proposed changes to URL and Config structure

[nyurik]

I would like to propose a number changes to Martin:

• URL structure in order to make it more generic, as well as expose less internal data to the world
• Improve config file structure to allow multiple postgreSQL connections and other source types

URL changes

• No /rpc/<id>/... paths. For backward support, we could do a redirect to /<id>/...
• Reconsider schema prefixes, e.g. "public." in public.mytable.
• Remove all suffixes, e.g. .pbf, .json, etc.

path	info
/<id>/z/x/y	Get a tile from the given source. The tile format is source-dependent.
/<id>	Get TileJSON description of the source
/	Get the index manifest -- a list of the available sources. TODO: This endpoint may need to be reworked - we do not want to expose too much information about the internal configuration here.
/health	Returns a 200 OK simple text response

URL source identifiers

• All source IDs must match a regex [_a-z0-9]+ for simplicity. No commas or slashes.
  • We may allow more symbols like -, _, and . -- having dots allow public.mytable style naming, which somewhat exposes internal postgreSQL detail, but this might be ok.
• An identifier must not be one of the reserved words like index or rpc
• It is guaranteed that someone will have multiple sources with the same ID, e.g. a table named the same as an .mbtiles file, etc. We need to decide the resolution process. Some ideas:
  • Introduce a --on-id-conflict (stop | rename | ignore) CLI/config setting that will control what Martin does in case of an ID
  • In case of the rename mode, add a counter at the end of an id, e.g. tbl, tbl1, tbl2, ...
  • In case of the ignore mode, some arbitrary source ID wins (possibly the latest one in the processing queue)

Source auto-discovery

We want to have a simple no-configuration mode, so that developers can start using Martin with the least amount of pain:

• If DATABASE_URL env var is not set, and Martin started with no parameters, look in the current directory for the *.mbtiles files, and make them all available as sources
• PostgreSQL table autodiscovery can be controlled with a discover_tables_as: (table | schema_table | schema_table_column) config parameter. Depending on the setting, tables with geo columns will become table, schema.table or schema.table.geo_column sources.
• PostgreSQL functions autodiscovery can be controlled with a discover_funcs_as: (func | schema_func) config parameter. Depending on the setting, functions will become func or schema.func sources.

Per-source Configuration

• Allow multiple data sources
• Do not allow duplicate IDs (or show a warning?)
• Document defaults

Proposed Config Structure

---
# Trust invalid certificates. This introduces significant vulnerabilities, and should only be used as a last resort.
danger_accept_invalid_certs: false
# Connection keep alive timeout [default: 75]
keep_alive: 75
# The socket address to bind [default: 127.0.0.1:3000]
listen_addresses: '0.0.0.0:3000'
# Number of web server workers
worker_processes: 8
# Allow users to discover available sources with a `GET /` [default: true] 
enable_sources_manifest: false
# Alias legacy /rpc/... requests to the root. This works for all sources, not just function calls. [default: false]
legacy_rpc_paths: true
# Allow the same source to be exposed as multiple IDs, or combine multiple sources into one
aliases:
  public.tbl1: tbl1
  composite1: tbl1,func1,tbl2
# In case multiple sources have the same ID during auto-discovery, this controls how they will be handled.
# Allowed values: "stop" - Martin exits, "ignore" - only one of the sources becomes available,
# and "rename" - generate unique source name automatically.
on_id_conflict: rename

# All .mbtiles file sources
# Each sub-key is a separate source ID
mbtiles:
  # If set, will automatically make all *.mbtiles files in this directory available as sources
  # The path is relative to the config file
  discover_files_in: ./
  # Map of source IDs -> mbtile file sources
  files:
    # Simple configuration with the value being the file path itself relative to the config file
    src1: path/to/file1.mbtiles
    # Customized configuration (actual fields are TBD)
    src2:
      file: path/to/file2.mbtiles
      format: jpeg

# One or several PostgreSQL sources
postgres:
  # Database connection string
  connection_string: 'postgres://postgres@localhost:5432/db'
  # If a spatial table has SRID 0, then this SRID will be used as a fallback
  default_srid: 4326
  # Maximum connections pool size [default: 20]
  pool_size: 20
  # Optionally limit the number of features a table source would return
  max_feature_count: 5000
  # If set to one of the valid values, automatically finds all tables and expose them as sources.
  # Valid values:  "table" - source ID is the table name,  "schema_table" - ID is "schema.table",
  # "schema_table_column" - source ID is "schema.table.geometry_column". 
  discover_tables_as: schema_table_column
  # If set to one of the valid values, automatically finds all MVT-generating functions and expose them as sources.
  # Valid values:  "func" - source ID is the name of the function,  "schema_func" - ID is "schema.func"
  discover_funcs_as: schema_table_column

  # Map of source IDs -> table sources
  tables:
    tbl1:
      # Table schema and name (required)
      schema: public
      table: my_table
      # Geometry SRID (optional)
      srid: 4326
      # Geometry column name (required)
      geom_column: geom
      # Feature id column name. If null (or missing), will not be set.
      id_column: null
      # An integer specifying the minimum zoom level
      minzoom: 0
      # An integer specifying the maximum zoom level. MUST be >= minzoom
      maxzoom: 30
      # The maximum extent of available map tiles. Bounds MUST define an area
      # covered by all zoom levels. The bounds are represented in WGS:84
      # latitude and longitude values, in the order left, bottom, right, top.
      # Values may be integers or floating point numbers.
      bounds: [-180.0, -90.0, 180.0, 90.0]
      # Tile extent in tile coordinate space
      extent: 4096
      # Buffer distance in tile coordinate space to optionally clip geometries
      buffer: 64
      # Boolean to control if geometries should be clipped or encoded as is
      clip_geom: true
      # Geometry type
      geom_type: GEOMETRY
      # Optionally limit the number of features a table source would return (overrides the one above)
      max_feature_count: 5000
      # List of columns, that should be encoded as tile properties. If not set, will be set automatically.
      properties:
        gid: int4

  # Map of source IDs -> function sources
  functions:
    func1:
      # Schema and function name (required)
      schema: public
      function: my_func
      # An integer specifying the minimum zoom level
      minzoom: 0
      # An integer specifying the maximum zoom level. MUST be >= minzoom
      maxzoom: 30
      # The maximum extent of available map tiles.
      bounds: [-180.0, -90.0, 180.0, 90.0]

[sharkAndshark]

The design of API and config above is simple and expressive. : D
A few naive thoughts here.

Feature count limit

Maybe it's an opportunity to resolve #384 .How about give Optional global and inline options:feature_limit. Like:

server:
    ....
    feature_limit: 10000
    ....
sources:
    postgres:
        ....
        tables:
            tbl1:
                ....
                feature_limit:5000
                ....

Duplicate id

A warning(and suggest user use aliases in config file?) and automatically rename duplicates like someDuplicate1,someDuplicate2..etc?

Composite

API	Features	Config
/id1,id2/z/x/y.mvt and /id1,id2.json	flexible
/compositeid1/z/x/y.mvt and /compositeid1.json	not flexible but back end can control everything	restrict composites by something like below

server:
  Composite:
    "compositeid1": tbl1,tbl2
    "compositeid2": tbl3,tbl4,

Positoin of alias part

Maybe place alias part under pg if it is dedicated to pg?

[nyurik]

Thanks @sharkAndshark !

• max_feature_count should only apply to the PG table sources - mbtiles and functions already generate complete tiles, and we can't change anything about them. We can place it in two spots - sources.postgres.max_feature_count (default per PG connection), and sources.postgres.tables.tbl1.max_feature_count (specific override for the tbl1 source). Note that this only does it per single PG source, not all PG sources, but I think this is good enough.
• yes, duplicate IDs would need to give a warning. I am not certain it would make sense to auto-rename them though, but we could do that too. We may want to introduce some --strict CLI flag so that martin wouldn't start if it detects a problem in the configuration.
• Good idea about the composites -- we can easily use aliases for that too, no need to introduce a new config section.

[sharkAndshark]

Great idea that viewing restricting and naming sources combinations as giving it an alias.The underneath logic is that any possible combination is on the table but you should name it before use it.

[nyurik]

I keep making more and more changes to the above proposal. URL suffixes, aliases, overall structure, etc. See edit history to see changes

[maxammann]

I have not yet gone through all the changes, but have a comment about breaking things and being backwards compatible.

Martin is not yet 1.0 so breaking things is not really a problem. Adding code for backwards compatibility should not be needed.

BUT: Changing stuff just for the sake of changing things should be avoided. For example .pbf is the default ending used everywhere. .mvt is quite uncommon.

Every change should have solve a clear problem.

[nyurik]

@maxammann I agree. We should not make any random changes "just because". We should also try to maintain some backward compatibility if it is relatively easy. There could be many users with Martin in production, and migrating servers is always tricky if you cannot easily change the clients at the same time.

The .pbf -> .mvt is part of a more fundamental discussion. TBH, I am not a big fan of .mvt either, and I would rather remove it. This is more question for the future-proofing and optimizing at the same time:

• We do NOT need any extensions for the vector tiles: GET /src/13/42/314 is sufficient without extension
• We MAY want a file extension if we serve static assets for a web site directly from a directory. In that case the source will simply be a path: GET /dir_src/index.html. We may want this path for users who don't want to set up an nginx or other proxy in front of martin.
• We MIGHT need a file extension if we ever want to support multiple styles of rendering, e.g. get me a .png vs .jpeg, but I think this is a useless feature that no one I know has actually needed, so YAGNI principle.
• The same source could be requested as both an MVT or as a PNG, and the extension could become the differentiation between the two, but this won't be enough -- most of the time you have more than one style (e.g. light vs dark), and that additional info needs to be passed to the server. I think GET /src-style/z/x/y is a better variant, also without an extension.

[maxammann]

TBH, I am not a big fan of .mvt either, and I would rather remove it.

I have no opinion about .mvt or .pbf or leaving it away. I, as a martin user, just would like to see benefits of potentially breaking changes. Else flashbacks from JS libraries are kicking in.

We do NOT need any extensions for the vector tiles: GET /src/13/42/314 is sufficient without extension

I agree, but everyone is having the .pbf extension, so users are familiar with the default zxy.pbf schema. I don't see a reason for change.

We MAY want a file extension if we serve static assets for a web site directly from a directory.

Sure, we could do that for users who do not want to setup nginx. But martin definitely will be worse at doing this job compared to nginx. I think we should start from real problems users have and then develop solutions. We should not start with the solution. Else we might get feature requests like:

• Add basic auth
• support cusotm 404 pages etc.

[ivarflakstad]

You could slap a versioning prefix to lessen the impact of breaking changes.
Mark old endpoints as deprecated and give a warning when used.
Just a thought.

[nyurik]

Thx @ivarflakstad - the problem then is that we would have to spend considerable efforts to maintain two different sets of endpoints, with different responses. Plus warnings (unlike errors) in a web service are often missed - noone looks at the logs if everything is working ok.

On top of all this, in a production environment you are likely to have some loadbalancer/nginx/CDN in front of the server - in which case you have a way to do URL rewriting externally if needed.

[ivarflakstad]

maintain two different sets of endpoints, with different responses.

Ah, I was thinking that you would no longer maintain the old ones.

Plus warnings (unlike errors) in a web service are often missed - noone looks at the logs if everything is ok

We do where I work, at least new warnings. I have a sneaking suspicion you are still correct though hehe.

in which case you have a way to do URL rewriting externally if needed.

I guess a user could have two instances of different versions running and move over gradually.

In any case breaking changes might be best/easiest/tidiest after all. It's not like it hasn't been done before :)

[michael-markl]

For the record: It seems these changes were done in #456 and released with v0.7.0 https://github.com/maplibre/martin/releases/tag/v0.7.0 release.

Is there a suggested/tested migration path (for users with Nginx)?