Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support z,x,y and record-returning funcs, table rework #380

Merged
merged 13 commits into from
Dec 10, 2022

Conversation

nyurik
Copy link
Member

@nyurik nyurik commented Aug 8, 2022

Can now handle several additional Postgres functions to get a tile, plus tons of small fixes

Multiple result variants

  • getmvt(z,x,y) -> [bytea,md5] (single row with two columns)
  • getmvt(z,x,y) -> [bytea] (single row with a single column)
  • getmvt(z,x,y) -> bytea (value)

Multiple input parameter variants

  • getmvt(z, x, y) or getmvt(zoom, x, y) (all 3 vars must be integers)
  • getmvt(z, x, y, url_query), where instead of url_query it could be any other name, but must be of type JSON

Breaking

  • srid is now the same type as PG -- i32
  • renamed config vals table_sources and function_sources into tables and functions

Features and fixes

Notes

  • More dynamic SQL generation in code instead of using external SQL files. Those should only be used when they are not parametrized.
  • The new function/table discovery mechanism: query for all functions in the database, and match up those functions with the ones configured (if any), plus adds all the rest of the un-declared ones if discovery mode is on.
  • During table and function discovery, the code generates a map of (PgSqlInfo, FunctionInfo) (or table) tupples containing SQL needed to get the tile.
  • Auto-discovery mode is currently hidden - the discovery is on only when no tables or functions are configured. TBD - how to configure it in the future
  • The new system allows for an easy way to auto-discover for the specific schemas only, solving martin config expose only specific schema #47
  • predictable order of table/function instantiation
  • bounding boxes computed in parallel for all tables (when not configured)
  • proper identifier escaping
  • test cleanup

fixes #378
fixes #466
fixes #65
fixes #389

@nyurik nyurik requested a review from stepankuzmin August 8, 2022 05:59
@nyurik nyurik changed the title [WIP] Support z,x,y and record-returning funcs Support z,x,y and record-returning funcs Nov 30, 2022
@nyurik nyurik marked this pull request as ready for review November 30, 2022 05:48
@nyurik nyurik enabled auto-merge (squash) November 30, 2022 20:30
@nyurik nyurik disabled auto-merge November 30, 2022 20:31
@nyurik nyurik enabled auto-merge (squash) November 30, 2022 20:31
@nyurik nyurik changed the title Support z,x,y and record-returning funcs Support z,x,y and record-returning funcs, table rework Dec 3, 2022
This was referenced Dec 4, 2022
nyurik added 12 commits December 6, 2022 21:41
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a subsequent PR to add other function tests
Can handle the `getmvt(z,x,y) -> [bytea,md5]` case and several others.
* All table sources now use the same system as functions
* Proper table name escaping
* Prepared statements might speed things up, but this is not certain because we prepare on each get request. At least now we can optimize in one place.
BREAKING:
* srid is now the same type as PG -- i32
* renamed config vals `table_sources` and `function_sources` into `tables` and `functions`

Fixes:
* predictable order of instantiation
* bounding boxes computed in parallel for all tables (when not configured)
* split up discovery and query creation - this way user overrides happen before the final query is generated
* more proper name escaping
* lots of test improvements
* Fix id column issue
* Detect if postgis is newer than 3.1 and use st_tileenvolope margin param
@nyurik nyurik disabled auto-merge December 7, 2022 17:00
@nyurik nyurik enabled auto-merge (squash) December 7, 2022 17:00
Copy link
Collaborator

@stepankuzmin stepankuzmin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good 👍

assert_eq!(t, &["bytea", "text"]);
}
(None, None) => {}
_ => panic!("Invalid output record names or types"),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe skip such functions and use warn! instead of panic! here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a validation that SQL matches Rust code -- and uses lots of asserts for that. In theory the whole thing can be rewritten to use warnings, but I would rather panic early and fix the logic bugs than to make it go unnoticed. TBD i guess

@nyurik nyurik merged commit 2ee517d into maplibre:main Dec 10, 2022
@nyurik nyurik deleted the omt-support branch December 10, 2022 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants