feat: titiler-pgstac v1 upgrade #398

smohiudd · 2024-07-18T14:56:29Z

Upgrade to titiler-pgstac 1.3.0 using migration guide.

Updated raster API deployed to dev.openveda.cloud/api/raster/docs. Changes include:

rename mosaic/ endpoint to searches/
add new /collections/{collection_id} endpoint
add new /colorMaps endpoint
rename stac/ endpoint to /collections/{collection_id}/items/{item_id}
rename stac-alt/ endpoint to /alt/collections/{collection_id}/items/{item_id}

smohiudd · 2024-07-18T14:57:16Z

@vincentsarago do we need to upgrade pgstac as well? We're currently running 0.7.10

vincentsarago · 2024-07-18T20:35:26Z

do we need to upgrade pgstac as well? We're currently running 0.7.10

no, I think you're fine

vincentsarago · 2024-07-18T20:37:15Z

✨ looks good to me, did you use https://stac-utils.github.io/titiler-pgstac/latest/migrations/v1_migration/? was it useful?

smohiudd · 2024-07-18T20:50:00Z

Yes very useful! I did notice a discrepancy in this section. It imports add_search_register_route and add_mosaic_register_route but uses:

add_search_register_route(
    app,
    # any dependency we want to validate
    # when creating the tilejson/map links
    tile_dependencies=[
        mosaic.layer_dependency,
        mosaic.dataset_dependency,
        mosaic.pixel_selection_dependency,
        mosaic.process_dependency,
        mosaic.rescale_dependency,
        mosaic.colormap_dependency,
        mosaic.render_dependency,
        mosaic.pgstac_dependency,
        mosaic.reader_dependency,
        mosaic.backend_dependency,
    ],
)
# add /list endpoint
add_search_list_route(app)

I'm assuming we have to import add_search_list_route instead?

vincentsarago · 2024-07-18T21:06:16Z

~~@smohiudd so weird the documentation web site doesn't point to the write .md files 😭~~

☝️ ignore, this was fixed in stac-utils/titiler-pgstac@6d84641

smohiudd · 2024-07-18T22:59:11Z

When running the raster api docker compose I got this error: titiler-pgstac 1.3.0 depends on titiler.core<0.19 and >=0.18.0. So upgrading titiler core/mosaic/extensions to >=0.18.5,<0.19

anayeaye · 2024-07-22T15:52:18Z

@smohiudd I think we also need to replace/remove VEDA_RASTER_ENABLE_MOSAIC_SEARCH from docker-compose

slesaad · 2024-07-22T16:13:26Z

@smohiudd we also wanted to add the colormap endpoints - should we do it in the same PR?

smohiudd · 2024-07-22T20:44:35Z

raster_api/runtime/src/app.py

@@ -67,24 +78,36 @@ async def lifespan(app: FastAPI):
 # /mosaic - PgSTAC Mosaic titiler endpoint
 ###############################################################################
 mosaic = MosaicTilerFactory(
-    router_prefix="/mosaic",
+    router_prefix="/searches/{search_id}",
+    path_dependency=SearchIdParams,
    optional_headers=optional_headers,
    environment_dependency=settings.get_gdal_config,
    process_dependency=PostProcessParams,


Do we still need process_dependency if we have tile_dependencies below?

@smohiudd the process dependency is different from the tile_dependency. I see that we have custom post_process method, so if they are still used we should keep the process_dependency

smohiudd · 2024-07-22T20:45:58Z

@vincentsarago how come we don't have the /collections/{collection_id} endpoint: https://dev.openveda.cloud/api/raster/docs

anayeaye · 2024-07-22T22:14:56Z

Are these deprecated searches visible in the swagger docs as a courtesy to inform users that users must now supply {TileMatrixSetId} instead of expecting a default web mercator?

anayeaye · 2024-07-22T22:45:25Z

raster_api/runtime/src/app.py

-# TODO
-# prefix will be replaced by `/mosaics/{search_id}` in titiler-pgstac 1.0.0
+# add /list endpoint
+add_search_list_route(app, prefix="/searches", tags=["Mosaic"])


Is this the equivalent of the list registered mosaics endpoint that we were setting in the .env as VEDA_RASTER_ENABLE_MOSAIC_SEARCH and then

Inserting in the lambda env in infrastructure construct

And using the runtime config to conditionally enable the list endpoint which could potentially be expensive because it is not paginated (or it was not yet paginated at some point)?

If so, do we want it always on now?

vincentsarago · 2024-07-23T07:03:23Z

@smohiudd

@vincentsarago how come we don't have the /collections/{collection_id} endpoint: https://dev.openveda.cloud/api/raster/docs

you need to register a new set of endpoint using the MosaicTilerFactory factory

https://github.com/stac-utils/titiler-pgstac/blob/6d84641beda8ab28a79a1864d7700b30926c141a/titiler/pgstac/main.py#L172-L188

vincentsarago · 2024-07-23T07:05:01Z

Are these deprecated searches visible in the swagger docs as a courtesy to inform users that users must now supply {TileMatrixSetId} instead of expecting a default web mercator?

@anayeaye, yes! in next major version of titiler we will remove the default tms and thus the user will always have to pass a TileMatrixSetId. This is to match the OGC Tiles specification

vincentsarago · 2024-07-24T15:02:10Z

raster_api/runtime/src/app.py

-# TODO
-# prefix will be replaced by `/mosaics/{search_id}` in titiler-pgstac 1.0.0
+# add /list endpoint
+add_search_list_route(app, prefix="/searches", tags=["Mosaic"])


I guess the tags should be Searches ?

vincentsarago · 2024-07-24T15:15:27Z

raster_api/runtime/src/app.py

+)
+app.include_router(mosaic.router, prefix="/searches/{search_id}", tags=["Mosaic"])
+
+add_search_register_route(


add

if settings.enable_mosaic_search: add_search_register_route(

vincentsarago · 2024-07-24T15:30:08Z

raster_api/runtime/src/app.py

+app.include_router(
+    collection.router, tags=["STAC Collection"], prefix="/collections/{collection_id}"
+)
+

 ###############################################################################
 # /stac - Custom STAC titiler endpoint


from titiler.pgstac.dependencies import ItemIdParams stac = MultiBaseTilerFactory( reader=PgSTACReader, path_dependency=ItemIdParams, optional_headers=optional_headers, router_prefix="/stac", environment_dependency=settings.get_gdal_config, router=APIRouter(route_class=LoggerRouteHandler), extensions=[ stacViewerExtension(), ], colormap_dependency=ColorMapParams, ) app.include_router(stac.router, tags=["Items"], prefix="/collections/{collection_id}/items/{item_id}") ############################################################################### # /alt - Custom STAC titiler endpoint for alternate asset locations ############################################################################### stac_alt = MultiBaseTilerFactory( reader=PgSTACReaderAlt, path_dependency=ItemIdParams, optional_headers=optional_headers, router_prefix="/alt/collections/{collection_id}/items/{item_id}", environment_dependency=settings.get_gdal_config, router=APIRouter(route_class=LoggerRouteHandler), extensions=[ stacViewerExtension(), ], colormap_dependency=ColorMapParams, ) app.include_router(stac_alt.router, tags=["Items"], prefix="/alt/collections/{collection_id}/items/{item_id}")

raster_api/runtime/src/app.py

…configuration for cloudfront, titiler-pgstac v1 upgrade (#405) ### 📣 Breaking - [feat!: titiler-pgstac v1 upgrade #398](#398) introduces breaking endpoint changes; see [titiler-pgstac v0.8 -> v1.0 migration docs](https://stac-utils.github.io/titiler-pgstac/latest/migrations/v1_migration/) ### ✨ Added - [feat: add optional web acl configuration for cloudfront #396](#396) ### 🪙 Changed/Updated - [chore: Improve stac integration tests #395](#395) - [chore: add gitignore entry for Jetbrains IDEs #409](#409) ### 🩹 Fixed - [fix: render fix for titiler-pgstac v1 upgrade #408](#408)

…1075) **Related Ticket:** #1070 ⚠️ **This needs to be coordinated with veda BE team on when this can be merged. BE team needs to merge their changes into their prod before this can merge.** ### Description of Changes Updating raster api (titiler) endpoints as part of new version breaking changes. This does not worry about integrating new endpoints. ### Notes & Questions About Changes * [VEDA BACKEND PR](NASA-IMPACT/veda-backend#398) for ref * [Titiler version migration guide](https://stac-utils.github.io/titiler-pgstac/latest/migrations/v1_migration/#searchid-id) for ref ### Relevant API Changes [**yes**] rename mosaic/ endpoint to searches/ [**no**] for `/register` endpoints, update `searchId` to `id` [**no**] a dd new /collections/{collection_id} endpoint [**no**] add new /colorMaps endpoint [**no**] rename stac/ endpoint to /collections/{collection_id}/items/{item_id} [**no**] rename stac-alt/ endpoint to /alt/collections/{collection_id}/items/{item_id} ### Validation / Testing _{Update with info on what can be manually validated in the Deploy Preview link for example "Validate style updates to selection modal do NOT affect cards"}_

initial changes for titiler pgstac v1 upgrade

213fcec

smohiudd requested review from botanical and vincentsarago July 18, 2024 14:56

format

6361c8d

smohiudd changed the title ~~Titiler-pgstac v1 upgrade~~ feat: titiler-pgstac v1 upgrade Jul 18, 2024

add add_search_list_route

12ec468

smohiudd and others added 4 commits July 19, 2024 07:03

update titiler.core

47ffa3d

fix jinja2 template

9342294

fix list

cdbaa28

fix dependency

fcf3fed

vincentsarago force-pushed the feature/pgstac-v1-upgrade branch from 9572a73 to fcf3fed Compare July 19, 2024 16:25

fix tests

a5ecbcc

smohiudd mentioned this pull request Jul 19, 2024

Plans for upgrading to TiTiler-PgSTAC 1.0? #364

Closed

3 tasks

smohiudd marked this pull request as ready for review July 19, 2024 16:51

smohiudd requested a review from anayeaye July 19, 2024 17:48

vincentsarago approved these changes Jul 19, 2024

View reviewed changes

smohiudd added 3 commits July 22, 2024 13:28

change route prefix

cdd8986

re-add colormap dependency import

ddb3a5f

format

d455440

smohiudd commented Jul 22, 2024

View reviewed changes

update tests

d4578f1

format

139182e

anayeaye reviewed Jul 22, 2024

View reviewed changes

aboydnw mentioned this pull request Jul 23, 2024

[E & A] Implement colormap configurability NASA-IMPACT/veda-ui#994

Closed

4 tasks

vincentsarago reviewed Jul 24, 2024

View reviewed changes

add collection endpoint

2db7ac6

vincentsarago reviewed Jul 24, 2024

View reviewed changes

smohiudd added 3 commits July 24, 2024 09:33

change items endpoint

89f2d0f

update alt endpoint

6fecd6c

add endpoint tags

5ba94f3

smohiudd requested review from anayeaye and slesaad July 24, 2024 19:38

smohiudd added 3 commits July 24, 2024 14:10

fix tests

dd96a7f

format

7a6848e

update tags

ab06ebb

anayeaye reviewed Jul 24, 2024

View reviewed changes

raster_api/runtime/src/app.py Outdated Show resolved Hide resolved

format, tag change

4652db8

anayeaye reviewed Jul 24, 2024

View reviewed changes

raster_api/runtime/src/app.py Show resolved Hide resolved

not include alt schema

4e10923

anayeaye approved these changes Jul 25, 2024

View reviewed changes

smohiudd merged commit f449325 into develop Jul 25, 2024
4 checks passed

smohiudd deleted the feature/pgstac-v1-upgrade branch July 25, 2024 15:27

This was referenced Jul 25, 2024

Update Raster API endpoints as part of new version breaking change NASA-IMPACT/veda-ui#1070

Open

Update Raster API endpoints as part of new version breaking changes NASA-IMPACT/veda-ui#1075

Merged

botanical mentioned this pull request Jul 30, 2024

feat(release)!: Improve stac integration tests, add optional web acl configuration for cloudfront, titiler-pgstac v1 upgrade #405

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: titiler-pgstac v1 upgrade #398

feat: titiler-pgstac v1 upgrade #398

smohiudd commented Jul 18, 2024 •

edited

Loading

smohiudd commented Jul 18, 2024

vincentsarago commented Jul 18, 2024

vincentsarago commented Jul 18, 2024

smohiudd commented Jul 18, 2024

vincentsarago commented Jul 18, 2024 •

edited

Loading

smohiudd commented Jul 18, 2024

anayeaye commented Jul 22, 2024

slesaad commented Jul 22, 2024

smohiudd Jul 22, 2024 •

edited

Loading

vincentsarago Jul 23, 2024

smohiudd commented Jul 22, 2024

anayeaye commented Jul 22, 2024

anayeaye Jul 22, 2024

vincentsarago commented Jul 23, 2024

vincentsarago commented Jul 23, 2024

vincentsarago Jul 24, 2024

vincentsarago Jul 24, 2024 •

edited

Loading

vincentsarago Jul 24, 2024

feat: titiler-pgstac v1 upgrade #398

feat: titiler-pgstac v1 upgrade #398

Conversation

smohiudd commented Jul 18, 2024 • edited Loading

smohiudd commented Jul 18, 2024

vincentsarago commented Jul 18, 2024

vincentsarago commented Jul 18, 2024

smohiudd commented Jul 18, 2024

vincentsarago commented Jul 18, 2024 • edited Loading

smohiudd commented Jul 18, 2024

anayeaye commented Jul 22, 2024

slesaad commented Jul 22, 2024

smohiudd Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

vincentsarago Jul 23, 2024

Choose a reason for hiding this comment

smohiudd commented Jul 22, 2024

anayeaye commented Jul 22, 2024

anayeaye Jul 22, 2024

Choose a reason for hiding this comment

vincentsarago commented Jul 23, 2024

vincentsarago commented Jul 23, 2024

vincentsarago Jul 24, 2024

Choose a reason for hiding this comment

vincentsarago Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

vincentsarago Jul 24, 2024

Choose a reason for hiding this comment

smohiudd commented Jul 18, 2024 •

edited

Loading

vincentsarago commented Jul 18, 2024 •

edited

Loading

smohiudd Jul 22, 2024 •

edited

Loading

vincentsarago Jul 24, 2024 •

edited

Loading