How to configure Titiler to use my data

[billyz313]

I have installed Titiler on an Ubuntu machine in a conda env using the pip install instructions.

python -m pip  install titiler.core
python -m pip  install titiler.extensions
python -m pip  install titiler.mosaic
python -m pip  install titiler.application
python -m pip install uvicorn

I then created a service to run the app with uvicorn

[Unit]
Description=servirtitiler daemon
After=network.target

[Service]
User=www-data
Group=www-data
SocketUser=www-data
WorkingDirectory=/opt/anaconda3/envs/titiler/lib/python3.9/site-packages/titiler
StandardOutput=file:/var/log/titiler/titiler_gunicorn.log
StandardError=file:/var/log/titiler/titiler_gunicornerror.log
ExecStart=/opt/anaconda3/envs/titiler/bin/uvicorn titiler.application.main:app --uds /servir_apps/socks/titiler.sock
Restart=always

[Install]
WantedBy=multi-user.target

Then I created an nginx config to run the sock

server {
    listen 443 ssl;
    server_name mydomain;
    add_header Access-Control-Allow-Origin *;
    client_max_body_size 8000M;
    client_body_buffer_size 8000M;
    client_body_timeout 120;

    proxy_read_timeout 300;
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_buffers 8 1024k;
    proxy_buffer_size 1024k;
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;
    fastcgi_connect_timeout 90s;
    fastcgi_send_timeout 90s;
    fastcgi_read_timeout 90s;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://unix:/servir_apps/socks/titiler.sock;
    }
}

# Reroute any non https traffic to https
server {
    listen 80;
    server_name mydomain;
    rewrite ^(.*) https://$server_name$1 permanent;
}

It runs perfectly, however I don't know how to point it at my data which is located at in directories for example:

/mnt/scap/fc/mapbiomas/fc_mapbiomas_cog_2000_1ha.tif
/mnt/scap/fc/rlcms_hkh/fc_RLCMS_HKH_cog_2000_1ha.tif
/mnt/scap/fcc/mapbiomas/fcc_mapbiomas_cog_2001_1ha.tif
/mnt/scap/fcc/rlcms_hkh/fcc_RLCMS_HKH_cog_2001_1ha.tif

How do I tell it where these data are?

[vincentsarago]

Connecting to locally files can be a pain 😬

TiTiler's endpoint usually use url={dataset path} query parameter, so I guess if scap is mounted where you are running the services you should be able to just pass ?url=/mnt/scap/fcc/rlcms_hkh/fcc_RLCMS_HKH_cog_2001_1ha.tif

[billyz313]

@vincentsarago Ahhhh, thank you, that was exactly what i needed to know. So to make calls for any of the files in the directory /mnt/scap/fcc/rlcms_hkh/ I would have to individually access them by full path and name, is that correct?

[vincentsarago]

I would have to individually access them by full path and name, is that correct?

Yes

[mradamcox]

It should also work to add this additional location to your nginx config:

server {
  ...
  location /files  {
    add_header Access-Control-Allow-Origin '*';
    alias /mnt/scap;
  }
}

Your files would then be publicly available at https://<yourdomain>/files/fc/mapbiomas/fc_mapbiomas_cog_2000_1ha.tif, though this may not be desirable.

I have a similar setup, but the titiler site and static site are on different subdomains.

[billyz313]

@mradamcox Thank you, this seems like a perfect plan!

[mradamcox]

You bet. I'm no nginx expert, but I assume you could set up some more sophisticated redirects or aliases if needed, or get more specific with the Access-Control-Allow-Origin value.

[billyz313]

I think ur right. We're trying to replace our thredds server with titiler and it has a lot of endpoints...

[billyz313]

All of my datasets are timeseries data, so for example in the directory fcc/rlcms_hkh there are files fc_mapbiomas_cog_{YYYY}_1ha.tif Is there a strategy to identify what is available in those directories? This example seems easy enough to just know, however I also have daily, 3 day, 5 day, 10 day data which gets new data in their interval, but also may have some missing days. Is there something like a virtual aggregation WMS endpoint? Or how do you typically handle this? Also some pertinent information, some of the daily datasets are over 43 years.

FYI: I will be using these in Leaflet to do timeseries animations in ClimateSERV.

[vincentsarago]

All of my datasets are timeseries data, so for example in the directory fcc/rlcms_hkh there are files fc_mapbiomas_cog_{YYYY}_1ha.tif Is there a strategy to identify what is available in those directories?

You could create a custom path_dependency

Also some pertinent information, some of the daily datasets are over 43 years.

It seems that you'll be better served with Zarr 😬

[billyz313]

Ok, i think this will be the final question. I am requesting the tiles and they are returning in the wrong projection "EPSG:4326" instead of "EPSG:3857" I tried adding &crs=EPSG:3857 to the call but it did nothing. I assume i'm missing something, but i can't find documentation about it.

[vincentsarago]

by default the tiles should be in WebMercator (epsg:3857) because the default TileMatrixSet is WebMercatorQuad. if you want the tiles in EPSG4326 you need to use the WGS1984Quad TileMatrixSet

https://titiler.xyz/cog/WGS1984Quad/map?url=s3://ds-satellite/cogs/NaturalEarth/world_grey.tif

https://titiler.xyz/cog/WebMercatorQuad/map?url=s3://ds-satellite/cogs/NaturalEarth/world_grey.tif

[billyz313]

that's what i was hoping for, but https://titiler.servirglobal.net/cog/tiles/3/6/3?tileMatrixSetId=WebMercatorQuad&scale=1&url=%2Fmnt%2Fscap%2Ffcc%2Frlcms_hkh%2Ffcc_RLCMS_HKH_cog_2002_1ha.tif&unscale=false&resampling=nearest&transparent=true&colormap_name=pink_r&return_mask=true&nodata=0 returns a tile that is in 4326 for some reason.
I changed the default projection of the map to confirm when i was seeing the data in the wrong location

[vincentsarago]

https://titiler.servirglobal.net/cog/tiles/WebMercatorQuad/3/6/3 gives tiles in EPSG:3857 projection

rio info https://titiler.servirglobal.net/cog/tiles/WebMercatorQuad/3/6/3.tif\?url\=%2Fmnt%2Fscap%2Ffcc%2Frlcms_hkh%2Ffcc_RLCMS_HKH_cog_2002_1ha.tif\&unscale\=false\&resampling\=nearest\&transparent\=true\&colormap_name\=pink_r\&return_mask\=true\&nodata\=0

{"blockysize": 8, "bounds": [10018754.171394616, 3.725290298461914e-09, 15028131.257091928, 5009377.085697314], "colorinterp": ["red", "green", "blue", "alpha"], "count": 4, "crs": "EPSG:3857", "descriptions": [null, null, null, null], "driver": "GTiff", "dtype": "uint8", "height": 256, "indexes": [1, 2, 3, 4], "interleave": "pixel", "lnglat": [112.49999999999994, 21.94304553343819], "mask_flags": [["per_dataset", "alpha"], ["per_dataset", "alpha"], ["per_dataset", "alpha"], ["all_valid"]], "nodata": null, "res": [19567.879241005125, 19567.879241005117], "shape": [256, 256], "tiled": false, "transform": [19567.879241005125, 0.0, 10018754.171394616, 0.0, -19567.879241005117, 5009377.085697314, 0.0, 0.0, 1.0], "units": [null, null, null, null], "width": 256}

[billyz313]

i just threw this codepen together with just that layer on it https://codepen.io/Billy-Ashmall/pen/WNPewPG (it's a little difficult to see since it's a bunch of dots basically) the dots show up in the ocean I circled them in this screen shot. I also am adding a screen shot of the same where i change leaflet to use crs: 'EPSG:4326' and they show up in the right location. Of course we need it to show up in 3857...