ASF GRFN products and GDAL3's vsis3

[piyushrpt]

Hi,
Don't know if this is the right place to report this. Most of us end up using jupyter via conda and looks like gdal3 build with conda might not be recognizing netcdf_mem.h correctly. If this is not an issue with custom build GDAL; then this should be reported to conda-forge/gdal-feedstock.

I remember this being addressed by OSGeo/gdal#1328

Here is a test script that shows that GRFN products are interpreted as HDF5:

#This is from https://media.asf.alaska.edu/uploads/InSAR/temporary_security_credentials.py
from json import loads
from requests import get

credential_url = 'https://grfn.asf.alaska.edu/door/credentials'
response = get(credential_url)
response.raise_for_status()

credentials = loads(response.text)['Credentials']

## This is how you use ASF s3 credentials with GDAL
from osgeo import gdal
gdal.SetConfigOption('AWS_REGION', 'us-east-1')
gdal.SetConfigOption('AWS_SECRET_ACCESS_KEY', credentials['SecretAccessKey'])
gdal.SetConfigOption('AWS_ACCESS_KEY_ID', credentials['AccessKeyId'])
gdal.SetConfigOption('AWS_SESSION_TOKEN', credentials['SessionToken'])

results = gdal.Info('/vsis3/grfn-content-prod/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc')
print(results)

Piyush

[piyushrpt]

Turns out this is related to them building conda packages on old kernels:

userfaultfd support: no
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=55722

This is unlikely to change anytime soon.

Piyush

[emlundell]

@piyushrpt We don't have a formal way to announce possible bugs that aren't directly related to to notebook servers but are worth mentioning. So I guess that issuing a issue is as good as anything else.

The current notebook servers are built using GDAL 2.4. For ISCE, It's install on Ubuntu 18.04 via (https://github.com/asfadmin/asf-jupyter-container/blob/master/dockerfile):

apt install gdal-bin
pip install 'gdal==2.4.0'

I ran the test script you mentioned in the production cluster and got

Driver: HDF5/Hierarchical Data Format Release 5
Files: /vsis3/grfn-content-prod/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc
Size is 512, 512
...

So it would appear that GRFN products are interpreted as HDF5.

[piyushrpt]

Tracking this here: conda-forge/gdal-feedstock#323

[emlundell]

I reran the script against production again and got

Driver: netCDF/Network Common Data Format
Files: /vsis3/grfn-content-prod/S1-GUNW-D-R-160-tops-20190710_20190628-162436-20935N_18926N-PP-6a53-v2_0_2.nc
Size is 512, 512
...

Since the tracked conda-forge link has closed the issue and the products are reported as netCDF, this issue appears to be resolved.