Inconsistent results against NETCDF-4 data #1328
Comments
|
gdalinfo on your local machine is correctly picking up the NETCDF driver whereas the version with vsicurl seems to be using the HDF5 driver, which is a fallback option when the file is not recognized by the NETCDF driver. Could you provide the output of the gdalinfo vsicurl version with CPL_DEBUG=ON set. |
|
/vsicurl/ support in the netCDF driver requires a recent enough version of libnetcdf which installs the netcdf_mem.h header file |
|
bash-4.2# gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc |
|
We were running against libnetcdf v4.3.3.1 which lacked the netcdf_mem.h header file. I built v4.6.2 by hand and confirmed that netcdf_mem.h is now there. I am still seeing GDAL selecting the wrong driver over vsicurl. An example is in the response above. |
|
Did you 'make clean' after running GDAL configure again ? |
|
I did a fresh build in a new container for the new version of netcdf. I see: about 10 lines from the end of my ./configure output. |
|
Does |
|
It does not. I'll investigate. Thanks! |
|
what does |
|
Linux 7df9fe41b690 4.14.101-91.76.amzn2.x86_64 #1 SMP Tue Feb 19 17:51:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
|
hum, well,then you can then try running under gdb, and set a break point on CPLIsUserFaultMappingSupported. It should be called and return true |
|
Ok, I think that's my next step. Thanks for the help @rouault ! |
|
The plot thickens! I tried building on another architecture and looked though the logs more carefully this time. In the gdal configure I noticed: checking libnetcdf compiler and linker flags with nc-config... I'm fairly certain I built netcdf correctly. Do you think this could be an indication that build is getting fouled? |
absolutely. It would seem that netcdf_mem.h cannot be included alone. (I haven't tested with netcdf 4.6, so it might be possible that the header in this version depends on other headers) |
|
It gets worse. I rolled back to netcdf version 4.6.0 after seeing a post describing a the same problem with v4.6.2 in another project. When I configured GDAL this time everything looks peachy: Full of excitement I ran gdal info only to see it was still selecting the HDF5 driver. |
|
What version of NETCDF are you running? |
hum, I've a weird setup currently... It appears I'm building against netcdf 4.4 from Ubuntu 16.04 but I manually added netcdf_mem.h in /usr/include, but at runtime I'm linking against a manually compiled netcdf 4.5.0, and somehow that works... |
|
@idixon ... do the debug messages show that it first attempts reading it with NETCDF before falling back on HDF5, now that you are building GDAL with netcdf_mem.h? |
It does not appear so. I'm going to try rebuilding with Even's versions today and see if that helps. |
|
Are you seeing results like mine on your end? |
|
The last time I tried (when the PR was merged), I was able to use my old proj library and it worked. I need to setup my proj6 before giving this a shot again. |
|
I've just committed in a529c1a a fix needed to detect netcdf_mem.h in recent netcdf versions (tested with 4.6.2). |
|
Change backported to release/2.4 branch |
|
Everything looks great against NETCDF v4.6.2 now! |
|
ok, closing that issue then |
|
Thank you for the help @rouault and @piyushrpt! |
Expected behavior and actual behavior.
Running GDAL against NETCDF-4 groups has different results depending on if the data is local or accessed using vsicurl.
Run locally gdal_translate extracts a geotiff with spatial information however a gdalinfo on the container doesn't appear to return the correct information.
Run over vsicurl it's the opposite problem. A gdalinfo on the container provides verbose information but a gdal_translate appears to be incomplete.
Ideally GDAL should behave the same locally and over vsicurl.
Steps to reproduce the problem.
A sample product has been staged here:
https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
Running GDAL against the sample file (support was just recently added for NETCDF-4 groups):
bash-4.2# gdalinfo /vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
bash-4.2# gdal_translate HDF5:"/vsicurl/https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc"://science/grids/data/unwrappedPhase unwrappedPhase.tif
bash-4.2# gdalinfo unwrappedPhase.tif
bash-4.2# curl -o S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc https://s3.amazonaws.com/idixon/S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc
bash-4.2# gdalinfo S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.n
bash-4.2# gdal_translate NETCDF:"S1-GUNW-D-R-042-tops-20190103_20181210-140759-38619N_36645N-PP-24f6-v2_0_0.nc":/science/grids/data/unwrappedPhase phase.tif
bash-4.2# gdalinfo phase.tif
Operating system
Amazon Linux 2 AMI 2.0.20190115 x86_64 HVM gp2
GDAL version and provenance
GDAL 2.5.0dev-a2d4e6740d, released 2019/02/26
The text was updated successfully, but these errors were encountered: