dem.py strange behavior

[frazucca]

Hi at all,
sorry for the silly question, but for a few days now, I have been having problems using dem.py.

In any configuration (stich or download....) I use the command I get messages like the one below

(isce2) root@9e6f56f50e3b:~# python3 $ISCE_HOME/applications/dem.py -a download -b 45 46 10 11 -s 1 -m xml -r -c
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
curl -n -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45E010.SRTMGL1.hgt.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to e4ftl01.cr.usgs.gov port 80 after 131 ms: Connection refused
2022-08-06 08:43:46,069 - isce.contrib.demUtils.DemStitcher - WARNING - There was a problem in retrieving the file http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45E010.SRTMGL1.hgt.zip. Exception
N45E010.SRTMGL1.hgt.zip = failed

If I use the link (eg. http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45E010.SRTMGL1.hgt.zip) in the program output of course I can download the data. It is therefore not, or at least it seems that way to me, a problem with the instability of http://e4ftl01.cr.usgs.gov/.

Do you have any suggestions in this regard?

Thank you very much and good work
fz

[costantino78]

Same problem for me.

[frazucca]

Hi
some more info:

1. of course I've already setup .netrc (https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+cURL+And+Wget) and indeed I've already use with success dem.py some weeks ago
2. I've extracted the dataset link from dem.py output, and used with success into wget http://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/N45E010.SRTMGL1.hgt.zipURL just by changing the http to https

TIA
regards
fz

[EJFielding]

It seems that the NASA Land Processes DAAC operated by the USGS has changed the server that hosts the SRTM data. It has been having capacity problems that make it unreachable at times, but it seems that they also changed the URL that we have to use. Because of the capacity problems, it has been hard to debug the problem.

Thanks for posting your observations including the successful wget connection.

[geniusinaction]

Is it a case of changing the hard-coded urls in the scripts, then? Where are they all?

I've poked around a little bit and found that they seem to be in different places for different scripts. I was trying something similar to get wbd.py to work (it fails for the same reason with downloading problems), and it does work now that I changed http to https in $ISCE_HOME/components/iscesys/DataManager/SWBDManager.py It seems, though, that I would need to change scripts in other places for dem.py to work?

[EJFielding]

Yes, it does seem that we only need to change those hard-coded URLs. I did that late yesterday in this Pull Request #558 for the DEM scripts. I did not change the water mask file.

[EJFielding]

Thanks @frazucca for tracking down the https.

[geniusinaction]

To confirm, the scripts we need to change are:

$ISCE_HOME/components/contrib/demUtils/DemStitcherND.py
and
$ISCE_HOME/components/contrib/demUtils/DemStitcherV3.py ?

[EJFielding]

I added another similar change to $ISCE_HOME/components/iscesys/DataManager/SWBDManager.py a few minutes ago. Thanks for reminding me about the SWBD download from the same servers.

[EJFielding]

We think this is now fixed by #560 that has already been merged. Please try it out. It works for me, although my first test took almost 15 minutes to get one tile of SRTM. It seems they still have server capacity issues.

[ericlindsey]

Any chance of cutting a new release for the conda-forge version of ISCE that would include this fix? We are required to use conda for our installations on our HPC systems here, so I'm stuck without a working dem.py until the next version is made. I'm thinking about making a full local copy of SRTM instead...

[EJFielding]

@ericlindsey We need to fix a problem with some key Fortran programs that the new gcc10 through gcc12 gfortran compilers reject. The conda-forge compiler is now gcc12.

An alternative is to use the new sardem program that @scottstanie wrote. It is at https://github.com/scottstanie/sardem
It has a pip install option. (See below; it seems there is now a conda-forge install also.)

[geniusinaction]

@ericlindsey I wonder if some kind of sed-based hack of the 6 or so python scripts that need http replaced with https would be a workable interim solution? All of those files should be installed locally through conda.

[scottstanie]

fyi sardem is also now available via conda install -c conda-forge sardem : https://anaconda.org/conda-forge/sardem
There's a notebook showing the differences between dem.py and sardem with the -isce for processing with ISCE2 : https://github.com/scottstanie/sardem/blob/master/notebooks/isce2_usage.ipynb

[EJFielding]

Yes, for the UNAVCO2022 InSAR with ISCE+ they did a standard conda install of version 2.6.0 and then copied over the six or seven Python scripts that need to be changed to fix the DEM downloading.

[ericlindsey]

Thanks for the suggestions @geniusinaction and @EJFielding! I was able to fix all the http -> https lines with this script, I hope others find it useful:

#!/bin/bash
# base path for conda installation of ISCE
path="$CONDA_PREFIX/lib/python3.8/site-packages/isce/components"
files=( contrib/demUtils/DemStitcherV3.py contrib/demUtils/DemStitcherND.py contrib/demUtils/SWBDStitcher.py 
isceobj/Alos2Proc/runDownloadDem.py iscesys/DataManager/Dem1Manager.py iscesys/DataManager/Dem3Manager.py 
iscesys/DataManager/SWBDManager.py )
for file in "${files[@]}"; do
    ls $path/$file
    sed -i 's|http://e4ftl01|https://e4ftl01|' $path/$file
    grep http://e4ftl01 $path/$file
done

However, the curl commands are still not working. Even running them manually, I can't figure out the issue - the server always gives '401 Unauthorized', regardless whether I use the .netrc or add -u username:password. Oddly though, with wget I am able to work through the 401 error; it gets the error but then seemingly finds another method (?) that allows it to download the files. Seems like this is beyond my pay grade... I'll check out sardem; thanks @scottstanie!

[piyushrpt]

There are issues with LPDAAC right now. The first URL needs to be called without auth and any redirects should then include the auth information. The initial call is what is raising the 401s, when it used to be ok in the past. They are working on trying to restore the old behavior

[hongsy0421]

I have a simple comment, which can revise all the "http://adress" to "https://adress",

1. cd to installation directory of isce
2. (sudo) sed -i 's?http://e4ft?https://e4ft?g' `grep -rl "http://e4ft" ./`
   This comment has been tested in Ubuntu. After the fix, the dem.py can download the SRTM normally.

[EJFielding]

Thanks for the sed command!