Failed to connect to download.agisoft.com port 80 after 131945 ms:

[abradley60]

The bug

Issues downloading DEMs. My workflow had worked previously for me but I am now running into an issues with the stitch_dem function timing out.

To Reproduce

python=3.10
dem_stitcher==2.5.5
rasterio==1.3.10

dem_data, dem_meta = stitch_dem((106.021858, -68.843872, 119.033562, -63.540306), dem_name='glo_30', dst_ellipsoidal_height=True, dst_area_or_point='Point' )

Additional context

`2024-06-27 05:39:12 INFO Scene bounds : (106.021858, -68.843872, 119.033562, -63.540306)
2024-06-27 05:39:12 INFO Downloding DEM for bounds : (106.021858, -68.843872, 119.033562, -63.540306)
2024-06-27 05:39:12 INFO type of DEM being downloaded : glo_30
2024-06-27 05:39:14 INFO Found credentials in environment variables.
2024-06-27 05:39:14 INFO Found credentials in environment variables.
2024-06-27 05:39:14 INFO Found credentials in environment variables.
2024-06-27 05:39:14 INFO Found credentials in environment variables.
2024-06-27 05:39:14 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 2%|██▏ | 1/46 [00:13<10:26, 13.92s/it]2024-06-27 05:39:28 INFO Found credentials in environment variables.
2024-06-27 05:39:28 INFO Found credentials in environment variables.
2024-06-27 05:39:28 INFO Found credentials in environment variables.
2024-06-27 05:39:28 INFO Found credentials in environment variables.
2024-06-27 05:39:28 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 13%|█████████████▎ | 6/46 [00:24<02:23, 3.59s/it]2024-06-27 05:39:42 INFO Found credentials in environment variables.
2024-06-27 05:39:42 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 15%|███████████████▌ | 7/46 [00:27<02:14, 3.45s/it]2024-06-27 05:39:42 INFO Found credentials in environment variables.
2024-06-27 05:39:42 INFO Found credentials in environment variables.
2024-06-27 05:39:42 INFO Found credentials in environment variables.
2024-06-27 05:39:55 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 24%|████████████████████████▏ | 11/46 [00:41<02:00, 3.44s/it]2024-06-27 05:39:55 INFO Found credentials in environment variables.
2024-06-27 05:39:55 INFO Found credentials in environment variables.
2024-06-27 05:39:55 INFO Found credentials in environment variables.
2024-06-27 05:39:55 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 35%|███████████████████████████████████▏ | 16/46 [00:54<01:32, 3.07s/it]2024-06-27 05:40:09 INFO Found credentials in environment variables.
2024-06-27 05:40:09 INFO Found credentials in environment variables.
2024-06-27 05:40:09 INFO Found credentials in environment variables.
2024-06-27 05:40:09 INFO Found credentials in environment variables.
2024-06-27 05:40:09 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 46%|██████████████████████████████████████████████ | 21/46 [01:05<01:06, 2.66s/it]2024-06-27 05:40:22 INFO Found credentials in environment variables.
2024-06-27 05:40:22 INFO Found credentials in environment variables.
2024-06-27 05:40:22 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 52%|████████████████████████████████████████████████████▋ | 24/46 [01:07<00:48, 2.22s/it]2024-06-27 05:40:22 INFO Found credentials in environment variables.
2024-06-27 05:40:22 INFO Found credentials in environment variables.
2024-06-27 05:40:36 INFO Found credentials in environment variables.
2024-06-27 05:40:36 INFO Found credentials in environment variables.
2024-06-27 05:40:36 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 57%|█████████████████████████████████████████████████████████ | 26/46 [01:21<01:02, 3.14s/it]2024-06-27 05:40:36 INFO Found credentials in environment variables.
2024-06-27 05:40:36 INFO Found credentials in environment variables.
2024-06-27 05:40:44 INFO Found credentials in environment variables.
2024-06-27 05:40:49 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 67%|████████████████████████████████████████████████████████████████████ | 31/46 [01:37<00:47, 3.15s/it]2024-06-27 05:40:54 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 70%|██████████████████████████████████████████████████████████████████████▎ | 32/46 [01:39<00:42, 3.06s/it]2024-06-27 05:40:54 INFO Found credentials in environment variables.
2024-06-27 05:40:54 INFO Found credentials in environment variables.
2024-06-27 05:40:54 INFO Found credentials in environment variables.
2024-06-27 05:40:54 INFO Found credentials in environment variables.
2024-06-27 05:41:07 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 83%|███████████████████████████████████████████████████████████████████████████████████▍ | 38/46 [01:52<00:21, 2.65s/it]2024-06-27 05:41:07 INFO Found credentials in environment variables.
2024-06-27 05:41:07 INFO Found credentials in environment variables.
2024-06-27 05:41:07 INFO Found credentials in environment variables.
Reading glo_30 Datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 46/46 [02:03<00:00, 2.68s/it]
2024-06-27 05:48:09 INFO GDAL signalled an error: err_no=11, msg="CURL error: Failed to connect to download.agisoft.com port 80 after 131945 ms: Couldn't connect to server"
Traceback (most recent call last):
File "rasterio/_base.pyx", line 310, in rasterio._base.DatasetBase.init
File "rasterio/_base.pyx", line 221, in rasterio._base.open_dataset
File "rasterio/_err.pyx", line 221, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_HttpResponseError: CURL error: Failed to connect to download.agisoft.com port 80 after 131945 ms: Couldn't connect to server

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "rtc_otf.py", line 208, in
dem_data, dem_meta = stitch_dem(scene_bounds_buf,
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/stitcher.py", line 367, in stitch_dem
dem_arr, dem_profile = merge_and_transform_dem_tiles(datasets,
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/stitcher.py", line 204, in merge_and_transform_dem_tiles
dem_arr = remove_geoid(dem_arr,
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/geoid.py", line 83, in remove_geoid
geoid_arr, geoid_profile = read_geoid(geoid_name,
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/geoid.py", line 41, in read_geoid
geoid_arr, geoid_profile = read_raster_from_window(geoid_path,
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/rio_window.py", line 113, in read_raster_from_window
with rasterio.open(raster_path) as ds:
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/rasterio/env.py", line 451, in wrapper
return f(*args, **kwds)
File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/rasterio/init.py", line 304, in open
dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.init
rasterio.errors.RasterioIOError: CURL error: Failed to connect to download.agisoft.com port 80 after 131945 ms: Couldn't connect to server

[jlmaurer]

@cmarshak I'm' seeing the same error in RAiDER, just started in the last couple of days.

[cmarshak]

It looks like the geoid link has been taken down from AGIsoft. We will have to put it somewhere else. Have to do this Monday as I am away from computer. It's an easy fix.

[abradley60]

Great, thanks team!

[asjohnston-asf]

Looking at https://www.agisoft.com/downloads/geoids/, the download URLs suggest http://download.agisoft.com/ has migrated to https://s3-eu-west-1.amazonaws.com/download.agisoft.com/ .

I suspect the two files to point geoid.py at are:

• EGM2008 1' geooid model: https://s3-eu-west-1.amazonaws.com/download.agisoft.com/gtg/us_nga_egm2008_1.tif
• EGM96 15' geooid model: https://s3-eu-west-1.amazonaws.com/download.agisoft.com/gtg/us_nga_egm96_15.tif

[asjohnston-asf]

While not linked on https://www.agisoft.com/downloads/geoids/ , files with the existing path/filenames are available at:

• https://s3-eu-west-1.amazonaws.com/download.agisoft.com/geoids/egm2008-1.tif
• https://s3-eu-west-1.amazonaws.com/download.agisoft.com/geoids/egm96-15.tif

I'm not familiar enough with the content to confirm whether they're the exact same files, however.

[jhkennedy]

@abradley60 @jlmaurer This should now be fixed in dem-sticher v2.5.6, which is available on PyPI and coming soon to conda-forge.

[abradley60]

@jhkennedy thanks, can confirm it's working as expected on my end.

[cmarshak]

Thank you, Joe and Andrew!