feat: resume interrupted assets download using HTTPDownload

[dalpasso]

In HTTPDownload the assets are downloaded to a temporary file with suffix ~. When the download is completed the file is renamed to remove the suffix. If the file was already correctly downloaded, the asset will not be downloaded again.

[github-actions]

Test Results

    4 files  ± 0      4 suites  ±0   4m 43s ⏱️ +10s
  471 tests +14    468 ✅ +14   3 💤 ±0  0 ❌ ±0 
1 884 runs  +56  1 802 ✅ +54  82 💤 +2  0 ❌ ±0 

Results for commit 5081684. ± Comparison against base commit ebfd3ec.

This pull request removes 10 and adds 24 tests. Note that renamed tests count towards both.

tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_ok
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_count_and_search_peps
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_discover_product_types
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_discover_product_types_keywords
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_distinct_product_type_mtd_mapping
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_no_count_and_search_peps
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_search_cloudcover_peps
tests.units.test_search_plugins.TestSearchPluginQueryStringSearchXml ‑ test_plugins_search_querystringseach_xml_count_and_search_mundi
tests.units.test_search_plugins.TestSearchPluginQueryStringSearchXml ‑ test_plugins_search_querystringseach_xml_distinct_product_type_mtd_mapping
tests.units.test_search_plugins.TestSearchPluginQueryStringSearchXml ‑ test_plugins_search_querystringseach_xml_no_count_and_search_mundi

tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_discover_queryables
tests.units.test_constraints.TestConstraints ‑ test_get_constraint_queryables_with_additional_params
tests.units.test_core.TestCore ‑ test_available_sortables
tests.units.test_core.TestCore ‑ test_list_queryables_with_constraints
tests.units.test_core.TestCoreSearch ‑ test_search_sort_by
tests.units.test_core.TestCoreSearch ‑ test_search_sort_by_raise_errors
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_assets_resume
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_file_without_zip_extension_ok
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_nonzip_file_with_zip_extension_ok
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_tar_file_with_zip_extension_ok
…

♻️ This comment has been updated with latest results.

[github-actions]

Code Coverage (Ubuntu)

File	Coverage
All files	81%	✅

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against 4790eda

[github-actions]

Code Coverage (Windows)

File	Coverage
All files	76%	✅

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against 4790eda

[github-actions]

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      27  91.35%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    740      74  90.00%   100-108, 381, 592, 637-640, 678, 781, 785-790, 816, 886, 1128-1140, 1180, 1182, 1186, 1207-1209, 1213-1224, 1237-1243, 1333-1336, 1365-1385, 1433, 1439-1442, 1451, 1832-1838, 2101, 2105-2108, 2122-2124, 2189, 2199, 2231, 2234
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          39       3  92.31%   27, 77, 124
api/product/_product.py                        200      30  85.00%   46-51, 55-57, 161-168, 252-253, 343, 379, 440, 454-457, 470, 490-493, 534-540
api/product/metadata_mapping.py                627      84  86.60%   64-66, 127-129, 230, 262-263, 315-327, 329, 340, 344-360, 399-402, 439, 460-463, 477, 535-536, 560-561, 567-570, 585-586, 750, 850-851, 894-899, 1023, 1037-1057, 1077, 1082, 1192, 1214, 1228, 1241-1260, 1299, 1351, 1389-1393, 1412
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            31       8  74.19%   26-29, 93, 97, 162, 207
plugins/apis/cds.py                            204      46  77.45%   71-74, 122-125, 172-184, 240, 305, 313-318, 331-338, 356-358, 382-384, 394-396, 410-411, 456, 459, 469, 481, 514, 516
plugins/apis/ecmwf.py                           94      12  87.23%   48-51, 152-154, 199-200, 226-228
plugins/apis/usgs.py                           165      34  79.39%   59-61, 122, 195, 229, 264-266, 271, 297-298, 303, 331-338, 347-352, 375-381, 383-389, 412
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              82      17  79.27%   32-34, 149-150, 180-201, 227-232
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py        99      60  39.39%   34-36, 133-145, 149-167, 175-209, 215-224, 233-273, 278-286, 291-292
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        478     160  66.53%   74-79, 276, 343-346, 360-364, 407-409, 413, 441-442, 448-452, 482, 550-558, 562, 600-608, 615-617, 658-732, 750-811, 822-827, 843-856, 885, 900-902, 905, 915-923, 931-944, 954-973, 980-992, 1033, 1059, 1104-1106, 1326
plugins/download/base.py                       259      54  79.15%   57-60, 141, 313-314, 334-340, 371-375, 381-382, 426, 429-443, 455, 459, 532-536, 566-567, 575-592, 599-607, 609-613, 660, 683, 705, 713
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       402      92  77.11%   68-73, 107, 151, 171-178, 188-201, 234-239, 289, 321, 348-350, 360-368, 427-431, 488, 563-615, 628, 652-654, 684-685, 693, 696, 712, 736, 758-759, 766, 827-833, 883-884, 890, 900, 966, 984-1000
plugins/download/s3rest.py                     111      24  78.38%   53-54, 158, 185, 215-222, 225-227, 231, 244-250, 258-259, 262-266, 289, 309-312
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                          91      11  87.91%   43-44, 91, 95, 121-125, 252, 272
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   52       3  94.23%   55, 73, 104
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          190      59  68.95%   52, 87-90, 106, 118, 137, 143, 149-150, 163-166, 220-221, 225, 235-241, 246, 270, 280-290, 301, 303, 310-311, 313-314, 330-334, 367, 374, 385, 398, 404-416, 421
plugins/search/qssearch.py                     514      54  89.49%   73, 340-341, 435-441, 468-471, 543-544, 603, 618, 671, 692, 695-696, 705, 716, 725, 736-737, 748, 808-813, 817-818, 846, 909, 928-945, 980, 1114-1115, 1136, 1178-1179, 1189-1195, 1232, 1245, 1265, 1353
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/server.py                                 290      36  87.59%   78-79, 104, 127-128, 237-239, 296-297, 309-325, 424-429, 464, 618-625, 661
rest/stac.py                                   417      78  81.29%   57-59, 226-228, 246, 273, 302, 306-315, 334-340, 368-370, 395, 430-431, 609, 791, 856-858, 1077, 1089-1101, 1114-1138, 1152-1199, 1358-1359
rest/utils.py                                  392      63  83.93%   76-78, 202-217, 241, 273, 299, 306, 316, 327-330, 338-339, 368, 500, 518, 546, 548, 581-584, 735-748, 757, 784-799, 824, 914, 933, 941, 1098-1099, 1102, 1104, 1107, 1110-1112, 1132-1136
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                      86      25  70.93%   99-109, 115-117, 123-125, 133-140, 146-148, 154-155, 160-165
rest/types/stac_queryables.py                   38       7  81.58%   28, 120-126, 129
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-60, 71-73, 84-86, 98-100, 112-114, 122
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              470      42  91.06%   76-78, 99-101, 179-180, 189-216, 219, 233, 315-319, 395-399, 420-422, 504, 509, 519, 557-558, 954-957, 965-966, 1007-1008, 1169, 1187, 1360
utils/constraints.py                           120      40  66.67%   81-90, 131, 136, 140, 151, 174, 185-193, 202, 216-232, 241-252
utils/exceptions.py                             36       1  97.22%   23
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8275    1518  81.66%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +6       0  +0.34%
TOTAL                          +6       0  +0.02%

Results for commit: 5081684

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

[github-actions]

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      28  91.03%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 625, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    740      78  89.46%   100-108, 381, 592, 637-640, 678, 781, 785-790, 816, 886, 1128-1140, 1180, 1182, 1186, 1207-1209, 1213-1224, 1237-1243, 1333-1336, 1365-1385, 1433, 1439-1442, 1451, 1832-1838, 2101, 2105-2108, 2122-2124, 2166-2173, 2189, 2199, 2231, 2234
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          39       3  92.31%   27, 77, 124
api/product/_product.py                        200      30  85.00%   46-51, 55-57, 161-168, 252-253, 343, 379, 440, 454-457, 470, 490-493, 534-540
api/product/metadata_mapping.py                627      85  86.44%   64-66, 127-129, 230, 262-263, 315-327, 329, 340, 344-360, 399-402, 439, 460-463, 477, 535-536, 560-561, 567-570, 585-586, 750, 850-851, 894-899, 1023, 1037-1057, 1077, 1082, 1192, 1214, 1228, 1241-1260, 1299, 1351, 1374, 1389-1393, 1412
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            31       9  70.97%   26-29, 93, 97, 110, 162, 207
plugins/apis/cds.py                            204      53  74.02%   71-74, 122-125, 172-184, 240, 305, 313-318, 331-338, 356-358, 382-384, 394-396, 410-411, 456, 459, 469, 481, 506-521
plugins/apis/ecmwf.py                           94      12  87.23%   48-51, 152-154, 199-200, 226-228
plugins/apis/usgs.py                           165      34  79.39%   59-61, 122, 195, 229, 264-266, 271, 297-298, 303, 331-338, 347-352, 375-381, 383-389, 412
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              82      17  79.27%   32-34, 149-150, 180-201, 227-232
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py        99      60  39.39%   34-36, 133-145, 149-167, 175-209, 215-224, 233-273, 278-286, 291-292
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        478     160  66.53%   74-79, 276, 343-346, 360-364, 407-409, 413, 441-442, 448-452, 482, 550-558, 562, 600-608, 615-617, 658-732, 750-811, 822-827, 843-856, 885, 900-902, 905, 915-923, 931-944, 954-973, 980-992, 1033, 1059, 1104-1106, 1326
plugins/download/base.py                       259      57  77.99%   57-60, 141, 176, 244-246, 313-314, 334-340, 371-375, 381-382, 426, 429-443, 455, 459, 532-536, 566-567, 575-592, 599-607, 609-613, 660, 683, 705, 713
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       402      92  77.11%   68-73, 107, 151, 171-178, 188-201, 234-239, 289, 321, 348-350, 360-368, 427-431, 488, 563-615, 628, 652-654, 684-685, 693, 696, 712, 736, 758-759, 766, 827-833, 883-884, 890, 900, 966, 984-1000
plugins/download/s3rest.py                     111      24  78.38%   53-54, 158, 185, 215-222, 225-227, 231, 244-250, 258-259, 262-266, 289, 309-312
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                          91      11  87.91%   43-44, 91, 95, 121-125, 252, 272
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   52       3  94.23%   55, 73, 104
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          190      59  68.95%   52, 87-90, 106, 118, 137, 143, 149-150, 163-166, 220-221, 225, 235-241, 246, 270, 280-290, 301, 303, 310-311, 313-314, 330-334, 367, 374, 385, 398, 404-416, 421
plugins/search/qssearch.py                     514      78  84.82%   73, 340-341, 435-441, 468-471, 543-544, 603, 618, 671, 692, 695-696, 705, 716, 725, 736-737, 748, 808-813, 817-818, 846, 909, 928-945, 980, 1114-1115, 1136, 1178-1179, 1189-1195, 1232, 1245, 1265, 1313-1382
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/server.py                                 290     290  0.00%    18-792
rest/stac.py                                   417     134  67.87%   57-59, 211, 226-228, 246, 273, 302, 306-315, 334-340, 368-370, 391, 395, 430-431, 609, 732-733, 791, 856-858, 875-877, 885-887, 900-902, 916-933, 943-964, 974-996, 1004-1021, 1044-1067, 1077, 1089-1101, 1114-1138, 1152-1199, 1352-1379
rest/utils.py                                  392     140  64.29%   76-78, 202-217, 241, 273, 299, 302-322, 327-330, 338-339, 368, 500, 518, 544, 546, 548, 575, 581-584, 605, 725-810, 817-825, 914, 928, 933, 935-942, 949, 962-1006, 1097-1127, 1132-1136
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                      86      29  66.28%   99-109, 115-117, 123-125, 133-140, 146-148, 154-155, 160-165, 170-173
rest/types/stac_queryables.py                   38      13  65.79%   28, 53-58, 120-126, 129, 133-134
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-60, 71-73, 84-86, 98-100, 112-114, 122
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              470      42  91.06%   76-78, 99-101, 179-180, 189-216, 219, 233, 315-319, 395-399, 420-422, 504, 509, 519, 557-558, 954-957, 965-966, 1007-1008, 1169, 1187, 1360
utils/constraints.py                           120      40  66.67%   81-90, 131, 136, 140, 151, 174, 185-193, 202, 216-232, 241-252
utils/exceptions.py                             36       1  97.22%   23
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8275    1987  75.99%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
plugins/download/http.py       +6       0  +0.34%
TOTAL                          +6       0  +0.02%

Results for commit: 5081684

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results