Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: generate record hash from product_type + id #1023

Merged
merged 3 commits into from
Feb 22, 2024
Merged

feat: generate record hash from product_type + id #1023

merged 3 commits into from
Feb 22, 2024

Conversation

dalpasso
Copy link
Collaborator

@dalpasso dalpasso commented Feb 14, 2024 •
edited by sbrunato
Loading

Fixes #994

Avoid downloading the same product type from different providers.

The record hash is build from the concatenation of product_type + "-" + id

@dalpasso dalpasso requested a review from sbrunato February 14, 2024 08:31
@dalpasso dalpasso self-assigned this Feb 14, 2024
@dalpasso dalpasso linked an issue Feb 14, 2024 that may be closed by this pull request
Support downloaded products check from different providers. #994
Closed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 14, 2024 •
edited
Loading

Test Results

    4 files  ± 0      4 suites  ±0   4m 39s ⏱️ +2s
  470 tests +13    467 ✅ +13   3 💤 ±0  0 ❌ ±0 
1 880 runs  +52  1 798 ✅ +50  82 💤 +2  0 ❌ ±0 

Results for commit 0729b71. ± Comparison against base commit 49209ff.

This pull request removes 10 and adds 23 tests. Note that renamed tests count towards both. 
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_ok
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_count_and_search_peps
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_discover_product_types
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_discover_product_types_keywords
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_distinct_product_type_mtd_mapping
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_no_count_and_search_peps
tests.units.test_search_plugins.TestSearchPluginQueryStringSearch ‑ test_plugins_search_querystringseach_search_cloudcover_peps
tests.units.test_search_plugins.TestSearchPluginQueryStringSearchXml ‑ test_plugins_search_querystringseach_xml_count_and_search_mundi
tests.units.test_search_plugins.TestSearchPluginQueryStringSearchXml ‑ test_plugins_search_querystringseach_xml_distinct_product_type_mtd_mapping
tests.units.test_search_plugins.TestSearchPluginQueryStringSearchXml ‑ test_plugins_search_querystringseach_xml_no_count_and_search_mundi

tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_discover_queryables
tests.units.test_constraints.TestConstraints ‑ test_get_constraint_queryables_with_additional_params
tests.units.test_core.TestCore ‑ test_available_sortables
tests.units.test_core.TestCore ‑ test_list_queryables_with_constraints
tests.units.test_core.TestCoreSearch ‑ test_search_sort_by
tests.units.test_core.TestCoreSearch ‑ test_search_sort_by_raise_errors
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_file_without_zip_extension_ok
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_nonzip_file_with_zip_extension_ok
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_tar_file_with_zip_extension_ok
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_zip_file_ok
…

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 14, 2024 •
edited
Loading

Code Coverage (Ubuntu)

File Coverage
All files 81% ✅

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against f47dae6

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 14, 2024 •
edited
Loading

Code Coverage (Windows)

File Coverage
All files 76% ✅

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against f47dae6

@dalpasso
feat: generate record hash from product_type + id
573ee40 
This avoids downloading the same product type from different providers.
@dalpasso
test: fix broken tests
f47dae6
@dalpasso dalpasso force-pushed the 994-support-downloaded-products-check-from-different-providers branch from 1389ebf to f47dae6 Compare February 14, 2024 09:15
@dalpasso dalpasso marked this pull request as ready for review February 14, 2024 09:16
sbrunato
sbrunato requested changes Feb 22, 2024
View reviewed changes
eodag/plugins/download/base.py Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link
Contributor

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      27  91.35%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    740      74  90.00%   100-108, 381, 592, 637-640, 678, 781, 785-790, 816, 886, 1128-1140, 1180, 1182, 1186, 1207-1209, 1213-1224, 1237-1243, 1333-1336, 1365-1385, 1433, 1439-1442, 1451, 1832-1838, 2101, 2105-2108, 2122-2124, 2189, 2199, 2231, 2234
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          39       3  92.31%   27, 77, 124
api/product/_product.py                        200      30  85.00%   46-51, 55-57, 161-168, 252-253, 343, 379, 440, 454-457, 470, 490-493, 534-540
api/product/metadata_mapping.py                627      84  86.60%   64-66, 127-129, 230, 262-263, 315-327, 329, 340, 344-360, 399-402, 439, 460-463, 477, 535-536, 560-561, 567-570, 585-586, 750, 850-851, 894-899, 1023, 1037-1057, 1077, 1082, 1192, 1214, 1228, 1241-1260, 1299, 1351, 1389-1393, 1412
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            31       8  74.19%   26-29, 93, 97, 162, 207
plugins/apis/cds.py                            204      46  77.45%   71-74, 122-125, 172-184, 240, 305, 313-318, 331-338, 356-358, 382-384, 394-396, 410-411, 456, 459, 469, 481, 514, 516
plugins/apis/ecmwf.py                           94      12  87.23%   48-51, 152-154, 199-200, 226-228
plugins/apis/usgs.py                           165      34  79.39%   59-61, 122, 195, 229, 264-266, 271, 297-298, 303, 331-338, 347-352, 375-381, 383-389, 412
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              82      17  79.27%   32-34, 149-150, 180-201, 227-232
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py        99      60  39.39%   34-36, 133-145, 149-167, 175-209, 215-224, 233-273, 278-286, 291-292
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        478     160  66.53%   74-79, 276, 343-346, 360-364, 407-409, 413, 441-442, 448-452, 482, 550-558, 562, 600-608, 615-617, 658-732, 750-811, 822-827, 843-856, 885, 900-902, 905, 915-923, 931-944, 954-973, 980-992, 1033, 1059, 1104-1106, 1326
plugins/download/base.py                       259      54  79.15%   57-60, 141, 313-314, 334-340, 371-375, 381-382, 426, 429-443, 455, 459, 532-536, 566-567, 575-592, 599-607, 609-613, 660, 683, 705, 713
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       396      92  76.77%   68-73, 107, 151, 171-178, 188-201, 234-239, 289, 321, 348-350, 360-368, 427-431, 488, 563-615, 628, 652-654, 684-685, 693, 696, 712, 736, 758-759, 766, 827-833, 883-884, 890, 900, 956, 974-990
plugins/download/s3rest.py                     111      24  78.38%   53-54, 158, 185, 215-222, 225-227, 231, 244-250, 258-259, 262-266, 289, 309-312
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                          91      11  87.91%   43-44, 91, 95, 121-125, 252, 272
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   52       3  94.23%   55, 73, 104
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          190      59  68.95%   52, 87-90, 106, 118, 137, 143, 149-150, 163-166, 220-221, 225, 235-241, 246, 270, 280-290, 301, 303, 310-311, 313-314, 330-334, 367, 374, 385, 398, 404-416, 421
plugins/search/qssearch.py                     514      54  89.49%   73, 340-341, 435-441, 468-471, 543-544, 603, 618, 671, 692, 695-696, 705, 716, 725, 736-737, 748, 808-813, 817-818, 846, 909, 928-945, 980, 1114-1115, 1136, 1178-1179, 1189-1195, 1232, 1245, 1265, 1353
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/server.py                                 290      36  87.59%   78-79, 104, 127-128, 237-239, 296-297, 309-325, 424-429, 464, 618-625, 661
rest/stac.py                                   417      78  81.29%   57-59, 226-228, 246, 273, 302, 306-315, 334-340, 368-370, 395, 430-431, 609, 791, 856-858, 1077, 1089-1101, 1114-1138, 1152-1199, 1358-1359
rest/utils.py                                  392      63  83.93%   76-78, 202-217, 241, 273, 299, 306, 316, 327-330, 338-339, 368, 500, 518, 546, 548, 581-584, 735-748, 757, 784-799, 824, 914, 933, 941, 1098-1099, 1102, 1104, 1107, 1110-1112, 1132-1136
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                      86      25  70.93%   99-109, 115-117, 123-125, 133-140, 146-148, 154-155, 160-165
rest/types/stac_queryables.py                   38       7  81.58%   28, 120-126, 129
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-60, 71-73, 84-86, 98-100, 112-114, 122
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              470      42  91.06%   76-78, 99-101, 179-180, 189-216, 219, 233, 315-319, 395-399, 420-422, 504, 509, 519, 557-558, 954-957, 965-966, 1007-1008, 1169, 1187, 1360
utils/constraints.py                           120      40  66.67%   81-90, 131, 136, 140, 151, 174, 185-193, 202, 216-232, 241-252
utils/exceptions.py                             36       1  97.22%   23
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8269    1518  81.64%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  --------
plugins/download/base.py         +2       0  +0.16%
plugins/download/s3rest.py       -2       0  -0.38%
TOTAL                             0       0  +100.00%

Results for commit: 0729b71

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@github-actions GitHub Actions
Copy link
Contributor

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      28  91.03%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 625, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    740      78  89.46%   100-108, 381, 592, 637-640, 678, 781, 785-790, 816, 886, 1128-1140, 1180, 1182, 1186, 1207-1209, 1213-1224, 1237-1243, 1333-1336, 1365-1385, 1433, 1439-1442, 1451, 1832-1838, 2101, 2105-2108, 2122-2124, 2166-2173, 2189, 2199, 2231, 2234
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          39       3  92.31%   27, 77, 124
api/product/_product.py                        200      30  85.00%   46-51, 55-57, 161-168, 252-253, 343, 379, 440, 454-457, 470, 490-493, 534-540
api/product/metadata_mapping.py                627      85  86.44%   64-66, 127-129, 230, 262-263, 315-327, 329, 340, 344-360, 399-402, 439, 460-463, 477, 535-536, 560-561, 567-570, 585-586, 750, 850-851, 894-899, 1023, 1037-1057, 1077, 1082, 1192, 1214, 1228, 1241-1260, 1299, 1351, 1374, 1389-1393, 1412
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            31       9  70.97%   26-29, 93, 97, 110, 162, 207
plugins/apis/cds.py                            204      53  74.02%   71-74, 122-125, 172-184, 240, 305, 313-318, 331-338, 356-358, 382-384, 394-396, 410-411, 456, 459, 469, 481, 506-521
plugins/apis/ecmwf.py                           94      12  87.23%   48-51, 152-154, 199-200, 226-228
plugins/apis/usgs.py                           165      34  79.39%   59-61, 122, 195, 229, 264-266, 271, 297-298, 303, 331-338, 347-352, 375-381, 383-389, 412
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              82      17  79.27%   32-34, 149-150, 180-201, 227-232
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py        99      60  39.39%   34-36, 133-145, 149-167, 175-209, 215-224, 233-273, 278-286, 291-292
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        478     160  66.53%   74-79, 276, 343-346, 360-364, 407-409, 413, 441-442, 448-452, 482, 550-558, 562, 600-608, 615-617, 658-732, 750-811, 822-827, 843-856, 885, 900-902, 905, 915-923, 931-944, 954-973, 980-992, 1033, 1059, 1104-1106, 1326
plugins/download/base.py                       259      57  77.99%   57-60, 141, 176, 244-246, 313-314, 334-340, 371-375, 381-382, 426, 429-443, 455, 459, 532-536, 566-567, 575-592, 599-607, 609-613, 660, 683, 705, 713
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       396      92  76.77%   68-73, 107, 151, 171-178, 188-201, 234-239, 289, 321, 348-350, 360-368, 427-431, 488, 563-615, 628, 652-654, 684-685, 693, 696, 712, 736, 758-759, 766, 827-833, 883-884, 890, 900, 956, 974-990
plugins/download/s3rest.py                     111      24  78.38%   53-54, 158, 185, 215-222, 225-227, 231, 244-250, 258-259, 262-266, 289, 309-312
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                          91      11  87.91%   43-44, 91, 95, 121-125, 252, 272
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   52       3  94.23%   55, 73, 104
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          190      59  68.95%   52, 87-90, 106, 118, 137, 143, 149-150, 163-166, 220-221, 225, 235-241, 246, 270, 280-290, 301, 303, 310-311, 313-314, 330-334, 367, 374, 385, 398, 404-416, 421
plugins/search/qssearch.py                     514      78  84.82%   73, 340-341, 435-441, 468-471, 543-544, 603, 618, 671, 692, 695-696, 705, 716, 725, 736-737, 748, 808-813, 817-818, 846, 909, 928-945, 980, 1114-1115, 1136, 1178-1179, 1189-1195, 1232, 1245, 1265, 1313-1382
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/server.py                                 290     290  0.00%    18-792
rest/stac.py                                   417     134  67.87%   57-59, 211, 226-228, 246, 273, 302, 306-315, 334-340, 368-370, 391, 395, 430-431, 609, 732-733, 791, 856-858, 875-877, 885-887, 900-902, 916-933, 943-964, 974-996, 1004-1021, 1044-1067, 1077, 1089-1101, 1114-1138, 1152-1199, 1352-1379
rest/utils.py                                  392     140  64.29%   76-78, 202-217, 241, 273, 299, 302-322, 327-330, 338-339, 368, 500, 518, 544, 546, 548, 575, 581-584, 605, 725-810, 817-825, 914, 928, 933, 935-942, 949, 962-1006, 1097-1127, 1132-1136
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                      86      29  66.28%   99-109, 115-117, 123-125, 133-140, 146-148, 154-155, 160-165, 170-173
rest/types/stac_queryables.py                   38      13  65.79%   28, 53-58, 120-126, 129, 133-134
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-60, 71-73, 84-86, 98-100, 112-114, 122
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              470      42  91.06%   76-78, 99-101, 179-180, 189-216, 219, 233, 315-319, 395-399, 420-422, 504, 509, 519, 557-558, 954-957, 965-966, 1007-1008, 1169, 1187, 1360
utils/constraints.py                           120      40  66.67%   81-90, 131, 136, 140, 151, 174, 185-193, 202, 216-232, 241-252
utils/exceptions.py                             36       1  97.22%   23
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8269    1987  75.97%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  --------
plugins/download/base.py         +2       0  +0.17%
plugins/download/s3rest.py       -2       0  -0.38%
TOTAL                             0       0  +100.00%

Results for commit: 0729b71

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@sbrunato sbrunato merged commit c8189af into develop Feb 22, 2024
10 checks passed
@sbrunato sbrunato deleted the 994-support-downloaded-products-check-from-different-providers branch February 22, 2024 17:42
@sbrunato sbrunato added this to the 3.0.0b1 milestone Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbrunato sbrunato sbrunato approved these changes

Assignees

@dalpasso dalpasso

Labels
None yet
Projects
None yet
Milestone
3.0.0b1
Development

Successfully merging this pull request may close these issues.

Support downloaded products check from different providers.
2 participants
@dalpasso @sbrunato