Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add cop_marine provider #1131

Merged
merged 38 commits into from
Jun 13, 2024
Merged

feat: add cop_marine provider #1131

merged 38 commits into from
Jun 13, 2024

Conversation

anesson-cs
Copy link
Collaborator

@anesson-cs anesson-cs commented May 4, 2024

  • Search, download and discover product types implemented
  • new search plugin which constructs the search results based on the products and datasets in the stac API and the items available in the s3
  • download is done via the AwsDownload plugin

Copy link
Contributor

github-actions bot commented May 4, 2024

Test Results

    4 files  ±0      4 suites  ±0   6m 2s ⏱️ +37s
  544 tests +2    541 ✅ +2   3 💤 ±0  0 ❌ ±0 
2 176 runs  +8  2 078 ✅ +8  98 💤 ±0  0 ❌ ±0 

Results for commit 909dcd3. ± Comparison against base commit 2c37db3.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented May 4, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     10       0  100.00%
cli.py                                         302      51  83.11%   60, 644-690, 792-843, 847
config.py                                      350      27  92.29%   81-83, 92, 100, 104-106, 183, 195, 459-461, 525-528, 575-576, 585-586, 665, 734-739, 741
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    762      84  88.98%   89-98, 372, 597, 641-644, 682, 719, 792, 796-801, 827, 923, 1000, 1113, 1203-1215, 1255, 1257, 1261, 1284, 1288-1299, 1312-1318, 1408-1411, 1440-1460, 1517, 1534-1537, 1549-1552, 1569-1576, 1944, 1977-1983, 2248, 2252-2255, 2269-2271, 2306
api/search_result.py                            50       6  88.00%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   56-63, 67-69, 173-180, 264-265, 347, 376, 437, 451-454, 467, 491-494, 537-543
api/product/metadata_mapping.py                678      83  87.76%   67-69, 131-133, 234, 266-267, 313-314, 324-336, 338, 349, 355-367, 412-413, 450, 471-474, 497, 505-506, 582-583, 607-608, 614-617, 632-633, 782, 828, 1002-1007, 1138, 1152-1172, 1192, 1197, 1307, 1329, 1343, 1356-1375, 1414, 1466, 1504-1508, 1527
api/product/drivers/__init__.py                  5       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             133      15  88.72%   49-51, 105-110, 156, 197, 219, 223, 248, 287-288
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           99      15  84.85%   46-54, 150-152, 199-200, 226-228
plugins/apis/usgs.py                           175      36  79.43%   60-65, 132, 205, 239, 274-276, 281, 307-308, 313, 343-350, 361-366, 388-394, 396-402, 425
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              49       6  87.76%   33-35, 132, 156-161
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       186      19  89.78%   41-43, 121, 135-160, 168, 322-325, 349
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      19  72.06%   28-30, 33, 72-75, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      57  78.08%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       531     133  74.95%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-840, 858, 891-900, 926-927, 954-959, 965, 968, 984, 1001-1002, 1029-1030, 1037, 1099-1105, 1160-1161, 1167, 1177, 1243, 1247, 1261-1277
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                      26       3  88.46%   27-31
plugins/search/base.py                         133      13  90.23%   48-53, 104, 108, 121, 271, 291, 350-351, 371, 380
plugins/search/build_search_result.py          183      24  86.89%   62, 97, 141-142, 148, 159, 295-298, 327, 384-401, 463, 466, 476, 493, 521, 523
plugins/search/cop_marine.py                   179      40  77.65%   43-44, 55, 65-66, 109-111, 123-124, 164-173, 177, 182, 197, 233, 237, 252, 256, 260, 264, 268-272, 278-281, 284-297, 314, 337, 344
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          108      83  23.15%   44-46, 58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          198      65  67.17%   53, 90-93, 109, 121, 125-126, 137, 142, 147, 154, 167-170, 224-225, 229, 239-245, 250, 276-279, 287-298, 315, 317, 324-325, 327-328, 346-350, 383, 390, 401, 414, 420-432, 437
plugins/search/qssearch.py                     646      69  89.32%   102, 386, 390-396, 404-405, 511-517, 567, 583, 593, 620, 622, 665-668, 743-744, 787, 806, 821, 879, 900, 903-904, 913, 924, 933, 956, 1020-1025, 1029-1030, 1063, 1137, 1186, 1260-1264, 1330-1331, 1352, 1380-1392, 1399, 1431-1433, 1443-1449, 1493, 1508, 1528, 1631-1641
plugins/search/static_stac_search.py            73      12  83.56%   37-38, 95-122, 138, 151
rest/__init__.py                                 4       2  50.00%   21-22
rest/core.py                                   225      35  84.44%   72-78, 267, 351, 611, 613, 616-618, 690, 697-745
rest/server.py                                 299      53  82.27%   80-81, 107, 130-131, 244-246, 302-303, 315-331, 423-428, 460, 627-634, 667, 711-712, 803-805, 822-827, 858, 860, 864-865, 869-870
rest/stac.py                                   480      91  81.04%   63-65, 328, 350, 422, 459-461, 484, 519-520, 606-645, 682-687, 777-781, 815, 835-836, 979, 1044-1046, 1265, 1275-1287, 1300-1322, 1336-1381, 1540-1541
rest/types/__init__.py                           0       0  100.00%
rest/types/eodag_search.py                     184       9  95.11%   52-55, 235-239, 292, 295, 363
rest/types/stac_queryables.py                   38       2  94.74%   34, 140
rest/types/stac_search.py                      121      10  91.74%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                          95      13  86.32%   51, 108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                              110      11  90.00%   53, 70, 129-132, 199, 214-218, 245, 258
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              501      41  91.82%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1204, 1380
utils/constraints.py                           119      37  68.91%   94-103, 144, 149, 153, 164, 190-192, 202, 216-232, 241-252
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   27       1  96.30%   49
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 143, 159-162, 215-224, 234-264
TOTAL                                         9475    1658  82.50%

Diff against develop

Filename                                Stmts    Miss  Cover
------------------------------------  -------  ------  -------
plugins/crunch/filter_overlap.py            0      +2  -2.94%
plugins/search/cop_marine.py             +179     +40  +77.65%
plugins/search/static_stac_search.py      +25      +9  -10.19%
utils/stac_reader.py                      +20     +17  -9.77%
TOTAL                                    +224     +68  -0.31%

Results for commit: 909dcd3

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented May 4, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     10       0  100.00%
cli.py                                         302      51  83.11%   60, 644-690, 792-843, 847
config.py                                      350      28  92.00%   81-83, 92, 100, 104-106, 183, 195, 459-461, 525-528, 575-576, 585-586, 665, 699, 734-739, 741
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    762      84  88.98%   89-98, 372, 597, 641-644, 682, 719, 792, 796-801, 827, 923, 1000, 1113, 1203-1215, 1255, 1257, 1261, 1284, 1288-1299, 1312-1318, 1408-1411, 1440-1460, 1517, 1534-1537, 1549-1552, 1569-1576, 1944, 1977-1983, 2248, 2252-2255, 2269-2271, 2306
api/search_result.py                            50       6  88.00%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   56-63, 67-69, 173-180, 264-265, 347, 376, 437, 451-454, 467, 491-494, 537-543
api/product/metadata_mapping.py                678      83  87.76%   67-69, 131-133, 234, 266-267, 313-314, 324-336, 338, 349, 355-367, 412-413, 450, 471-474, 497, 505-506, 582-583, 607-608, 614-617, 632-633, 782, 828, 1002-1007, 1138, 1152-1172, 1192, 1197, 1307, 1329, 1343, 1356-1375, 1414, 1466, 1504-1508, 1527
api/product/drivers/__init__.py                  5       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             133      15  88.72%   49-51, 105-110, 156, 197, 219, 223, 248, 287-288
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           99      15  84.85%   46-54, 150-152, 199-200, 226-228
plugins/apis/usgs.py                           175      36  79.43%   60-65, 132, 205, 239, 274-276, 281, 307-308, 313, 343-350, 361-366, 388-394, 396-402, 425
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              49       6  87.76%   33-35, 132, 156-161
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       186      19  89.78%   41-43, 121, 135-160, 168, 322-325, 349
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      19  72.06%   28-30, 33, 72-75, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      59  77.31%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       531     133  74.95%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-840, 858, 891-900, 926-927, 954-959, 965, 968, 984, 1001-1002, 1029-1030, 1037, 1099-1105, 1160-1161, 1167, 1177, 1243, 1247, 1261-1277
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                      26       3  88.46%   27-31
plugins/search/base.py                         133      18  86.47%   48-53, 104, 108, 121, 271, 291, 350-351, 371, 374-382, 384
plugins/search/build_search_result.py          183      31  83.06%   62, 97, 141-142, 148, 159, 295-298, 327, 384-401, 463, 466, 476, 493, 513-528
plugins/search/cop_marine.py                   179      40  77.65%   43-44, 55, 65-66, 109-111, 123-124, 164-173, 177, 182, 197, 233, 237, 252, 256, 260, 264, 268-272, 278-281, 284-297, 314, 337, 344
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          108      83  23.15%   44-46, 58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          198      65  67.17%   53, 90-93, 109, 121, 125-126, 137, 142, 147, 154, 167-170, 224-225, 229, 239-245, 250, 276-279, 287-298, 315, 317, 324-325, 327-328, 346-350, 383, 390, 401, 414, 420-432, 437
plugins/search/qssearch.py                     646      97  84.98%   102, 386, 390-396, 404-405, 511-517, 567, 570, 583, 593, 612-627, 665-668, 743-744, 787, 806, 821, 879, 900, 903-904, 913, 924, 933, 956, 1020-1025, 1029-1030, 1063, 1137, 1186, 1260-1264, 1330-1331, 1352, 1380-1392, 1399, 1431-1433, 1443-1449, 1493, 1508, 1528, 1590-1661
plugins/search/static_stac_search.py            73      12  83.56%   37-38, 95-122, 138, 151
rest/__init__.py                                 4       2  50.00%   21-22
rest/core.py                                   225     114  49.33%   72-78, 159, 161, 163, 169-170, 191-199, 205-211, 258-312, 325-361, 511-542, 560, 610-649, 690, 697-745
rest/server.py                                 299     299  0.00%    18-883
rest/stac.py                                   480     150  68.75%   63-65, 262, 328, 350, 422, 459-461, 484, 519-520, 606-645, 682-687, 777-781, 815, 823-824, 828-836, 921, 979, 1044-1046, 1063-1065, 1073-1075, 1088-1090, 1104-1121, 1131-1152, 1162-1184, 1192-1209, 1232-1255, 1265, 1275-1287, 1300-1322, 1336-1381, 1534-1560
rest/types/__init__.py                           0       0  100.00%
rest/types/eodag_search.py                     184      18  90.22%   52-55, 235-239, 272-274, 292, 295, 301, 305, 363, 375-378
rest/types/stac_queryables.py                   38       8  78.95%   34, 63-68, 77-78, 140
rest/types/stac_search.py                      121      12  90.08%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                          95      31  67.37%   51, 79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                              110      36  67.27%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-221, 226, 245, 250, 258, 268
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              501      41  91.82%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1204, 1380
utils/constraints.py                           119      37  68.91%   94-103, 144, 149, 153, 164, 190-192, 202, 216-232, 241-252
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   27       1  96.30%   49
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 143, 159-162, 215-224, 234-264
TOTAL                                         9475    2173  77.07%

Diff against develop

Filename                                Stmts    Miss  Cover
------------------------------------  -------  ------  -------
plugins/crunch/filter_overlap.py            0      +2  -2.94%
plugins/search/cop_marine.py             +179     +40  +77.65%
plugins/search/static_stac_search.py      +25      +9  -10.19%
utils/stac_reader.py                      +20     +17  -9.77%
TOTAL                                    +224     +68  -0.18%

Results for commit: 909dcd3

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@anesson-cs anesson-cs marked this pull request as draft May 4, 2024 22:37
@jlahovnik jlahovnik force-pushed the add-cop-marine-provider branch 2 times, most recently from 4d6e8f3 to b25a6d2 Compare May 24, 2024 15:09
@jlahovnik jlahovnik self-assigned this May 24, 2024
@jlahovnik jlahovnik marked this pull request as ready for review May 24, 2024 15:49
@sbrunato sbrunato force-pushed the add-cop-marine-provider branch from 3bd1fea to f410f02 Compare June 4, 2024 07:56
@jlahovnik jlahovnik force-pushed the add-cop-marine-provider branch from 20ef280 to 526f0cc Compare June 11, 2024 10:16
eodag/cli.py Outdated Show resolved Hide resolved
eodag/cli.py Outdated Show resolved Hide resolved
eodag/resources/product_types.yml Outdated Show resolved Hide resolved
eodag/resources/providers.yml Outdated Show resolved Hide resolved
@sbrunato sbrunato marked this pull request as draft June 13, 2024 07:16
@sbrunato sbrunato marked this pull request as ready for review June 13, 2024 08:57
@sbrunato sbrunato changed the title Add cop_marine provider feat: add cop_marine provider Jun 13, 2024
@sbrunato sbrunato merged commit 19e0d05 into develop Jun 13, 2024
10 checks passed
@sbrunato sbrunato deleted the add-cop-marine-provider branch June 13, 2024 09:01
@sbrunato sbrunato added this to the 3.0.0b1 milestone Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants