Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: provider groups #1071

Merged
merged 7 commits into from
May 24, 2024
Merged

feat: provider groups #1071

merged 7 commits into from
May 24, 2024

Conversation

alambare
Copy link
Collaborator

in EODAG library:

  • add a by_group option on core.available_providers. If set, we return the unique list of groups instead of the list of provider names.
  • update list_product_types to accept group name as provider names. Calling list_product_types with a group name set in the provider name will return the union of product types from all the providers of this group.
  • update get_search_plugin to accept group names as provider names. Calling get_search_plugin with a group name as provider name will yield all the plugins belonging to the group.

in EODAG server:

  • use group when set instead of provider name

group is an optional field

Example:

I have 2 provider configuration in EODAG:

  • earth_search
  • earth_search_cog

I configure both with group earth_search.

  • When I list the product types for provider earth_search, EODAG return the merged list of product types from earth_search and earth_search_cog.

  • In server mode, the returned product types display earth_search as provider name.

  • When I search for products with provider earth_search, EDOAG search on both provider using the existing fallback mechanism.

  • When I ask for queryables with provider earth_search the response is the intersect of queryables from earth_search and earth_search_cog.

  • available_providers returns only one entry for both: earth_search.

@alambare alambare self-assigned this Mar 22, 2024
@alambare alambare linked an issue Mar 22, 2024 that may be closed by this pull request
Add a provider public name #951
Closed
5 tasks
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024 •
edited
Loading

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         303      51  83.17%   60, 644-690, 792-843, 847
config.py                                      347      27  92.22%   81-83, 92, 100, 104-106, 183, 195, 450-452, 516-519, 566-567, 576-577, 656, 725-730, 732
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    743      76  89.77%   88-97, 371, 596, 640-643, 681, 786, 790-795, 821, 917, 994, 1107, 1197-1209, 1249, 1251, 1255, 1278, 1282-1293, 1306-1312, 1402-1405, 1434-1454, 1511, 1528-1531, 1543-1546, 1914, 1947-1953, 2218, 2222-2225, 2239-2241, 2276
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   52-59, 63-65, 169-176, 260-261, 343, 372, 433, 447-450, 463, 487-490, 533-539
api/product/metadata_mapping.py                677      84  87.59%   67-69, 130-132, 233, 265-266, 312-313, 323-335, 337, 348, 354-366, 407-410, 447, 468-471, 494, 502-503, 579-580, 604-605, 611-614, 629-630, 779, 825, 999-1004, 1135, 1149-1169, 1189, 1194, 1304, 1326, 1340, 1353-1372, 1411, 1463, 1501-1505, 1524
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             131      14  89.31%   49-51, 105-110, 156, 197, 219, 244, 283-284
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      15  84.69%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       117      11  90.60%   40-42, 153, 159, 172-173, 184-189
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      57  78.08%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       529     132  75.05%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-837, 855, 888-897, 923-924, 951-956, 962, 978, 995-996, 1023-1024, 1031, 1093-1099, 1154-1155, 1161, 1171, 1237, 1241, 1255-1271
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         133      14  89.47%   49-54, 108, 112, 275, 295, 380, 383-391
plugins/search/build_search_result.py          179      24  86.59%   62, 97, 142-143, 149, 160, 296-299, 333, 390-407, 469, 472, 482, 499, 527, 529
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          197      65  67.01%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 278-281, 289-300, 317, 319, 326-327, 329-330, 348-352, 385, 392, 403, 416, 422-434, 439
plugins/search/qssearch.py                     559      58  89.62%   96, 368-374, 382-383, 490-496, 551-554, 627-628, 669, 688, 703, 756, 777, 780-781, 790, 801, 810, 833, 895-900, 904-905, 933, 1004, 1051, 1127-1131, 1197-1198, 1219, 1244-1256, 1263, 1294-1295, 1306-1312, 1355, 1370, 1390, 1492
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 5       2  60.00%   21-22
rest/core.py                                   207      17  91.79%   66-72, 261, 345, 605, 607, 610-612, 684, 691-696
rest/server.py                                 299      54  81.94%   80-81, 107, 130-131, 244-246, 262, 302-303, 315-331, 423-428, 460, 627-634, 667, 711-712, 803-805, 822-827, 858, 860, 864-865, 869-870
rest/stac.py                                   455      88  80.66%   63-65, 279, 301, 373, 410-412, 435, 470-471, 557-596, 633-638, 669, 689-690, 875, 940-942, 1161, 1171-1183, 1196-1218, 1232-1277, 1436-1437
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185       9  95.14%   52-55, 236-240, 293, 296, 364
rest/types/stac_queryables.py                   32       2  93.75%   28, 116
rest/types/stac_search.py                      122      10  91.80%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                          96      13  86.46%   51, 108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                              111      11  90.09%   53, 70, 129-132, 199, 214-218, 245, 258
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      42  91.63%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1186, 1204, 1380
utils/constraints.py                           118      38  67.80%   30, 90-99, 140, 145, 149, 160, 186-188, 196, 210-226, 235-246
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   28       1  96.43%   49
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         9033    1554  82.80%

Diff against develop

Filename              Stmts    Miss  Cover
------------------  -------  ------  -------
config.py                +4       0  +0.09%
api/core.py              +4       0  +0.05%
plugins/manager.py       -1       0  -0.08%
rest/stac.py             -9      -2  +0.06%
TOTAL                    -2      -2  +0.02%

Results for commit: 6345230

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024 •
edited
Loading

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         303      51  83.17%   60, 644-690, 792-843, 847
config.py                                      347      28  91.93%   81-83, 92, 100, 104-106, 183, 195, 450-452, 516-519, 566-567, 576-577, 656, 690, 725-730, 732
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    743      76  89.77%   88-97, 371, 596, 640-643, 681, 786, 790-795, 821, 917, 994, 1107, 1197-1209, 1249, 1251, 1255, 1278, 1282-1293, 1306-1312, 1402-1405, 1434-1454, 1511, 1528-1531, 1543-1546, 1914, 1947-1953, 2218, 2222-2225, 2239-2241, 2276
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   52-59, 63-65, 169-176, 260-261, 343, 372, 433, 447-450, 463, 487-490, 533-539
api/product/metadata_mapping.py                677      85  87.44%   67-69, 130-132, 233, 265-266, 312-313, 323-335, 337, 348, 354-366, 407-410, 447, 468-471, 494, 502-503, 579-580, 604-605, 611-614, 629-630, 779, 825, 999-1004, 1135, 1149-1169, 1189, 1194, 1304, 1326, 1340, 1353-1372, 1411, 1463, 1486, 1501-1505, 1524
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             131      14  89.31%   49-51, 105-110, 156, 197, 219, 244, 283-284
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      15  84.69%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       117      11  90.60%   40-42, 153, 159, 172-173, 184-189
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      59  77.31%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       529     132  75.05%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-837, 855, 888-897, 923-924, 951-956, 962, 978, 995-996, 1023-1024, 1031, 1093-1099, 1154-1155, 1161, 1171, 1237, 1241, 1255-1271
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         133      15  88.72%   49-54, 108, 112, 275, 295, 380, 383-391, 393
plugins/search/build_search_result.py          179      31  82.68%   62, 97, 142-143, 149, 160, 296-299, 333, 390-407, 469, 472, 482, 499, 519-534
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          197      65  67.01%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 278-281, 289-300, 317, 319, 326-327, 329-330, 348-352, 385, 392, 403, 416, 422-434, 439
plugins/search/qssearch.py                     559      82  85.33%   96, 368-374, 382-383, 490-496, 551-554, 627-628, 669, 688, 703, 756, 777, 780-781, 790, 801, 810, 833, 895-900, 904-905, 933, 1004, 1051, 1127-1131, 1197-1198, 1219, 1244-1256, 1263, 1294-1295, 1306-1312, 1355, 1370, 1390, 1452-1521
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 5       2  60.00%   21-22
rest/core.py                                   207      96  53.62%   66-72, 153, 155, 157, 163-164, 185-193, 199-205, 252-306, 319-355, 505-536, 554, 604-643, 684, 691-696
rest/server.py                                 299     299  0.00%    18-883
rest/stac.py                                   455     147  67.69%   63-65, 232, 279, 301, 373, 410-412, 435, 470-471, 557-596, 633-638, 669, 677-678, 682-690, 817, 875, 940-942, 959-961, 969-971, 984-986, 1000-1017, 1027-1048, 1058-1080, 1088-1105, 1128-1151, 1161, 1171-1183, 1196-1218, 1232-1277, 1430-1456
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185      18  90.27%   52-55, 236-240, 273-275, 293, 296, 302, 306, 364, 376-379
rest/types/stac_queryables.py                   32       6  81.25%   28, 55-60, 116
rest/types/stac_search.py                      122      12  90.16%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                          96      31  67.71%   51, 79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                              111      36  67.57%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-221, 226, 245, 250, 258, 268
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      43  91.43%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1185-1186, 1204, 1380
utils/constraints.py                           118      38  67.80%   30, 90-99, 140, 145, 149, 160, 186-188, 196, 210-226, 235-246
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   28       1  96.43%   49
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         9033    2060  77.19%

Diff against develop

Filename              Stmts    Miss  Cover
------------------  -------  ------  -------
config.py                +4       0  +0.09%
api/core.py              +4       0  +0.05%
plugins/manager.py       -1       0  -0.08%
rest/stac.py             -9      -2  -0.20%
TOTAL                    -2      -2  +0.01%

Results for commit: 6345230

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@alambare alambare requested review from sbrunato and jlahovnik and removed request for jlahovnik March 22, 2024 14:51
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024 •
edited
Loading

Test Results

    4 files  ± 0      4 suites  ±0   5m 14s ⏱️ -1s
  536 tests + 3    533 ✅ + 3   3 💤 ±0  0 ❌ ±0 
2 144 runs  +12  2 050 ✅ +12  94 💤 ±0  0 ❌ ±0 

Results for commit 6345230. ± Comparison against base commit 4040285.

♻️ This comment has been updated with latest results.

@alambare alambare force-pushed the feat/providers-group-2 branch from 8eb6aa0 to aed086d Compare March 26, 2024 20:58
@alambare alambare force-pushed the feat/providers-group-2 branch from 6f9254c to 3efb3d5 Compare May 22, 2024 15:18
@alambare alambare added dedl DEDL related enhancement New feature or request labels May 22, 2024
@alambare alambare force-pushed the feat/providers-group-2 branch from 3efb3d5 to 2c47142 Compare May 23, 2024 08:42
sbrunato
sbrunato requested changes May 24, 2024
View reviewed changes
eodag/api/core.py Outdated Show resolved Hide resolved
eodag/api/core.py Outdated Show resolved Hide resolved
tests/units/test_core.py Outdated Show resolved Hide resolved
@sbrunato sbrunato marked this pull request as draft May 24, 2024 13:32
sbrunato
sbrunato requested changes May 24, 2024
View reviewed changes
eodag/rest/stac.py Outdated Show resolved Hide resolved
@sbrunato sbrunato force-pushed the feat/providers-group-2 branch from 2c47142 to ff5b54f Compare May 24, 2024 13:59
@sbrunato sbrunato marked this pull request as ready for review May 24, 2024 14:39
@sbrunato sbrunato merged commit 7523dcf into develop May 24, 2024
10 checks passed
@sbrunato sbrunato deleted the feat/providers-group-2 branch May 24, 2024 14:40
@sbrunato sbrunato mentioned this pull request Jun 17, 2024
Open
@sbrunato sbrunato added this to the 3.0.0b1 milestone Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbrunato sbrunato sbrunato approved these changes

Assignees

@alambare alambare

Labels
dedl DEDL related enhancement New feature or request
Projects
None yet
Milestone
3.0.0b1
Development

Successfully merging this pull request may close these issues.

Add a provider public name
2 participants
@alambare @sbrunato