Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: server mode collections search #909

Merged
merged 10 commits into from
Mar 8, 2024

Conversation

dalpasso
Copy link
Collaborator

@dalpasso dalpasso commented Nov 1, 2023

Implements server-mode free text search of collections https://github.com/stac-api-extensions/collection-search#free-text-search.

Uses updated guess_product_type method that now accepts free_text_filter and intersect parameters.
See updated examples on how to guess a product type.

@dalpasso dalpasso requested a review from sbrunato November 1, 2023 08:21
@dalpasso dalpasso linked an issue Nov 1, 2023 that may be closed by this pull request
Copy link
Contributor

github-actions bot commented Nov 1, 2023

Test Results

    4 files  ±0      4 suites  ±0   5m 55s ⏱️ +58s
  503 tests +2    500 ✅ +2   3 💤 ±0  0 ❌ ±0 
2 012 runs  +8  1 926 ✅ +6  86 💤 +2  0 ❌ ±0 

Results for commit 55a9a62. ± Comparison against base commit d1df25e.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Nov 1, 2023

Code Coverage (Ubuntu)

File Coverage
All files 81%

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against c056162

Copy link
Contributor

github-actions bot commented Nov 1, 2023

Code Coverage (Windows)

File Coverage
All files 76%

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against c056162

@alambare alambare marked this pull request as draft November 16, 2023 16:37
@alambare alambare removed the request for review from sbrunato November 16, 2023 16:37
@alambare
Copy link
Collaborator

@dalpasso the following STAC extension should be added to the server conformance classes:

https://api.stacspec.org/v1.0.0-rc.1/collection-search#free-text

@dalpasso dalpasso force-pushed the 852-server-mode-collections-search branch from 3171d91 to 73ee524 Compare November 22, 2023 11:06
@dalpasso
Copy link
Collaborator Author

Feature reimplemented in EODataAccessGateway.list_product_types() using whoosh

@dalpasso dalpasso marked this pull request as ready for review November 22, 2023 11:09
@dalpasso dalpasso requested a review from sbrunato November 22, 2023 11:09
@alambare alambare force-pushed the 852-server-mode-collections-search branch from 73ee524 to f6ae81b Compare November 23, 2023 17:17
@alambare alambare removed the request for review from sbrunato November 23, 2023 17:19
@alambare
Copy link
Collaborator

@dalpasso dalpasso marked this pull request as draft November 24, 2023 08:14
@dalpasso dalpasso marked this pull request as ready for review November 24, 2023 09:34
@alambare alambare requested a review from sbrunato November 24, 2023 15:10
@alambare alambare force-pushed the 852-server-mode-collections-search branch from 8297df8 to 89dacda Compare February 6, 2024 11:51
@dalpasso dalpasso force-pushed the 852-server-mode-collections-search branch from c056162 to 6e9e38f Compare February 29, 2024 15:31
Copy link
Contributor

github-actions bot commented Feb 29, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      27  91.35%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    750      79  89.47%   101-110, 383, 594, 638-641, 679, 784, 788-793, 819, 889, 959, 1076-1081, 1167-1179, 1219, 1221, 1225, 1246-1248, 1252-1263, 1276-1282, 1372-1375, 1404-1424, 1472, 1478-1481, 1490, 1841, 1874-1880, 2143, 2147-2150, 2164-2166, 2231, 2241, 2273, 2276
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        208      32  84.62%   48-55, 59-61, 165-172, 256-257, 347, 383, 444, 458-461, 474, 498-501, 544-550
api/product/metadata_mapping.py                648      87  86.57%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 346-362, 401-404, 441, 462-465, 479, 487-488, 561-562, 586-587, 593-596, 611-612, 776, 822, 879-880, 923-928, 1059, 1073-1093, 1113, 1118, 1228, 1250, 1264, 1277-1296, 1335, 1387, 1425-1429, 1448
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            38      10  73.68%   36-43, 103, 107, 174, 219
plugins/apis/cds.py                            204      49  75.98%   72-80, 126-129, 176-188, 244, 309, 317-322, 335-342, 360-362, 383-385, 395-397, 411-412, 457, 460, 470, 482, 515, 517
plugins/apis/ecmwf.py                           98      15  84.69%   48-56, 157-159, 206-207, 233-235
plugins/apis/usgs.py                           168      36  78.57%   59-64, 125, 198, 232, 267-269, 274, 300-301, 306, 334-341, 350-355, 377-383, 385-391, 414
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       2  89.47%   26, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        490     165  66.33%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 453-454, 460-464, 497, 532, 536, 543, 573-581, 585, 623-631, 638-640, 681-755, 773-834, 845-850, 866-879, 908, 923-925, 928, 938-946, 954-967, 977-996, 1003-1015, 1056, 1082, 1127-1129, 1349
plugins/download/base.py                       261      57  78.16%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       418      97  76.79%   81-88, 122, 170, 191-198, 208-221, 258-263, 313, 345, 372-374, 384-392, 414, 453-457, 514, 587-643, 657, 681-683, 713-714, 722, 725, 741, 765, 788-789, 796, 855-861, 916-917, 923, 933, 999, 1017-1033
plugins/download/s3rest.py                     116      27  76.72%   55-58, 124, 165, 192, 222-229, 232-234, 238, 251-257, 265-266, 269-273, 296, 316-319
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         101      13  87.13%   43-47, 101, 105, 131-135, 264, 284
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          195      60  69.23%   52, 89-92, 108, 120, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 276, 286-297, 309, 311, 318-319, 321-322, 337-341, 374, 381, 392, 405, 411-423, 428
plugins/search/qssearch.py                     517      55  89.36%   86, 353-354, 448-454, 481-484, 557-558, 599, 617, 632, 685, 706, 709-710, 719, 730, 739, 750-751, 762, 822-827, 831-832, 860, 923, 942-959, 994, 1128-1129, 1150, 1192-1193, 1203-1209, 1249, 1262, 1282, 1370
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/core.py                                   168      13  92.26%   238, 506-507, 510, 512, 515, 518-520, 572, 579-583
rest/server.py                                 306      54  82.35%   81-82, 106, 129-130, 241-243, 259, 299-300, 312-328, 410-415, 445, 607-614, 645, 688-689, 782-784, 801-806, 836, 838, 842-843, 847-848
rest/stac.py                                   435      95  78.16%   59-61, 229-231, 273, 286-295, 314-320, 365, 402-404, 427, 462-463, 549-594, 639, 659-660, 839, 904-906, 1125, 1135-1147, 1160-1182, 1196-1241, 1400-1401
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185       9  95.14%   52-55, 231-235, 288, 291, 359
rest/types/stac_queryables.py                   38       7  81.58%   28, 120-126, 129
rest/types/stac_search.py                      122      10  91.80%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                         116      14  87.93%   53, 108-109, 128-130, 180, 190-204, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              486      43  91.15%   82, 87, 108-110, 188-189, 198-225, 228, 242, 324-328, 404-408, 429-431, 513, 518, 528, 566-567, 963-966, 974-975, 1016-1017, 1097, 1181, 1199, 1372
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8654    1568  81.88%

Diff against develop

Filename        Stmts    Miss  Cover
------------  -------  ------  -------
api/core.py        +8       0  +0.12%
rest/stac.py       +7      +2  -0.11%
TOTAL             +15      +2  +0.01%

Results for commit: 55a9a62

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Feb 29, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      28  91.03%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 625, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    750      83  88.93%   101-110, 383, 594, 638-641, 679, 784, 788-793, 819, 889, 959, 1076-1081, 1167-1179, 1219, 1221, 1225, 1246-1248, 1252-1263, 1276-1282, 1372-1375, 1404-1424, 1472, 1478-1481, 1490, 1841, 1874-1880, 2143, 2147-2150, 2164-2166, 2208-2215, 2231, 2241, 2273, 2276
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        208      32  84.62%   48-55, 59-61, 165-172, 256-257, 347, 383, 444, 458-461, 474, 498-501, 544-550
api/product/metadata_mapping.py                648      88  86.42%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 346-362, 401-404, 441, 462-465, 479, 487-488, 561-562, 586-587, 593-596, 611-612, 776, 822, 879-880, 923-928, 1059, 1073-1093, 1113, 1118, 1228, 1250, 1264, 1277-1296, 1335, 1387, 1410, 1425-1429, 1448
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            38      11  71.05%   36-43, 103, 107, 120, 174, 219
plugins/apis/cds.py                            204      56  72.55%   72-80, 126-129, 176-188, 244, 309, 317-322, 335-342, 360-362, 383-385, 395-397, 411-412, 457, 460, 470, 482, 507-522
plugins/apis/ecmwf.py                           98      15  84.69%   48-56, 157-159, 206-207, 233-235
plugins/apis/usgs.py                           168      36  78.57%   59-64, 125, 198, 232, 267-269, 274, 300-301, 306, 334-341, 350-355, 377-383, 385-391, 414
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        490     165  66.33%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 453-454, 460-464, 497, 532, 536, 543, 573-581, 585, 623-631, 638-640, 681-755, 773-834, 845-850, 866-879, 908, 923-925, 928, 938-946, 954-967, 977-996, 1003-1015, 1056, 1082, 1127-1129, 1349
plugins/download/base.py                       261      59  77.39%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       418      97  76.79%   81-88, 122, 170, 191-198, 208-221, 258-263, 313, 345, 372-374, 384-392, 414, 453-457, 514, 587-643, 657, 681-683, 713-714, 722, 725, 741, 765, 788-789, 796, 855-861, 916-917, 923, 933, 999, 1017-1033
plugins/download/s3rest.py                     116      27  76.72%   55-58, 124, 165, 192, 222-229, 232-234, 238, 251-257, 265-266, 269-273, 296, 316-319
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         101      13  87.13%   43-47, 101, 105, 131-135, 264, 284
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          195      60  69.23%   52, 89-92, 108, 120, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 276, 286-297, 309, 311, 318-319, 321-322, 337-341, 374, 381, 392, 405, 411-423, 428
plugins/search/qssearch.py                     517      79  84.72%   86, 353-354, 448-454, 481-484, 557-558, 599, 617, 632, 685, 706, 709-710, 719, 730, 739, 750-751, 762, 822-827, 831-832, 860, 923, 942-959, 994, 1128-1129, 1150, 1192-1193, 1203-1209, 1249, 1262, 1282, 1330-1399
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/core.py                                   168      60  64.29%   135, 137, 139, 145-146, 163-171, 178-184, 230-259, 413-440, 458, 505-535, 572, 579-583
rest/server.py                                 306     306  0.00%    18-861
rest/stac.py                                   435     154  64.60%   59-61, 214, 229-231, 273, 286-295, 314-320, 365, 402-404, 427, 462-463, 549-594, 639, 647-648, 652-660, 781, 839, 904-906, 923-925, 933-935, 948-950, 964-981, 991-1012, 1022-1044, 1052-1069, 1092-1115, 1125, 1135-1147, 1160-1182, 1196-1241, 1394-1420
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185      18  90.27%   52-55, 231-235, 268-270, 288, 291, 297, 301, 359, 371-374
rest/types/stac_queryables.py                   38      13  65.79%   28, 53-58, 120-126, 129, 133-134
rest/types/stac_search.py                      122      12  90.16%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                         116      31  73.28%   53, 79-85, 105, 108-109, 128-130, 147, 173-181, 188-209, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              486      43  91.15%   82, 87, 108-110, 188-189, 198-225, 228, 242, 324-328, 404-408, 429-431, 513, 518, 528, 566-567, 963-966, 974-975, 1016-1017, 1097, 1181, 1199, 1372
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8654    2034  76.50%

Diff against develop

Filename        Stmts    Miss  Cover
------------  -------  ------  -------
api/core.py        +8       0  +0.12%
rest/stac.py       +7      +7  -1.05%
TOTAL             +15      +7  -0.04%

Results for commit: 55a9a62

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@sbrunato sbrunato force-pushed the 852-server-mode-collections-search branch 2 times, most recently from 787bfc6 to 9ecb183 Compare March 8, 2024 10:38
@sbrunato sbrunato force-pushed the 852-server-mode-collections-search branch from 9ecb183 to 241127f Compare March 8, 2024 10:42
@sbrunato sbrunato force-pushed the 852-server-mode-collections-search branch from 88e0ec5 to 55a9a62 Compare March 8, 2024 14:07
@sbrunato sbrunato merged commit b7e64c3 into develop Mar 8, 2024
10 checks passed
@sbrunato sbrunato deleted the 852-server-mode-collections-search branch March 8, 2024 14:20
@sbrunato sbrunato added this to the 3.0.0b1 milestone Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

server mode collections search
3 participants