fix the 10k max results problem #705
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
agnesgaroux
approved these changes
Aug 23, 2023
paul-butcher
added a commit
that referenced
this pull request
Feb 5, 2024
* Genre concept ids (#639) * add genre concept ids to image filters * Apply auto-formatting rules * update test docs * update test docs * add works filter test * new test docs * genre concept id filter on works * add genre concept ids to image filters * Apply auto-formatting rules * use new availability data in aggregation test * update images includes tests * update availabilities test data * update test for new data * update test for new data * update test for new data * remove .id from genre concept filters --------- Co-authored-by: Buildkite on behalf of Wellcome Collection <[email protected]> * Apply auto-formatting rules * Copy across the new example documents for images * Add filtering by production date on images * Add sorting by source date on images * Apply auto-formatting rules * Annotate diff tool (#651) * first stab * annotate on nano * Apply auto-formatting rules * try without the colours * term wrapping * annotation styling * annotation styling * double dollar * put it in the right pipeline * add artifact path --------- Co-authored-by: Buildkite on behalf of Wellcome Collection <[email protected]> * update the rank tests to use a changed work id * delete rogue m * corrected [search_docs] and [snapshots] URLs * mark test which searches for multiple reference numbers as a knownFailure * Remove references to _queryType * Apply auto-formatting rules * Bump scala-libs to v32.25.0 * Bump scala-libs to v32.26.0 * 5695-use new 2023-06-01 pipeline, now with added Accession Numbers * These are not index tests * Allow failures of e2e tests * new pipeline (#663) * update id for failing reference number test * Reduce the cost of running the catalogue API * Use Fargate Spot for running stage services * Turn off the stage APIs outside office hours Note: this check is deliberately a conservative check for the `stage` environment (opting in to cost-saving behaviour) rather than a loose check that we're not in the `prod` environment (opting out). This means we're less likely to implement cost-saving in an environment which is part of a production service. * Remove AnyFunSpec from base trait (#673) * remove AnyFunSpec from base class * remove AnyFunSpec from base class * commit missed file * removes mustQueries and QueryConfig * Do not return unwanted empty buckets (#674) * remove AnyFunSpec from base class * remove AnyFunSpec from base class * commit missed file * Do not return unwanted empty buckets * Do not return unwanted empty buckets * extend no-empties rule to images * avoid naming collision * better naming, as per review * Bump scala-libs to v32.27.0 * updates imagesParams to use RBGColor decoder + new ColorQuery * Fix paired aggregation (#679) * remove AnyFunSpec from base class * remove AnyFunSpec from base class * commit missed file * Do not return unwanted empty buckets * Do not return unwanted empty buckets * extend no-empties rule to images * avoid naming collision * better naming, as per review * Fix paired aggregation behaviour (#676) * Fix paired aggregation behaviour * improve safety * fix sorting even more * improve commentary * Karl and Jake * remove redundant test * bumps akka to match scala-libs * image filter test fails * Apply auto-formatting rules * Aggregation tests (#667) * start gherkining the facets * start filling out the faceting features * Apply auto-formatting rules * messing about with portability of the features file * only return populated buckets * finish new faceting feature tests for works. * add todo about order * a bit more faceting test finessing * WorkFacet tests pass (todo: spread the filtering business to other fields) * aggregate properly on language * workType filtering and aggregation * subjects and contributors * availabilities * tidy up bucket matching * improve filter test output * license works properly * skip tests that are not yet ready * filters now only return the filtered value in aggregations * remove AnyFunSpec from base class * remove AnyFunSpec from base class * commit missed file * Do not return unwanted empty buckets * Do not return unwanted empty buckets * extend no-empties rule to images * avoid naming collision * new faceting paradigm * enforce order in aggregations * remove unnecessary ignored scenarios * fix test to correspond ot actual filtering * fix low-level tests that expected the old-style query * fix the "all other" test * improve commentary * tidying * revert irrelevant change * improve commentary * revert irrelevant change * revert irrelevant change * Fix paired aggregation behaviour * add test to ensure all filtered values are returned * improve safety * include parens in a test * fix sorting even more * improve commentary * better rule name * Karl and Jake * remove redundant test * start adding Image faceting * finish adding Image faceting tests * better naming, as per review --------- Co-authored-by: Buildkite on behalf of Wellcome Collection <[email protected]> * Define the main Works search query in plain Json (#682) * start gherkining the facets * start filling out the faceting features * Apply auto-formatting rules * messing about with portability of the features file * only return populated buckets * finish new faceting feature tests for works. * add todo about order * a bit more faceting test finessing * WorkFacet tests pass (todo: spread the filtering business to other fields) * aggregate properly on language * workType filtering and aggregation * subjects and contributors * availabilities * tidy up bucket matching * improve filter test output * license works properly * skip tests that are not yet ready * filters now only return the filtered value in aggregations * remove AnyFunSpec from base class * remove AnyFunSpec from base class * commit missed file * Do not return unwanted empty buckets * Do not return unwanted empty buckets * extend no-empties rule to images * avoid naming collision * new faceting paradigm * enforce order in aggregations * remove unnecessary ignored scenarios * fix test to correspond ot actual filtering * fix low-level tests that expected the old-style query * fix the "all other" test * improve commentary * tidying * revert irrelevant change * improve commentary * revert irrelevant change * revert irrelevant change * Fix paired aggregation behaviour * add test to ensure all filtered values are returned * improve safety * include parens in a test * fix sorting even more * improve commentary * better rule name * Karl and Jake * remove redundant test * start adding Image faceting * finish adding Image faceting tests * better naming, as per review * some fiddling to get round E4S Template limitations * add example to work towards * Apply auto-formatting rules * Inline templates work, now to put all the other bits in * still more messing about * don't query if no query term * Fix paired aggregation behaviour (#676) * Fix paired aggregation behaviour * improve safety * fix sorting even more * improve commentary * Karl and Jake * remove redundant test * It Works! * Apply auto-formatting rules * tidy whitespace * Apply auto-formatting rules * don't use E4s Indexes * Apply auto-formatting rules * tidying * Apply auto-formatting rules * tidy * Apply auto-formatting rules * more tidying * Apply auto-formatting rules * autoformat * Apply auto-formatting rules * remove redundant note * match ImageFilter search to main * Apply auto-formatting rules * minimise diff * Apply auto-formatting rules * switch off intellij autoformat and try again * Apply auto-formatting rules * bit more tidying * better template params * improve commentary * Update search/src/main/scala/weco/api/search/services/WorksRequestBuilder.scala Co-authored-by: Jamie Parkinson <[email protected]> * improve commentary --------- Co-authored-by: Buildkite on behalf of Wellcome Collection <[email protected]> Co-authored-by: Jamie Parkinson <[email protected]> * Bump scala-libs to v32.28.0 * add new required parameter (#684) * knn color query works but only if defined * Apply auto-formatting rules * Bump scala-libs to v32.29.0 * optionally calls knn when building images request * Bump scala-libs to v32.30.0 * Bump scala-libs to v32.31.0 * Bump scala-libs to v32.32.0 * Bump scala-libs to v32.33.0 * Bump scala-libs to v32.34.0 * Fix the paths in the storage imports * Bump scala-libs to v32.35.0 * Bump scala-libs to v32.36.0 * Bump scala-libs to v32.37.0 * updated test-documents with paletteEmbedding * updated test-documents with paletteEmbedding take2 * updated expected_responses to match updated test-documents * updated test-documents with paletteEmbedding take3 * removes request with blended and colour similarities * lil test fixes * Apply auto-formatting rules * removes stray comment/unused import * removed unused test_documents * use the right pipeline * Revert "removed unused test_documents" This reverts commit 5b29ac2. * fixes decluttering gone too far * Revert "Image query with vector" * fixes works whose id has changed * Remove a couple of unused terraform pieces * Remove a now-unused terraform module * Point at the new location of the remote account state * Apply auto-formatting rules * fixes rank test: use reference number that is in sync in both pipelines * Bump scala-libs to v32.38.0 * Bump scala-libs to v32.39.0 * Revert "Revert "Image query with vector"" This reverts commit 27760ca. * fixes the rounding error when generating queryVector * adds test for combined query and knn colour search * fix the 10k max results problem (#705) * fix the 10k max results problem * remove debug code * Switch to using the official Black Docker image for Python formatting * sets number of bins to 10 and adds blurring to queryVector * code style and test fixes * start converting image search to a template * Bump to the newest Elasticsearch index * revert irrelevant changes * remove commented code * make template configs protected * better whitespace normalisation * better whitespace normalisation * remove KNN boost * sort out SearchTemplateResponse for images * reinstate change merged from main * Roll back to 06-09 * Bump to 06-10 * copies test data from catalogue-pipeline * updates colour filter tests + suitable test-documents * uses latest pipeline * remove extraneous dots from filter terms * remove extraneous dots from filter terms * fixes trailing . in test-docs for aggs * Run new rank in branches * We don't want a shell entrypoint * How can we run a command * Relative path * What if we put the command here * Break up command * Role needs to be assumed for everything * Propagate environment * Run new rank in branches * We don't want a shell entrypoint * How can we run a command * Relative path * What if we put the command here * Break up command * Role needs to be assumed for everything * Propagate environment * Test images too * Paired aggregation clarity (#714) * limit self aggregation to filter value list size * Improve clarity in the weird aggregation stuff * unmock tests to cope with stricter FAAB * join knn filter terms with AND (#716) * Use new rank for scheduled tests * New rank for post-deploy tests * Remove old rank * Don't need volumes when not using local query * Prefiltering in template queries (#717) * start prepping for prefiltering * join knn filter terms with AND * Distinguish between pairable and unpairable filters * escape optional regex characters (#719) * Bump scala-libs to v32.40.0 * Construct the client using an API key * new pipeline (#724) * Use the pipeline API key in the snapshot reporter * Apply auto-formatting rules * Bump lambdas to Python 3.10 * Changes for Python 3.10 * Apply auto-formatting rules * new pipeline (#728) * Bump pipeline date * Update test documents * New queries * Update index config for tests * Deal with moved feature vector * Use filterableValues * Test fixes/additions * Fix filter field names * Somewhat more appropriate knn filter for new query * partOf no longer does title as well * Create appropriate test documents in-memory too * Apply auto-formatting rules * Use correct file for rank tests * Fix search tests * Snapshot test data needed updating * Can we choose the cluster dynamically? * Bad copypaste * Remove query _names for actual application queries * Make the query respect all terms * Update index config etc in tests * Reinstate some query names * Add a tie breaker to the image query * Case insensitive identifiers filter * AWS CLI ECR BBQ login commmand has changed * Don't need eval now * I love bash scripting * Bump pipeline date to 2024-01-03 * Bump pipeline date to 2024-01-09 * Add option for HTTP healthcheck, enable for works * Apply auto-formatting rules * bool tcp_healthcheck test is backwards, fix it * matcher is required for http healthcheck * Bump scala-libs to v32.40.1 * Bump the SBT version from 1.4.1 to 1.9.8 The 1.4.1 version of SBT has an issue with the version of the JNA library is uses which prevents it from loading on M1 macs. See: sbt/io#320 The most recent version (1.9.8) of sbt works with this project, so upgrading. - [ ] Build and test locally, does it succeed? - [ ] Does this PR get a green tick when running in CI? * Bump scala-libs to v32.40.2 * Add addDependencyTreePlugin plugin This change adds a plugin to generate and search dependency graphs with sbt. This is useful when we see errors caused by transitive dependencies, so we can work out which libraries we directly specify pull these in. See https://github.com/sbt/sbt-dependency-graph for the history of this plugin (new built into sbt after version 1.4. * Bump scala-libs to v32.40.3 * Bump scala-libs to v32.40.4 * Removes soft fails for e2es as not flaky no mo * Adds a simple healthcheck endpoint for the items service This change follows #736, and adds an HTTP healthcheck to the items API to ensure the scala service has started before it is registered healthy at the NLB and starts serving requests. * enable the http healthcheck for items api * Apply auto-formatting rules * Add default code owners * Add wider net of approvers for changes to codeowners and buildkite config * add requests api healthcheck endpoint * Add healthcheck controller & endpoint This change adds a healthcheck endpoint to provide to a loadbalancer for the purposes of checking application health at deployment and under normal service. * remove optional tcp healthcheck, no longer used * Update docs/requesting_flow.md Co-authored-by: Paul Butcher <[email protected]> * remove some bad tf around healthchecks [skip ci] * Add @wellcomecollection/developers to /terraform folder in CODEOWNERS ## What is this change? This change allows reviews from all developers to be sufficient for terraform changes. At present an approval to merge a PR for a terraform change must come from someone in the developers group, we probably want to widen that. ## How to test? Merge it and see if developers outside the scala-reviewers group can approve terraform changes. * Update requesting_flow.md Mermaid on Github doesn't like it when an arrow is unlabelled in a sequence diagram, even when it's just a response to the previous message in the opposite direction. I have added an "OK" to each of these and turned them all into dashed arrows to signify they are responses. * Add a diagram showing what the request flow looks like * Update docs/requesting_flow.md Co-authored-by: Paul Butcher <[email protected]> * Update requesting_flow.md Mermaid on Github doesn't like it when an arrow is unlabelled in a sequence diagram, even when it's just a response to the previous message in the opposite direction. I have added an "OK" to each of these and turned them all into dashed arrows to signify they are responses. --------- Co-authored-by: Paul Butcher <[email protected]> Co-authored-by: Buildkite on behalf of Wellcome Collection <[email protected]> Co-authored-by: Harrison Pim <[email protected]> Co-authored-by: harrisonpim <[email protected]> Co-authored-by: georgiaewhitney <[email protected]> Co-authored-by: Raphaelle Cantin <[email protected]> Co-authored-by: Raphaelle Cantin <[email protected]> Co-authored-by: Agnes Garoux <[email protected]> Co-authored-by: Jamie Parkinson <[email protected]> Co-authored-by: agnesgaroux <[email protected]> Co-authored-by: jamie <[email protected]> Co-authored-by: Robert Kenny <[email protected]> Co-authored-by: Robert Kenny <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #704