Skip to content
This repository has been archived by the owner on Feb 22, 2023. It is now read-only.

Only fill in pages if at least one link from first page is not dead #855

Closed
1 task
sarayourfriend opened this issue Aug 5, 2022 · 2 comments · Fixed by #865
Closed
1 task

Only fill in pages if at least one link from first page is not dead #855

sarayourfriend opened this issue Aug 5, 2022 · 2 comments · Fixed by #865
Assignees
@sarayourfriend
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟥 priority: critical Must be addressed ASAP

Comments

@sarayourfriend
Copy link
Contributor

Problem

If we make a request to ES for 20 items and find that any of them are dead, we attempt to "fill the page in" with subsequent "deeper page" queries. If all the links for a query are dead, this means we will slam ES with tons of fast queries to try to fill in the page.

Description

Kudos @obulat for the idea. We should not try to fill pages if all of the links on the first page are dead.

Additional context

We are also working on excluding providers that account for a large number of dead links.

Implementation

  • 🙋 I would be interested in implementing this feature.
@sarayourfriend sarayourfriend added 🟥 priority: critical Must be addressed ASAP ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository labels Aug 5, 2022
@AetherUnbound
Copy link
Contributor

AetherUnbound commented Aug 5, 2022 •
edited
Loading

Some semi-relevant discussion here: cc-archive/cccatalog-api#565

@AetherUnbound AetherUnbound self-assigned this Aug 5, 2022
@AetherUnbound
Copy link
Contributor

Might attempt tackling with #857

@AetherUnbound AetherUnbound removed their assignment Aug 8, 2022
@sarayourfriend sarayourfriend self-assigned this Aug 8, 2022
@sarayourfriend sarayourfriend mentioned this issue Aug 9, 2022
Merged
7 tasks
@sarayourfriend sarayourfriend mentioned this issue Aug 15, 2022
Merged
6 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@sarayourfriend sarayourfriend

Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟥 priority: critical Must be addressed ASAP
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only backfill dead links if at least one on first page was not dead WordPress/openverse-api
2 participants
@AetherUnbound @sarayourfriend