Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Elasticsearch management functions #4399

Closed
4 tasks
lbeaufort opened this issue Jun 3, 2020 · 2 comments · Fixed by #5367
Closed
4 tasks

Refactor Elasticsearch management functions #4399

lbeaufort opened this issue Jun 3, 2020 · 2 comments · Fixed by #5367

Comments

@lbeaufort
Copy link
Member

lbeaufort commented Jun 3, 2020

Summary
In the past, when citations or any entity were missing for a specific AO's or a Case (MUR, ADR, AF) on the website, we run the zero downtime reload command which upload all the legal documents onto elasticsearch service. The Zero downtime task takes ~1.5 hours without causing any downtime on the website and uploads all legal documents. With the regular initialize load commands (for each document type) we anticipate downtime.

Things to consider:

  • Create a new index (for ex: docs_ao) for AO's only???
  • Refactor the current code and upload AO documents to the new index on elasticsearch service???
  • Refactor or write a new zero-downtime task to reload AO???

Completion criteria:

  • Modify the zero-downtime task to optionally reload a specific document type
@lbeaufort lbeaufort added this to the Sprint 13.11 milestone Jan 4, 2021
@pkfec pkfec modified the milestones: Sprint 13.11, PI 13 innovation 2 and PI 14 planning Jan 7, 2021
@pkfec
Copy link
Contributor

pkfec commented Jan 7, 2021

Two tasks that we run to upload documents onto elasticsearch service after which the legal documents are available on the website for legal search:

  1. Zero downtime - Causes no downtime. Loads all the docs and take ~1.5 hours.
  2. Initialize docs - Its real time process. Causes downtime until each document is uploaded on the elasticsearch service.

It appears that we have all of our legal documents uploaded onto one elasticsearch index. After discussing with @fec-jli @lbeaufort we want to create a separate index for each document type and write a zero downtime task for each one of the document type so that we can speed up the upload documents without causing any downtime on the website.

@pkfec
Copy link
Contributor

pkfec commented Feb 8, 2021

Un assigning myself as I am not actively working on this issue.

@patphongs patphongs modified the milestones: Sprint 18.1, Sprint 18.6 Apr 28, 2022
@patphongs patphongs modified the milestones: Sprint 18.6, Sprint 19.1 Jul 7, 2022
@patphongs patphongs removed this from the Sprint 19.1 milestone Aug 10, 2022
@fec-jli fec-jli self-assigned this Feb 14, 2023
@fec-jli fec-jli added this to the Sprint 20.6 milestone Feb 14, 2023
@fec-jli fec-jli changed the title Zero-downtime legal reload takes too long Refactor Elasticsearch management functions Feb 23, 2023
@cnlucas cnlucas modified the milestones: Sprint 20.6, PI 20 innovation Mar 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants