Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lighthorizon: Enable parallel creation of indices via map-reduce #4403

Closed
Tracked by #4571
Shaptic opened this issue May 24, 2022 · 1 comment
Closed
Tracked by #4571

lighthorizon: Enable parallel creation of indices via map-reduce #4403

Shaptic opened this issue May 24, 2022 · 1 comment
Assignees
@Shaptic

Comments

@Shaptic
Copy link
Contributor

Shaptic commented May 24, 2022 •
edited
Loading

See parent issue for context: #4317

MapReduce (/index/cmd/batch): Indices are massive and take a long time to build, so we need a clean way to be able to build these using MapReduce, e.g. via AWS Batch jobs. The work has been started but has not been updated with the latest implementation from the single-process version. Ideally, these should use the same "core" code for index creation and just add abstractions for map/reduce parallelism vs. direct execution.
@Shaptic Shaptic mentioned this issue May 24, 2022
Closed
7 tasks
@Shaptic Shaptic changed the title MapReduce (<code class="notranslate">./index/cmd/batch`): Indices are massive and take a long time to build, so we need a clean way to be able to build these using MapReduce, e.g. via AWS Batch jobs. The work has been started but has not been updated with the latest implementation from the single-process version. Ideally, these should use the same "core" code for index creation and just add abstractions for map/reduce parallelism vs. direct execution. lighthorizon: Enable parallel creation of indices via map-reduce alongside AWS Batch May 24, 2022
@Shaptic Shaptic self-assigned this May 24, 2022
@Shaptic Shaptic changed the title lighthorizon: Enable parallel creation of indices via map-reduce alongside AWS Batch lighthorizon: Enable parallel creation of indices via map-reduce May 24, 2022
This was referenced May 25, 2022
Merged
Closed
@Shaptic
Copy link
Contributor Author

Shaptic commented Jun 10, 2022

Closed by #4423 and #4424 (with #4422 as a supporting PR).

@Shaptic Shaptic closed this as completed Jun 10, 2022
@jcx120 jcx120 mentioned this issue Sep 1, 2022
Closed
64 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Shaptic Shaptic

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Shaptic