Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new CDK stack for Bulk Export #3500

Open
ab295382 opened this issue Oct 16, 2024 · 0 comments
Open

Create new CDK stack for Bulk Export #3500

ab295382 opened this issue Oct 16, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@ab295382
Copy link
Member

ab295382 commented Oct 16, 2024

Background

Create a new CDK stack for Bulk Export

Parent issue

Description

A new optional stack called the BulkExportStack. This will need to contain a queue for the export request. This request will be picked up by a lambda, which will act similarly to the query planner, i.e. break the request up into sub-export requests, one for each leaf partition. We will then need an ECS cluster to run tasks to process these sub requests. The scaling up of tasks can happen in the same way we scale up tasks in other situations, e.g. compactions.

@ab295382 ab295382 added the enhancement New feature or request label Oct 16, 2024
@ab295382 ab295382 self-assigned this Oct 16, 2024
@patchwork01 patchwork01 added this to the 0.26.0 milestone Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants