Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to cockroach DB #16

Open
5 tasks
Bjorn248 opened this issue Jan 8, 2022 · 0 comments
Open
5 tasks

Migrate to cockroach DB #16

Bjorn248 opened this issue Jan 8, 2022 · 0 comments
Assignees

Comments

@Bjorn248
Copy link
Owner

Bjorn248 commented Jan 8, 2022

First we should choose an execution environment, the candidates are:

  • Github Actions (leaning towards this, as I've had nothing but good experiences with it)
  • AWS Lambda (the existing infra, but I'm only using it because it's free...no real reason to stay)

The high level requirements of the ingestor are

  • dynamic schema generation from CSVs (existing python already does this, will need to be modified to work with cockroach schemas)
  • dynamic service ingestion (as new services are added by AWS the ingestor automatically picks them up and imports them, this already happens with the python scripts)
  • drop any table we are about to import
  • trigger an async cockroach import of a delimited data file, this has been tested in a local POC and works wonderfully
  • create an index on the EC2 table after it has been generated
@Bjorn248 Bjorn248 self-assigned this Jan 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant