Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve verify-range testing #2761

Closed
2opremio opened this issue Jun 29, 2020 · 2 comments · Fixed by #3797
Closed

Improve verify-range testing #2761

2opremio opened this issue Jun 29, 2020 · 2 comments · Fixed by #3797
Assignees
Labels

Comments

@2opremio
Copy link
Contributor

While testing captive core with verify-range in AWS batch I encountered a few problems which made the experience unpleasant:

  1. Manually editing the Job Definitions/Queues/Compute environments in the AWS console was hairy and error-prone.
  2. Spawning jobs suffers from (1) aggravated by the need to calculate the ledger ranges (we have a spreadsheet for that, but still)
  3. When a job fails, there is no way to simply respawn jobs. You need to manually create a new job by copy-pasting the fields of the failed job (increasing the number of retries is often not enough).

To improve the situation, I propose the following:

  1. Use Terraform to keep track of the definiton AWS Batch resources and apply them automatically on merge. This will probably require having a separate repo for verify-range.
  2. Spawn test jobs semi-automatically through CI (specify the base branch, click a button and have CI spawn the jobs). For GitHub Actions we could use https://www.actionspanel.app/ (I don't know if there is an equivalent solution for CircleCI)
  3. CI could monitor jobs for failures and respawn them when the job failure is transient. We could also have a manual CI action which clones a job given its identifier.
@2opremio
Copy link
Contributor Author

Some other things which we could do:

  1. Automatically push the verify-range image when modifying it
  2. Automatically test the verify-range image in CI with a small range

@ire-and-curses
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants