Skip to content

Commit

Permalink
[Issue #2528] Change job schedule ELT process to hourly (#2641)
Browse files Browse the repository at this point in the history
## Summary
Fixes #2528

### Time to review: __3 mins__

## Changes proposed
Modified ELT process to run hourly

Removed the load job for the v0 data

## Context for reviewers
I can't imagine there is much value in the v0 job now that v1 works.
Probably will save us a bit as we won't be running a process near
constantly that no one uses.

I changed the schedule expression to a cron so that the schedule is a
bit more consistent. The `rate(..)` configuration seems to be based on
when you turn it on, or last did a deploy. This will keep it from
varying every time we deploy and instead make it so it's always at the
top of the hour.

## Additional information

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-scheduled-rule-pattern.html
  • Loading branch information
chouinar authored Oct 30, 2024
1 parent 55173de commit 17321cf
Showing 1 changed file with 3 additions and 7 deletions.
10 changes: 3 additions & 7 deletions infra/api/app-config/env-config/scheduled_jobs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,10 @@ locals {
],
}
scheduled_jobs = {
copy-oracle-data = {
task_command = ["poetry", "run", "flask", "data-migration", "copy-oracle-data"]
schedule_expression = "rate(2 minutes)"
state = "ENABLED"
}
load-transform = {
task_command = local.load-transform-args[var.environment]
schedule_expression = "rate(1 days)"
task_command = local.load-transform-args[var.environment]
# Every hour at the top of the hour
schedule_expression = "cron(0 * * * ? *)"
state = "ENABLED"
}
}
Expand Down

0 comments on commit 17321cf

Please sign in to comment.