Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adjust transform job to run hourly #2528

Closed
1 task
chouinar opened this issue Oct 21, 2024 · 0 comments · Fixed by #2641
Closed
1 task

Adjust transform job to run hourly #2528

chouinar opened this issue Oct 21, 2024 · 0 comments · Fixed by #2641
Assignees
@chouinar

Comments

@chouinar
Copy link
Collaborator

Summary

We want to adjust the rate at which the transform job runs from daily to hourly: https://github.com/HHS/simpler-grants-gov/blob/main/infra/api/app-config/env-config/scheduled_jobs.tf#L50

I'd recommend instead of specifying a rate, we instead specify a cron (the rate approach just arbitrarily chooses when the day/hour begins based on when you enable it). https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-scheduled-rule-pattern.html

Acceptance criteria

  • Config updated
@chouinar chouinar self-assigned this Oct 29, 2024
@chouinar chouinar moved this from Icebox to Todo in Simpler.Grants.gov Product Backlog Oct 29, 2024
@chouinar chouinar moved this from Todo to In Progress in Simpler.Grants.gov Product Backlog Oct 29, 2024
chouinar added a commit that referenced this issue Oct 29, 2024
@chouinar
[Issue #2528] Change job schedule ELT process to hourly
8c4c9ce
@chouinar chouinar mentioned this issue Oct 29, 2024
Merged
chouinar added a commit that referenced this issue Oct 30, 2024
@chouinar
[Issue #2528] Change job schedule ELT process to hourly (#2641)
17321cf 
## Summary
Fixes #2528

### Time to review: __3 mins__

## Changes proposed
Modified ELT process to run hourly

Removed the load job for the v0 data

## Context for reviewers
I can't imagine there is much value in the v0 job now that v1 works.
Probably will save us a bit as we won't be running a process near
constantly that no one uses.

I changed the schedule expression to a cron so that the schedule is a
bit more consistent. The `rate(..)` configuration seems to be based on
when you turn it on, or last did a deploy. This will keep it from
varying every time we deploy and instead make it so it's always at the
top of the hour.

## Additional information

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-scheduled-rule-pattern.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chouinar chouinar

Labels
None yet
Projects
Simpler.Grants.gov Product Backlog
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Issue #2528] Change job schedule ELT process to hourly HHS/simpler-grants-gov
1 participant
@chouinar