Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store failed and non-build jobs in the analytics database #942

Merged
merged 14 commits into from
Aug 21, 2024

Conversation

jjnesbitt
Copy link
Collaborator

This PR updates our process_job function and all child data gathering functions to operate on failed jobs, as well as other job types like generate, etc. This implementation has the following effects:

  1. The "type" of a job is now stored in the database (build, generate, etc.)
  2. I've created some util functions for retrieving gitlab data, due to our current cluster timeout issues.
  3. The gitlab section timers are now stored in the JobDataDimension
  4. I've created a script to migrate existing LegacyJobAttempt rows to the current format. This must be run manually, after this is deployed, as it's lengthy and not suitable to be run automatically in a Django migration.

I've tested both the regular process_job workflow on various job types with real webhook payloads taken from Gitlab, as well as the job migration management command. Both seem to work well.

This should remove the last vestiges of the Legacy models that are used as a part of the job processor. After this is merged, we can check that none of the legacy data is unaccounted for or otherwise still being relied upon, and make a follow on PR to finally remove them.

@jjnesbitt jjnesbitt force-pushed the store-failed-jobs branch 3 times, most recently from 04f45c4 to 7a194e6 Compare August 8, 2024 22:24
@jjnesbitt jjnesbitt marked this pull request as ready for review August 8, 2024 22:28
@jjnesbitt jjnesbitt merged commit 22c7c9a into main Aug 21, 2024
15 checks passed
@jjnesbitt jjnesbitt deleted the store-failed-jobs branch August 21, 2024 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants