Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarking workflow to evaluate classification and OD runtimes #638

Merged
merged 30 commits into from
Jul 3, 2024

Conversation

ntlind
Copy link
Contributor

@ntlind ntlind commented Jun 27, 2024

Use Cases

  • We want to look at recent commits to figure out which one caused a change in performance
  • We want to quickly test performance of potential changes locally using standardized benchmarks

Changes

  • Delete stress-testing.yml and associated files since we never trigger these manual jobs
  • Create an automatic benchmarking job to track Valor performance for classification and object detection evaluations. The workflow will fail if any of the individual evaluation tests takes longer than 30 seconds.

Output

image image

@ntlind ntlind self-assigned this Jun 27, 2024
@ntlind ntlind added the enhancement New feature or request label Jun 27, 2024
@ntlind ntlind marked this pull request as ready for review July 1, 2024 07:09
@ntlind ntlind requested review from czaloom and ekorman as code owners July 1, 2024 07:09
@ntlind ntlind changed the title Add benchmarking workflow to provide performance signals for each commit Add benchmarking workflow to evaluate classification and OD runtimes Jul 1, 2024
czaloom
czaloom previously approved these changes Jul 1, 2024
@czaloom czaloom dismissed their stale review July 1, 2024 22:20

stale - need to review with Eric.

@ekorman
Copy link
Contributor

ekorman commented Jul 3, 2024

we can host the data in a public s3. prefer not having in version control

@ekorman
Copy link
Contributor

ekorman commented Jul 3, 2024

we can host the data in a public s3. prefer not having in version control
files:

https://pub-fae71003f78140bdaedf32a7c8d331d2.r2.dev/classification_data.json
https://pub-fae71003f78140bdaedf32a7c8d331d2.r2.dev/detection_data.json

Copy link
Contributor

@ekorman ekorman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make the benchmark-evaluations job fail if it takes too long?

@ntlind ntlind merged commit 0694e28 into main Jul 3, 2024
12 checks passed
@ntlind ntlind deleted the add_benchmark_utilities branch July 3, 2024 20:20
@Striveworks Striveworks deleted a comment from czaloom Jul 11, 2024
@Striveworks Striveworks deleted a comment from czaloom Jul 11, 2024
@Striveworks Striveworks deleted a comment from czaloom Jul 11, 2024
@Striveworks Striveworks deleted a comment from czaloom Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants