-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmark runner test #488
Conversation
While writing this I made the following changes: * benchmark_repos was made a subdir of benchmarks so verify functions could import from generated code. Otherwise python complains it isn't part of a module. * In fact setup_repo does change the current directory. evalute_py and evaluate_sample now change the directory back. * The whole functions have a try now so spurious errors won't crash a run.
@@ -9,31 +9,33 @@ def test_empty_sequence(self): | |||
|
|||
def test_pow(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't let the actual accumulate get formatted because it could make the saved gpt output invalid but I'm inclined to let our linter change the test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It runs fine, I've just opened a big refactor to integrate the sample more so I'm fine merging this as-is.
|
||
[tool.ruff] | ||
line-length = 120 | ||
ignore = ["E731"] | ||
|
||
[tool.pytest.ini_options] | ||
addopts = "--ignore=vscode/bundled --ignore=benchmark_repos --ignore=testbed/exercism-python" | ||
addopts = "--ignore=vscode/bundled --ignore=benchmarks/benchmark_repos --ignore=testbed/exercism-python" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should also update pyrightconfig.json, change benchmark_repos
to benchmarks
.
While writing this I made the following changes:
It's a little less comprehensive than the exercism analogue but I think it'll still be valuable.
Pull Request Checklist