Benchmark Runner results shown in viewer #390
Conversation
The goal is for this to be reused by the new benchmark runner. A benchmark results fields can specify how they are aggregated in which case their total/count/percent true/histogram will be displayed in the header. Or not in which case they will be viewable if you click through to the benchmark. TODO: actually use it in the benchmark runner. Have indentation/syntax errors be aggregated.
|
Here's a screenshot of what it looks like with on the new benchmarks: You can reproduce this with the command: It takes a couple of minutes to run, the following is faster: |
jakethekoenig
changed the title
|
It seems not all of the code is appearing in the code box? |
Oops, git diffs don't display untracked files. I think I'll run |
| </style> | ||
| <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/default.min.css"> | ||
| <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script> | ||
| <script>hljs.initHighlightingOnLoad();</script> |
There was a problem hiding this comment.
It looks like this is making booleans bold/green and strings and numbers red. I like having colors but I mistook this for indicating failure/success. Could it be different colors?
There was a problem hiding this comment.
Oh yeah, that does sort of seem bad in the json code blocks where true mostly means error detected. I'll check. If it's hard I'll just turn off syntax highlighting for json since it doesn't add much anyway
| if len(values) == 0: | ||
| summary[name] = (0, 0) | ||
| else: | ||
| percent_set = len(values) / len(self.results) |
There was a problem hiding this comment.
I wonder if n/N would be clearer than a percent? I had to guess what the two different percents meant my first time.
tests/benchmarks/benchmark_runner.py
Outdated
|
|
||
| diff = repo.git.diff(start_commit) | ||
| # Set syntax and response grade information | ||
| diff = repo.git.diff() |
There was a problem hiding this comment.
afaik this only saves edits, not new/untracked files. I ran into the same thing in #374 and added a helper func in git_handler. (I think this is why the code was missing in my earlier comment)
There was a problem hiding this comment.
Now i see you already caught it. I think stage/diff/unstage is the right approach too.
|
Very nice! I really like the top bar. Instead of showing a percentage in parenthesis when only a subset of benchmarks define an attribute, how about we show the actual counts? I.e. below it could show: 87.5% (8/14). That'd be useful even when all benchmarks do define the attribute, so you can instantly see how significant the percentage is (i.e. 100% 2/2 vs 100% 20/20) Edit: oh I see this was already suggested 😄 |
|
No code showing for this one either, but this seems different as this file shouldn't be untracked? 
|
I think this is an example of mentat making a mistake. You can see in the code block that the line it's replacing already says |
|
After this PR I'll edit the cron job to run the new benchmarks as well as all the javascript, python and clojure exercisms. When I do that I'll change the Benchmark summary to be saved as json, get a summary message from it and send it along with the link to slack. And also sync that json to its own bucket so it will be easy to parse old data and make graphs in the future. |
|
LGTM |
The BenchmarkResult class and it's viewer are made more general with the following changes:
Now if you introduce a new benchmark with a new property you can add the property to the BenchmarkResults class along with the appropriate metadata. This won't break previous benchmarks and if appropriate it is easy to set it for them.
misc:
todo: