Skip to content

Commit

Permalink
More consistency checks + improvements (#719)
Browse files Browse the repository at this point in the history
* Make journal_path required option

* Check multiple journals with one command

* nicer cli formatting

* summarize checks for multiple journals

* checker obj does not store memory-intensive journal search engine

* print invalid journals + add more tests

* Collect annotators from test directly + bug fix + annotator-check tests

* add check: each prompt queued once

* Each annotation translated once

* items finished == num. measured items

* Check that annotations are merged correctly

* table formatting

* add todo

* fix tests

* check min. # invalid items for annotators

* linterrr

* ensure the CLI tool's output order is deterministic for the tests to pass (#720)

* Make invalid annotator threshold a constant

* cleaup

* lint

* change --journal-path to required arg

* Format column names to have spaces

* move to rich library for table formatting (handles wrapping

* sort rows alphabetically

* Add detailed messages about why prompts failed AnnotationsMergedCorrectly check

* Add fraction of prompt errors in warning message for merge check

* Only run certain checks for official benchmarks

* lint

---------

Co-authored-by: Roger <[email protected]>
  • Loading branch information
bkorycki and rogthefrog authored Dec 9, 2024
1 parent d190109 commit a893b5f
Show file tree
Hide file tree
Showing 4 changed files with 636 additions and 102 deletions.
Loading

0 comments on commit a893b5f

Please sign in to comment.