Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
More consistency checks + improvements (#719)
* Make journal_path required option * Check multiple journals with one command * nicer cli formatting * summarize checks for multiple journals * checker obj does not store memory-intensive journal search engine * print invalid journals + add more tests * Collect annotators from test directly + bug fix + annotator-check tests * add check: each prompt queued once * Each annotation translated once * items finished == num. measured items * Check that annotations are merged correctly * table formatting * add todo * fix tests * check min. # invalid items for annotators * linterrr * ensure the CLI tool's output order is deterministic for the tests to pass (#720) * Make invalid annotator threshold a constant * cleaup * lint * change --journal-path to required arg * Format column names to have spaces * move to rich library for table formatting (handles wrapping * sort rows alphabetically * Add detailed messages about why prompts failed AnnotationsMergedCorrectly check * Add fraction of prompt errors in warning message for merge check * Only run certain checks for official benchmarks * lint --------- Co-authored-by: Roger <[email protected]>
- Loading branch information