Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More consistency checks + improvements #719

Merged
merged 28 commits into from
Dec 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
9aa5c18
Make journal_path required option
bkorycki Nov 25, 2024
de4b3ae
Check multiple journals with one command
bkorycki Nov 26, 2024
9e6b8be
nicer cli formatting
bkorycki Nov 26, 2024
350446a
summarize checks for multiple journals
bkorycki Nov 27, 2024
a13a5f2
checker obj does not store memory-intensive journal search engine
bkorycki Nov 27, 2024
86fa7b7
print invalid journals + add more tests
bkorycki Nov 27, 2024
0999181
Collect annotators from test directly + bug fix + annotator-check tests
bkorycki Nov 27, 2024
b06636c
add check: each prompt queued once
bkorycki Nov 27, 2024
c7550e1
Each annotation translated once
bkorycki Nov 27, 2024
b946c7b
items finished == num. measured items
bkorycki Nov 27, 2024
8418780
Check that annotations are merged correctly
bkorycki Nov 27, 2024
684c8e7
table formatting
bkorycki Nov 27, 2024
93bae78
add todo
bkorycki Nov 27, 2024
925ab78
fix tests
bkorycki Nov 27, 2024
5bc6ae5
check min. # invalid items for annotators
bkorycki Nov 27, 2024
c8d2990
linterrr
bkorycki Nov 27, 2024
8edb16b
ensure the CLI tool's output order is deterministic for the tests to …
rogthefrog Nov 27, 2024
cd753be
Make invalid annotator threshold a constant
bkorycki Nov 27, 2024
24cc8bf
cleaup
bkorycki Nov 27, 2024
cd385b7
lint
bkorycki Nov 27, 2024
1fdc849
change --journal-path to required arg
bkorycki Dec 3, 2024
57bebc7
Format column names to have spaces
bkorycki Dec 5, 2024
45b53ae
move to rich library for table formatting (handles wrapping
bkorycki Dec 5, 2024
0d071a9
sort rows alphabetically
bkorycki Dec 6, 2024
16f5cd7
Add detailed messages about why prompts failed AnnotationsMergedCorre…
bkorycki Dec 6, 2024
6e431bc
Add fraction of prompt errors in warning message for merge check
bkorycki Dec 6, 2024
b92b951
Only run certain checks for official benchmarks
bkorycki Dec 6, 2024
7dcd2b2
lint
bkorycki Dec 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading