feat(ingest): Use pycountry and fuzzywuzzy to format countries #3026
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
More clean up Move checks from snakefile to config fix config update deployment update tests ci Add trigger from db option Fix cronjob Fix link to config-file fix deployment install package in dockerfile install at correct location Remove snakemake as no longer needed Add missing dependency try to debug Create an XmlNone dataclass - this is required since package update test threads stop revert exception test test upload to ena dev still works on preview Make sure test is set correctly!!! remove debug print statements Improve logs Fix merge errors Update ena-submission/README.md Co-authored-by: Cornelius Roemer <[email protected]> Apply suggestions from code review Co-authored-by: Cornelius Roemer <[email protected]> Cronjob: create results directory before writing to it format authors in prepro Fix ingest try to fix pattern simplify regex fix check Add tests # Conflicts: # preprocessing/nextclade/tests/test.py Add to ena submission fix fix other edge case Update ena-submission/scripts/ena_submission_helper.py Co-authored-by: Cornelius Roemer <[email protected]> Update ena-submission/scripts/ena_submission_helper.py Update ena-submission/scripts/ena_submission_helper.py Co-authored-by: Cornelius Roemer <[email protected]> Update ena-submission/scripts/ena_submission_helper.py Update ingest/scripts/prepare_metadata.py Update preprocessing/nextclade/src/loculus_preprocessing/processing_functions.py rename Update reformat_authors_from_genbank_to_loculus Additionally format authors with correct white space Improve error message add tests fix missing pattern improve error logs fix error Update preprocessing/nextclade/src/loculus_preprocessing/processing_functions.py improve logging more feat(ingest): Do not use processed tsv but raw jsonl when ingesting data from NCBI Virus (#2990) * Use raw jsonl instead of generated tsv when ingesting data from NCBI virus * Do not require authors list to end in ';', capitalize names correctly. * Add tests for capitalization * Add a warning if author list might be in wrong format * Add ascii specific warning * Add tests for warnings and errors * Only capitalize if full authors string is upper case * Properly capitalize initial * Move titlecase option to ingest only - add ingest tests Move author formatting functions to format_ncbi_metadata as this is a more logical location Remove duplicate group name # Conflicts: # ena-submission/scripts/get_ena_submission_list.py # ena-submission/src/ena_deposition/config.py
…LocAdmin2 as well)
anna-parker
changed the title
anna-parker
force-pushed
the
format_authors
branch
2 times, most recently
from
c3cb79c
to
6e05226
Compare
anna-parker
force-pushed
the
format_authors
branch
from
8fc0a47
to
0f60fd4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #
preview URL: https://use-pycountry.loculus.org/
Discussion: https://docs.google.com/document/d/1TQiE66Hk6WjgkMvvMMhu8uMSEhHFP2AwINMAIsyODCw/edit
Summary
Pycountry is a popular python library with lists of all ISO-3166 accepted countries, subdivisions and historic countries. It includes their official names, country codes and subdivision level. Fuzzywuzzy is another popular python library which matches strings using the Levenshtein distance (an edit distance algorithm), it can be modified to weight string matches higher when individual words in the string match.
Takes INSDC geolocation string in format
country: divisionReturns country and attempts to split division into geoLocAdmin1 and geoLocAdmin2.
Tests
Can use comparison script in https://github.com/loculus-project/loculus/blob/show_geoloc_diffs/ingest/scripts/comparison.py to compare against main and augur curate
Here you can see sorted tsv of what changes the two options will make:
PR Checklist
version_commentfield with previous geolocation information if changed due to fuzzy matching