Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add people and parliament #137

Merged
merged 55 commits into from
Nov 18, 2024
Merged

Add people and parliament #137

merged 55 commits into from
Nov 18, 2024

Conversation

carschno
Copy link
Contributor

@carschno carschno commented Nov 13, 2024 •
edited
Loading

  • Add PeopleAndParliament corpus configuration
  • Optionally split large corpora, default for ingestion.
  • Set alias for id metadata field

Accidentally, this PR also includes #135.

@carschno
Add ner_script.py
3846fcd
@carschno
Add map_places.py
a46a5c6
@carschno
Cache Spacy model.
037ea46
@carschno
Output HeatMapWithTime.
4bff264
@carschno
Handle gecodong with caching in class.
682237e
@carschno
Move scripts to sub-directories.
a79e27b
@carschno
Add docstrings, fix minor things.
080fec3
@carschno
Use date as index for HeatMapWithTime.
3cf2e11
@carschno
Use rolling window for heatmap.
8bbcf6e
@carschno
Filter out place names with less than 3 letters.
0a6e655
@carschno
Skip caching in case of exception.
74adba4
@carschno
Rename heat map layer, hide pins by default.
409895a
@carschno
Add title argument, add locations only once.
6f0e15d
@carschno
Reduce title width to 50%.
5c7a4f6
@carschno
Add all spellings and dates to each location pin.
3fe155f
@carschno
Extract read_data_list()
a67d26f
@carschno
Extract create_markers()
963ae02
@carschno
Extract create_smoothed_heat_data()
62241ba
@carschno
Refactor create_markers() to add_markers().
fd68d9f
@carschno
Rename to extract_places.py
65eb1a1
@carschno
Remove default limit.
af792f8
@carschno
Fix: skip redundant opening of output file.
8dd088a
@carschno
Log name of the file that is being processed.
231848f
@carschno
Fix command line argument name.
85598fa
@carschno
Catch more generic GeocoderServiceError.
8643e1c
@carschno
Handle csv parsing errors.
3b5c74e
@carschno
Update scikit-learn dependency to ~1.5.2
f6357e8
@carschno
Add --resume flag.
de47d88
@carschno
Add TestGeocoder, rename internal Geocoder methods.
897856a
@carschno
Move ner dependencies to install_requires, download Spacy models wher…
511cf36 
…e needed.
@carschno
Remove explicit matplotlib dependency, sort dependencies.
c27dccb
@carschno
Update ChromaDB dependency for NumPy 2.0 compatibility.
4cc78e2
@carschno
Refine load_spacy_model.
662bfe1
@carschno
Add year range option.
e20acb6
@carschno
Add docstrings, type hints, TODOs.
23d1591
@carschno
Exclude end year.
49adf6d
@carschno
Use nl_core_news_lg for NL.
f40b52b
@carschno
Add PeopleAndParliament corpus.
6d0ff1e
@carschno
Add max_corpus_size argument, make Segmenter and CorpusReader yield P…
66e9acc 
…assage/Corpus sequences.
@carschno
Set alias for id field.
6d2fbbf
@carschno
Pin SciPy dependency.
966150a
@carschno
Revert "Pin SciPy dependency."
496eec9 
This reverts commit 966150a.
@carschno
Use date field, datetime format for year filtering by default.
92aaa7d
@carschno
Add doc_frequencies_per_year().
4540eaf
@carschno
Derive fields from metadata by default.
2ec9644
@carschno
Filter derived metadata fields.
3cd418e
@carschno
Use date instead of year, add timezone to dates.
1ee7f72
@carschno
Adjust term_frequencies notebook.
bafaa5b
@carschno
Do not include pins into map by default.
8433657
@carschno
Fix: replace global logger with __name__ (invalid in Python 3.9).
e017ebf
@carschno
Add Passage.Metadata.model_field_names() to get model field names, wo…
30f43b0 
…rking around Python 3.9 restrictions.
@carschno
Fix logger name.
f472ad4
@carschno
Extract YearSpan._to_datetimes()
1eb7618
@carschno
Fix left-over comments/returns.
f354854
@carschno
Update term_frequencies notebook.
e538563
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@carschno carschno merged commit 7df0ba7 into main Nov 18, 2024
16 checks passed
@carschno carschno deleted the add_people_and_parliament branch November 18, 2024 16:14
@carschno carschno mentioned this pull request Nov 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@carschno