Corpus cleanup: progress tracker

Corpus cleanup: progress tracker

This page allows tracking the progress of the corpus cleanup.

Metadata

• enp_de.bio
  • L000001 - L002000]
• enp_fr.bio
  • L000001 - L082493]

OCR errors

• enp_de.bio
  • L000001 - L002000]

Sentence splits

• enp_de.bio
  • L000001 - L002000]

Hyphenation

• enp_de.bio
  • L000001 - L002000]

Missing named entity tags

• enp_de.bio
  • L000001 - L002000]

Punctuation

• enp_fr.bio
• enp_de.bio
  • L000001 - L005000]

Other

• fix use of B-/I- in compliance with CoNLL convention
• replace B/I-LIEU with B/I-LOC and B/I-PERS with B/I-PER in French corpus
• assemble basic metadata (titles, issues, dates)
• harmonize files and directories
• remove unnecessary 'POS' tags (reduces file size considerably)
• remove metadata noise and empty files (0 bytes)
• consistently use spaces instead of tabs