Code

KenLM based content detection#72

Code

Merged

Uinelj merged 19 commits intodevoscar-project/ungoliant:devfrom dev-kenlmoscar-project/ungoliant:dev-kenlmCopy head branch name to clipboardDec 13, 2022

+672-84

Commits

Commits on Nov 25, 2022

• WIP: LRU cache for multiple KenLMs

  Uinelj

  committed
  86d58c5

Commits on Nov 28, 2022

• make annotator immutable

  Uinelj

  committed
  123ae34

Commits on Nov 29, 2022

• simplify models struct, add LRU later on

  Uinelj

  committed
  a84bc53

Commits on Nov 30, 2022

• thread safe model hashmap

  Show description for da787c6
  Uinelj

  committed
  da787c6

Commits on Dec 1, 2022

• doc: add comments, link to ctclib fork

  Uinelj

  committed
  2641f71

• ci: add cpp dependencies for ctclib/KenLM

  Uinelj

  committed
  670cc1d

• ci: add sudo on ctclib cpp deps install

  Uinelj

  committed
  82e10b1

• ci: change blocklist fetching from ftp to git

  Uinelj

  committed
  6bb7bb0

• fix: add simple errors on no model found

  Uinelj

  committed
  5326c33

Commits on Dec 2, 2022

• test: test loading of correct kenlm model

  Uinelj

  committed
  eeb1520

• begin integration of kenlm in main pipeline

  Uinelj

  committed
  e2f7a04

Commits on Dec 5, 2022

• feat(kenlm): remove error prone defaults, remove commented code, adapt to ctclib update

  Uinelj

  committed
  d68f740

Commits on Dec 6, 2022

• ci: add full stacktrace

  Uinelj

  committed
  68efc77

Commits on Dec 8, 2022

• add fallback on arpa models if binary not found

  Uinelj

  committed
  385059d

• ci: fix ci script and code to run properly the rebuilding integration test

  Uinelj

  committed
  b0be72a

• feat: make kenlm an optional dep, add readme instructions

  Uinelj

  committed
  f1c52ba

Commits on Dec 12, 2022

• make kenlm an optional feature

  Uinelj

  committed
  17cceda

• feat(kenlm): builders list now built at pipeline start.

  Show description for a4364c0
  Uinelj

  committed
  a4364c0

• fix(logs): change log level of adult doc id from info to debug

  Uinelj

  committed
  91a0d00