feat: Adds German compound words decomposition with new segmenter

[luflow]

Pull Request

What does this PR do?

• Adds first version of decomposition for german compound words based on a dictionary (based on https://github.com/uschindler/german-decompounder/)
• Adds benchmark with german sentences

PR checklist

Please check if your PR fulfills the following requirements:

• Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
• Have you read the contributing guidelines?
• Have you made sure that the title is accurate and descriptive of the changes?

[luflow]

I assume this could be a very expensive algorithm because all word lengths are checked against the dict?

Not sure if there is a better solution, but at least a first version for compound words :)

[luflow]

Also another open question: can we even use the dictionary?

The orignal author has it under GNU GPL
https://github.com/uschindler/german-decompounder/blob/master/NOTICE.txt

[luflow]

@curquiza @ManyTheFish fixed the fmt and clippy issues, Please rerun

[ManyTheFish]

Hello @luflow,

Could you add a feature flag on your implementation as I suggested, please? Then add it as a default feature in the Cargo.toml file.

In terms of implementation, you chose to rely on an HashSet to split your words, but I don't think it's the best approach.
I highly suggest using an FstSegmenter like in the Thai tokenizer, it's a bit complex to build but way more efficient in time and space, or you could use an AhoCorasick automaton using the LeftmostLongest match kind.

Sorry for the delays!
Let me know if you have a question

[luflow]

Hi @ManyTheFish!

Do you have any instructions to build the fst file? I could not find any material online - especially because FST is also used in other contexts like R but does something totally different 🤣

Otherwise the leftmostmatch functionality also works with a word dictionary if i understand it correctly?

[ManyTheFish]

Do you have any instructions to build the fst file? I could not find any material online - especially because FST is also used in other contexts like R but does something totally different 🤣

You can use the CLI fst-bin to build your dictionary from a source file. 😄

Otherwise the leftmostmatch functionality also works with a word dictionary if i understand it correctly?

Yes you can build it from an iterator over str, so it's convenient

[luflow]

@ManyTheFish I extended the FstSegmenter with two options to also be able to handle a min lemma length and being able to hinder the segmenter from spitting out single letters. That keeps my dictionary even smaller and may be also useful for other languages later?

The dictionary is now also transformed into an FST file.

Let me know what you think :)

[luflow]

@ManyTheFish dud you find time yet to look over the changes? Do you need anything else from my side? :)

[ManyTheFish]

Hello @luflow,
sorry for the delay, LGTM!

bors merge

[meili-bors]

Build failed:

• tests

[luflow]

@ManyTheFish ok applied suggestion :)

[ManyTheFish]

Hello @luflow,

the test and clippy are not happy,

could you ensure that:

• cargo clippy
• cargo test

work on your machine please?

I'll merge as soon as the tests pass 😃

[luflow]

@ManyTheFish done 👍🏻

[ManyTheFish]

Nice!

Thank you for the contribution!

bors merge

[meili-bors]

Build succeeded:

• Run Clippy
• Run Rustfmt
• tests