Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opus_read fails to extract CCMatrix #32

Open
Waino opened this issue Nov 18, 2021 · 3 comments
Open

opus_read fails to extract CCMatrix #32

Waino opened this issue Nov 18, 2021 · 3 comments

Comments

@Waino
Copy link

Waino commented Nov 18, 2021

I tried to extract the aligned sentence pairs from CCMatrix, previously downloaded using opus_express. The command I used was

opus_read --source en --target fi --directory CCMatrix --preprocess xml --leave_non_alignments_out --write_mode moses --write CCMatrix.raw.en CCMatrix.raw.fi --write_ids CCMatrix.raw.ids

The command runs for several days at 100% CPU, without producing any output. Perhaps expat is choking on some error in the data. To rule out package corruption after download, I allowed opus_read to download it again, with the same hanging result.

Traceback when killed:
  File "/home/stiggronroos/venvs/opustools/bin/opus_read", line 135, in <module>
    OpusRead(**vars(args)).printPairs()
  File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/opus_read.py", line 214, in printPairs
    self.alignmentParser.collect_links()
  File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/parse/alignment_parser.py", line 107, in collect_links
    blocks = self.bp.get_complete_blocks()
  File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/parse/block_parser.py", line 98, in get_complete_blocks
    self.parse_line(line)
  File "/home/stiggronroos/venvs/opustools/lib/python3.6/site-packages/opustools/parse/block_parser.py", line 82, in parse_line
    self.p.Parse(line)
KeyboardInterrupt

Workaround: (re)download the corpus directly in moses format from https://opus.nlpl.eu/CCMatrix.php

@jorgtied
Copy link
Member

I guess it's because CCMatrix is so big that reading only the sentence IDs and links to be retrieved from the monolingual corpora takes too much memory to run efficiently. We don't have a good solution for this at the moment but should add some robustness to the tools to also run on bigger data sets. The workaround with moses files is the only solution I can recommend at this moment ....

@shaoyangxu
Copy link

Why can't the "preprocess" parameter be set to "Moses" directly ? I mean, "xml"、"raw" and "parsed" are all time-conmusing relatively.

@miau1
Copy link
Member

miau1 commented Nov 29, 2022

There is now --chunk_size parameter to control memory consumption, although the current implementation is still slow for corpora with huge documents.

Regarding moses, it also possible to download moses files with the opus_get script. For example:
to list available files:
opus_get -s en -t fi -d CCMatrix -p moses -l
to download the files:
opus_get -s en -t fi -d CCMatrix -p moses

I'm still leaving this issue open until we find a better solution for processing huge documents.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants