Materialize and copy the corpus passed to SoftCosineSimilarity

[Witiko]

SoftCosineSimilarity currently does not perform any indexing and keeps a reference to the corpus passed to the constructor. This can cause confusion if a corpus is lazy and requests for documents are only resolved when reading the corpus. In general, subclasses of the SimilarityABC interface are expected to materialize and copy (i.e. index) the corpus rather than just reference it.

Some classes, such as WmdSimilarity, do not conform to this expectation. However, these classes also don't work with traditional BoW corpora and should be considered novelties/outliers. SoftCosineSimilarity, on the other hand, is very similar to the core SparseMatrixSimilarity class, and should satisfy the same invariants even if they are generally unspoken.

This pull request makes SoftCosineSimilarity materialize and copy the corpus passed to the constructor using the list() built-in. This is a minimal change with no change of interface and little chance of regressions. A more thorough solution would be to index the corpus to a sparse CSC matrix. However, this would likely break the interface, which may be undesirable so soon after 4.0.0.

[mpenkov]

Looks good to me. Thank you @Witiko !

[sug4ndh]

Hi,

I have an update to a problem that I reported here https://groups.google.com/g/gensim/c/3ksqzgCTWxI. Not sure if this affects this PR merge. I noticed that the problem reported in the google group thread does not occur if I generate corpus and dictionary afresh before I train word2vec or fasttext models and then subsequently generate SoftCosineSimilarity index. However, if used a saved corpus and dictionary to train fasttext or word2vec model and then generate SoftCosineSimilarity index, I always get the problem that "No such file or directory: 'corpus.mm' if corpus.mm is not present in dir from where I am executing the program. If I copy corpus.mm to the dir from where I am executing the program then no error appears.

[Witiko]

@sug4ndh Hello, this PR should solve your problem. As a temporary workaround with your release of Gensim, you can create the SoftCosineSimilarity index with list(corpus) instead of corpus. Then, you should be able to also save and subsequently reload your SoftCosineSimilarity index. Hope this helps!