NMF metrics and wikipedia

[anotherbugmaster]

Add clean up and fixes on top of #2361:

• Fix l2 of all models
• Fix perplexity of the sklearn NMF
• Change loss function in the logs of the model
• Print topics in the wikipedia notebook
• Reorder the benchmark tables
• Fix an out-of-bounds error in grouper
• Specify the shape of the corpus in the module docstring
• Fix the generator issue
• Add more unittests
• More explanations in wikipedia
• Fix flake8
• Add comment about sparsity method
• Mention pass parameter behavior for generators/iterators
• Fix smart_open import
• Fix tests

[piskvorky]

I can add a comment. If W error diff equals zero than it means that the model already reconstructs current batch as good as it gets and doesn't learn nothing new.

I understand that, but what does this mean for the user? Subsample? Increase precision somewhere?

Surely "nothing is being trained" is not a desirable training progress. It sounds like a waste, a warning sign, as opposed to a best practice.

[anotherbugmaster]

What is your explanation for sklearn being ~5x faster?

Obviously, it doesn't need to learn iteratively, and due to the small size of the dataset it became an advantage. Things work differently on the Wikipedia: large corpus is learned much faster with an iterative approach.

[anotherbugmaster]

I understand that, but what does this mean for the user? Subsample? Increase precision somewhere?

Surely "nothing is being trained" is not a desirable training progress.

Oh, I see. Yes, user can either subsample the trainset or increase the precision of the model (w_stop_iteration and h_stop_iteration parameters).

[piskvorky]

Things work differently on the Wikipedia: large corpus is learned much faster with an iterative approach.

Yes, but see our other parallel thread, about the model not really learning anything new in reality. Is it really faster, on a large corpus where it learns something? Wouldn't a subsample learn similar/better topics, in faster time (and 5x faster with sklearn)?

The tutorial nicely shows how to do the training, but the case for why to use Gensim's NMF is not yet convincing.

[anotherbugmaster]

Hm, I see. I think that the most obvious advantage over the Sklearn is the huge difference in RAM footprint, but I'll definitely need to check the subsampling case.

[mpenkov]

@piskvorky My understanding is that you'd like to quietly merge this branch without mentioning anything in the change log. Is that correct?

@anotherbugmaster Please let me know when this is ready to merge.

[piskvorky]

@mpenkov correct. Let's merge and release 3.7.2 ASAP (and finish the rest of NMF later).

without mentioning anything in the change log

We can mention these NMF fixes in the release log. It's not a channel many people actively follow, so still counts as "silent" :) I'll wait with real promo until the tutorial is more compelling.

[anotherbugmaster]

@mpenkov Sure!

Ok then, I'll check the possible caveats more thoroughly and report the results next week.

[piskvorky]

@anotherbugmaster any progress on the SVD (LSI) stats, for comparison?

Plus investigating the "NMF not learning anything" – effects of subsampling, precision on speed vs memory. Especially with regard to our recommended "best practices" and comparison to sklearn. Cheers!

[mpenkov]

@anotherbugmaster I'm going to merge this so we can go ahead with the next release.

Please continue working on this PR and pushing your changes. We'll merge again in the near future.

[anotherbugmaster]

@piskvorky, sorry, I forgot to make an experiment :(

I think I'll report something on this weekend.

[piskvorky]

@anotherbugmaster please start a new branch from the develop branch, to have a clean PR. A previously squashed merge is showing as hundreds of new commits in this PR; let's avoid that in the new PR. Cheers.

[piskvorky]

@anotherbugmaster what's the status here? I'd love to get NMF wrapped up nicely, so we can promote it.

[anotherbugmaster]

@piskvorky Sorry, Radim, still haven't finished the research yet. I'll do it ASAP.

[piskvorky]

@anotherbugmaster how is it going? Let's get NMF finished while there's still momentum (or remove it).

[anotherbugmaster]

@piskvorky, I've changed the algorithm so that the model updates every batch even if an error is already low. Good news is that it didn't affect performance, here are the results:

I've also changed an error in logs, now it looks like this:

The lower the error, the better.

[piskvorky]

@anotherbugmaster What's your rationale for these changes? What was the train-time + L2_norm before? (to easily compare to this latest 27:20 and 94.97)
Can you also give the reconstruction error for LSI on the same data please, for anchoring.

In short, the same questions as before:

Would you mind adding LsiModel (SVD) and report the L2 error there? Just for comparison. LSI is another matrix decomposition, directly comparable to NMF (except it doesn't have the sparsity constraints).

How do you interpret the mostly-zero-errors during the training? What conclusions should users draw from that? Subsample their input? Increase precision? What is going on?

I'm not sure how forcing near-zero updates answers them.

Thanks!

[piskvorky]

@anotherbugmaster ping -- can we finish this please?

[anotherbugmaster]

@piskvorky I'll run an LSI tomorrow.

[anotherbugmaster]

@piskvorky Ok, here's the Wikipedia comparison:

Seems like training an LSI takes the same amount of time as an LDA. L2 norm is a bit better, though coherence is not.

[anotherbugmaster]

I'm not sure how forcing near-zero updates answers them.

Model updates every batch now (it skipped a lot of them before, that wasn't correct behavior), I don't see any problem with update rate now.

[piskvorky]

Thanks @anotherbugmaster . Let me ping @mpenkov and @gojomo for a final review, especially of the new notebook -- are the motivation, evaluation and usage instructions convincing? Anything crucial missing, before we push this out "for real"? Thanks!

[piskvorky]

@anotherbugmaster can you open a new PR (starting from upstream develop), with your recent changes to the update process + the LSI comparison? Cheers.

[mpenkov]

@anotherbugmaster There are some logging-related bugs. I suspect they are blocking the release of our conda feedstock. I'll highlight the problem iines in a separate review.

[mpenkov]

some bugs in logging calls

[mpenkov]

lencorpus may be a float (inf) here, but the formatstring requires it to be an integer

[mpenkov]

The formatstr expects an integer, but you will end up passing a string if lencorpus is np.inf.

[piskvorky]

@anotherbugmaster can you open a new PR (starting from upstream develop), with your recent changes to the update process + the LSI comparison? Cheers.

@anotherbugmaster ping. Do you plan to finish NMF?