[SPARK-6141][MLlib] Upgrade Breeze from 0.10 to 0.11 to fix convergence bug

[dbtsai]

LBFGS and OWLQN in Breeze 0.10 has convergence check bug.
This is fixed in 0.11, see the description in Breeze project for detail:

scalanlp/breeze#373 (comment)

[SparkQA]

Test build #28238 has started for PR 4879 at commit 397a208.

• This patch merges cleanly.

[SparkQA]

Test build #28238 has finished for PR 4879 at commit 397a208.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28238/
Test FAILed.

[coderxiang]

Good job!
Do we have enough time to catch the release, especially if there are some incompatible APIs.
For my case, the coefficients differ but the performance (accuracy, auc) are similar. Might need to evaluate the trade-off here if the change is going to 1.3

[dbtsai]

@coderxiang Breeze seems to accidentally remove the public constructor of CSCMatrix, and we have a PR to Breeze to address it. Let's see if we can make it.

[srowen]

In the meantime, I do not think a 1.3 release should block on this, if 0.11 isn't quite suitable. The issue being fixed is minor-ish and I'm concerned other more subtle things might go wrong. It might be a little risky to rush this at the last minute.

[SparkQA]

Test build #28245 has started for PR 4879 at commit d848f65.

• This patch merges cleanly.

[dbtsai]

This is the fix in breeze side for missing public constructor of CSCMatrix scalanlp/breeze#375

[SparkQA]

Test build #28245 has finished for PR 4879 at commit d848f65.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28245/
Test FAILed.

[mengxr]

test this please

[SparkQA]

Test build #28255 has started for PR 4879 at commit d848f65.

• This patch merges cleanly.

[SparkQA]

Test build #28255 has finished for PR 4879 at commit d848f65.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28255/
Test PASSed.

[mengxr]

LGTM. Merged into master and branch-1.3. Thanks!

[nishkamravi2]

@mengxr @srowen This change introduces a 10% regression in K-Means (SPARK-6234). We seem to have the following options:

1. Absorb the perf regression
2. Find the problem in Breeze and fix it while retaining 0.11 in Spark
3. Revert back to 0.10 (potentially open a JIRA for Breeze and upgrade to 0.11 when fixed)

What do you guys think?

[mengxr]

@nishkamravi2 I don't think the k-means implementation uses breeze ops now, but I may be wrong. Which versions/hashtags did you test?

[nishkamravi2]

@mengxr The version that I run uses Breeze. Btw, a re-compile is required as well (which is a regression of sorts but probably ignorable).

[mengxr]

@nishkamravi2 Let's continue the discussion on the JIRA you created.