[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator

[fjiang6]

Huber Robust Regression including the test case and an example.
Passed the style checks

[mengxr]

Jenkins, test this please.

[mengxr]

Jenkins, add to whitelist.

[mengxr]

@fjiang6 Could you try LBFGS instead of SGD?

[SparkQA]

QA tests have started for PR 2096 at commit ab0f539.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2096 at commit ab0f539.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Params(
  • class HuberRobustGradient extends Gradient
  • class HuberRobustRegressionModel (

[fjiang6]

The failed test is "org.apache.spark.graphx.lib.TriangleCountSuite.Count two triangles", which is a part I never touched. What could be the possible reason for this failure?

[ankurdave]

Jenkins, retest this please.

This was a problematic commit in GraphX which I just reverted, so a retest should fix the problem.

[fjiang6]

Can you please retest this? Thanks!

[fjiang6]

Can you please retest this? Thanks!

发自我的 iPhone

在 Sep 3, 2014，11:22 PM，Apache Spark QA [email protected] 写道：

QA tests have finished for PR 2096 at commit ab0f539.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Params(
class HuberRobustGradient extends Gradient
class HuberRobustRegressionModel (
—
Reply to this email directly or view it on GitHub.

[fjiang6]

ERROR: Timeout after 10 minutes
FATAL: Failed to fetch from https://github.com/apache/spark.git

Can you please retest?

[fjiang6]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 2096 at commit 9e34bd2.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2096 at commit 9e34bd2.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2096 at commit 9e34bd2.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2096 at commit 9e34bd2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I think this contribution may have "timed out", along with #2110 . They're probably good implementations, but I am not clear if this will be taken forward to be part of Spark. In any event it doesn't merge and is not necessarily written for the new ML pipelines API. Does anyone else have an opinion on whether this should be closed out, or needs to be revived?

[debasish83]

@fjiang6 in breeze we added ADMM based proximal minimizer and a proximal algorithm for Huber Loss...Could you please take a look at breeze.optimize.proximal.NonlinearMinimizer and Proximal.ProximalHuber and see if you can use NonlinearMinimizer ? Huber is as difficult as L1 (perhaps more difficult) since I have not seen a projection operator for Huber...For reference please follow Professor Boyd's paper on ADMM and Proximal Algorithms

The jar is available on 0.12-SNAPSHOTS

[SparkQA]

QA tests have started for PR 2096 at commit 9e34bd2.

• This patch does not merge cleanly!

[SparkQA]

QA tests have finished for PR 2096 at commit 9e34bd2.

• This patch fails unit tests.
• This patch does not merge cleanly!

[fjiang6]

Failed Tests:

org.apache.spark.streaming.kafka.JavaKafkaStreamSuite.testKafkaStream
org.apache.spark.streaming.kafka.KafkaStreamSuite.Kafka input stream
org.apache.spark.sql.hive.thriftserver.CliSuite.simple commands

[SparkQA]

QA tests have started for PR 2096 at commit 9e34bd2.

• This patch does not merge cleanly!

[SparkQA]

QA tests have finished for PR 2096 at commit 9e34bd2.

• This patch fails unit tests.
• This patch does not merge cleanly!