[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator #8013

fjiang6 · 2015-08-07T00:19:20Z

Huber Robust Regression under spark/ml/regression
Unit Tests

SparkQA · 2015-08-07T01:06:32Z

Test build #40111 has finished for PR 8013 at commit 2f67e63.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RobustRegression(override val uid: String)

fjiang6 · 2015-08-07T01:11:20Z

@mengxr @dbtsai @srowen had RobustRegression in the same LinearRegression codebase as requested. And included the Unit Tests.

dbtsai · 2015-08-07T16:15:05Z

Still a lot of duplication. We're adding new features into LiR now, and it will be hard to maintain. Is it possible that you just add the objective function, and use Params to switch between different objective function? Thanks.

SparkQA · 2015-08-08T03:26:24Z

Test build #40222 has finished for PR 8013 at commit 96e38a7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-08T04:20:28Z

Test build #40223 has finished for PR 8013 at commit 23e4c62.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

fjiang6 · 2015-08-08T04:22:29Z

@dbtsai ust added the objective function, and use Params to switch between different objective function. Thanks!

dbtsai · 2015-08-11T07:34:16Z

mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala

@@ -325,4 +325,21 @@ private[ml] trait HasStepSize extends Params {
  /** @group getParam */
  final def getStepSize: Double = $(stepSize)
 }
+


sharedParams.scala can not be edited directly. Please look at SharedParamsCodeGen.scala.

Also, make HasRobust as HasRobustRegression in SharedParamsCodeGen.scala.

SparkQA · 2015-08-11T23:07:54Z

Test build #40530 has finished for PR 8013 at commit 51e47dc.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EqualNullSafe(attribute: String, value: Any) extends Filter

fjiang6 · 2015-08-11T23:10:20Z

This class was not added by me. I didn't touch PySpark.

SparkQA · 2015-08-23T14:25:48Z

Test build #41422 has finished for PR 8013 at commit 1567635.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-23T15:08:14Z

Test build #41421 has finished for PR 8013 at commit 3bb5930.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-23T15:10:35Z

Test build #41423 has finished for PR 8013 at commit a04179b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

feynmanliang · 2015-08-24T18:39:28Z

mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala

@@ -65,6 +65,10 @@ private[shared] object SharedParamsCodeGen {
        isValid = "ParamValidators.inArray(Array(\"skip\", \"error\"))"),
      ParamDesc[Boolean]("standardization", "whether to standardize the training features" +
        " before fitting the model.", Some("true")),
+      ParamDesc[Boolean]("robustRegression", "whether to use robust Huber Cost Function",


I think it would be better to introduce a "costFunction" param which defaults to "LeastSquares" and pattern match in LinearRegression#L195 since that will force mutual exclusivity when more than two cost functions are possible

SparkQA · 2015-08-27T01:45:36Z

Test build #41662 has finished for PR 8013 at commit e447623.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

feynmanliang · 2015-08-27T02:29:05Z

mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala

@@ -97,6 +97,22 @@ class LinearRegression(override val uid: String)
  setDefault(standardization -> true)

  /**
+   * Set the robust Option to determine whether to use robust Huber Cost Function


nit: This is not really an "Option", can we just make this say " Set whether to use robust Huber Cost Function"

feynmanliang · 2015-08-31T22:36:27Z

There is a lot of code repetition between this and #2096, perhaps you can make the mllib one wrap this?

dbtsai · 2015-09-01T01:29:01Z

Hello, robust tuning parameter k should not be a constant as you implemented.
In the paper, http://users.stat.umn.edu/~sandy/courses/8053/handouts/robust.pdf
k = 1.345σ where σ is the square error of current square loss. But this will be very expensive to compute the current square error of current square loss and then compute the huber loss, so I think it's reasonable to approximate the square error from previous weight.

add the objective function, and use Params to switch edit to pass scala style tests make HasRobustRegression in SharedParamsCodeGen.scala, Make the document more explicitly and make k tunable and default to 1.345 by having another param UnitTests with Outliers UnitTests with Outliers Edit HuberAggregator scala codestyle Update LinearRegression.scala

SparkQA · 2016-01-17T16:14:38Z

Test build #49555 has finished for PR 8013 at commit 01601ee.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-06-15T22:20:25Z

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.

@dbtsai there are a few pull requests that were waiting on your review. Can you revisit them even if they are closed?

dbtsai · 2016-07-04T23:00:32Z

@rxin @mengxr I'm back to US from a leave. Going to revisit PRs under me.

I had worked with @MechCoder to implement Huber estimator in python scikit scikit-learn/scikit-learn#5291 which had been merged. @fjiang6, @MechCoder, @sethah, are you interested in porting this feature to Spark which should be fairly straightforward?

Thanks.

yanboliang · 2016-07-05T11:12:28Z

@dbtsai I'm interested in porting Huber estimator to Spark. If you did not start it, I can send a PR in a few days. Thanks!

dbtsai · 2016-07-05T15:20:40Z

@yanboliang Sounds great! Thanks.

MechCoder · 2016-07-05T16:26:24Z

I'll be happy to review it.

dbtsai reviewed Aug 11, 2015
View reviewed changes

feynmanliang reviewed Aug 24, 2015
View reviewed changes

feynmanliang reviewed Aug 27, 2015
View reviewed changes

MechCoder force-pushed the Huawei-Robust branch from e447623 to 01601ee Compare January 17, 2016 16:01

asfgit closed this in 1a33f2e Jun 15, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator #8013

[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator #8013

fjiang6 commented Aug 7, 2015

SparkQA commented Aug 7, 2015

fjiang6 commented Aug 7, 2015

dbtsai commented Aug 7, 2015

SparkQA commented Aug 8, 2015

SparkQA commented Aug 8, 2015

fjiang6 commented Aug 8, 2015

dbtsai Aug 11, 2015

dbtsai Aug 11, 2015

SparkQA commented Aug 11, 2015

fjiang6 commented Aug 11, 2015

SparkQA commented Aug 23, 2015

SparkQA commented Aug 23, 2015

SparkQA commented Aug 23, 2015

feynmanliang Aug 24, 2015

SparkQA commented Aug 27, 2015

feynmanliang Aug 27, 2015

feynmanliang commented Aug 31, 2015

dbtsai commented Sep 1, 2015

SparkQA commented Jan 17, 2016

rxin commented Jun 15, 2016

dbtsai commented Jul 4, 2016

yanboliang commented Jul 5, 2016

dbtsai commented Jul 5, 2016

MechCoder commented Jul 5, 2016

[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator #8013

[SPARK-3181][MLLIB]: Add Robust Regression Algorithm with Huber Estimator #8013

Conversation

fjiang6 commented Aug 7, 2015

SparkQA commented Aug 7, 2015

fjiang6 commented Aug 7, 2015

dbtsai commented Aug 7, 2015

SparkQA commented Aug 8, 2015

SparkQA commented Aug 8, 2015

fjiang6 commented Aug 8, 2015

dbtsai Aug 11, 2015

Choose a reason for hiding this comment

dbtsai Aug 11, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 11, 2015

fjiang6 commented Aug 11, 2015

SparkQA commented Aug 23, 2015

SparkQA commented Aug 23, 2015

SparkQA commented Aug 23, 2015

feynmanliang Aug 24, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 27, 2015

feynmanliang Aug 27, 2015

Choose a reason for hiding this comment

feynmanliang commented Aug 31, 2015

dbtsai commented Sep 1, 2015

SparkQA commented Jan 17, 2016

rxin commented Jun 15, 2016

dbtsai commented Jul 4, 2016

yanboliang commented Jul 5, 2016

dbtsai commented Jul 5, 2016

MechCoder commented Jul 5, 2016