[SPARK-18693][ML][MLLIB] ML Evaluators should use weight column

[imatiach-msft]

What changes were proposed in this pull request?

The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data.
The updates to the regression metrics were based on (and updated with new changes based on comments):
https://issues.apache.org/jira/browse/SPARK-11520
("RegressionMetrics should support instance weights")
but the pull request was closed as the changes were never checked in.

How was this patch tested?

This is still a work in progress, I will be adding more tests soon. I took the regression tests from:
#9907
Which was closed as a stale PR but I updated it with some changes.

[SparkQA]

Test build #71247 has finished for PR 16557 at commit 2b624c0.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"2.2.0\") (
• class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

[SparkQA]

Test build #71270 has finished for PR 16557 at commit f6d6fef.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

it looks like a random test timed out:
org.apache.spark.scheduler.BasicSchedulerIntegrationSuite.job with fetch failure
Error Details
java.util.concurrent.TimeoutException: Futures timed out after [1 second]
I will try to run again.

[imatiach-msft]

Jenkins retest this please

[imatiach-msft]

Jenkins, retest this please

[SparkQA]

Test build #71272 has finished for PR 16557 at commit f6d6fef.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71278 has finished for PR 16557 at commit 54fe922.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71291 has started for PR 16557 at commit 397c26b.

[imatiach-msft]

Jenkins, retest this please

[imatiach-msft]

Jenkins, retest this please

[imatiach-msft]

Jenkins, retest this please

[imatiach-msft]

Jenkins doesn't seem to be working ...

[imatiach-msft]

@sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

[imatiach-msft]

Jenkins, retest this please

[SparkQA]

Test build #71331 has finished for PR 16557 at commit 808ca6b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"2.2.0\") (
• class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

[SparkQA]

Test build #71334 has finished for PR 16557 at commit 808ca6b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"2.2.0\") (
• class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

[imatiach-msft]

Jenkins, retest this please

[SparkQA]

Test build #71523 has finished for PR 16557 at commit 228bbfb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"2.2.0\") (
• class MulticlassMetrics @Since(\"1.1.0\") (predAndLabelsWithOptWeight: RDD[_])

[SparkQA]

Test build #71531 has finished for PR 16557 at commit e2873f2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

Jenkins, retest this please

[SparkQA]

Test build #71544 has finished for PR 16557 at commit e2873f2.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

Jenkins, retest this please

[imatiach-msft]

Jenkins, retest this please

[SparkQA]

Test build #71604 has finished for PR 16557 at commit e2873f2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71607 has finished for PR 16557 at commit d589dfe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71670 has finished for PR 16557 at commit c1f5f09.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class SparkListenerExecutorBlacklisted(
• case class SparkListenerExecutorUnblacklisted(time: Long, executorId: String)
• case class SparkListenerNodeBlacklisted(
• case class SparkListenerNodeUnblacklisted(time: Long, hostId: String)
• case class QualifiedTableName(database: String, name: String)
• class FindHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]

[imatiach-msft]

Jenkins, retest this please

[imatiach-msft]

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

[SparkQA]

Test build #71858 has finished for PR 16557 at commit ba68f72.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #71860 has finished for PR 16557 at commit 6dbb0ad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

[imatiach-msft]

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley would you be able to take a look at the changes to add a weight column to binary/multiclass/regression evaluators/metrics classes? It looks like you are familiar with this code. Thank you!

[imatiach-msft]

@srowen @yanboliang might you be able to take a look at this PR? Is it possibly too large and I should break it up into 3 PRs, one per evaluator/metrics class?

[srowen]

I wouldn't ping that frequently, please. I don't feel qualified to review this myself.

[sethah]

+1 for breaking it up, maybe starting with regression. Also, just because something hasn't been reviewed in two weeks does not mean that there is no interest in it. Two weeks is not all that long (I've seen valuable PRs sit for much longer than that) and it likely just means people are busy. As Sean pointed out, it's probably not necessary to ping once per day.

[thunterdb]

I agree, let's break this PR. It will go faster, and some changes may require longer discussions.

[imatiach-msft]

ok, I will close this and create three new PRs, one for each of the evaluators

[imatiach-msft]

I've created 3 PRs, located here:
#17084
#17085
#17086

[SparkQA]

Test build #73525 has finished for PR 16557 at commit a0fc4c3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.