[SPARK-24103][ML][MLLIB] ML Evaluators should use weight column - added weight column for binary classification evaluator

[imatiach-msft]

What changes were proposed in this pull request?

The evaluators BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator and the corresponding metrics classes BinaryClassificationMetrics, RegressionMetrics and MulticlassMetrics should use sample weight data.

I've closed the PR: #16557
as recommended in favor of creating three pull requests, one for each of the evaluators (binary/regression/multiclass) to make it easier to review/update.

How was this patch tested?

I added tests to the metrics and evaluators classes.

[SparkQA]

Test build #73526 has finished for PR 17084 at commit 98652cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"2.2.0\") (

[imatiach-msft]

@sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley @actuaryzhang @srowen would you be able to take a look? I've split the larger pull request into three parts as suggested.

[imatiach-msft]

ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley @actuaryzhang @srowen could you please take a look? thank you!

[actuaryzhang]

Thanks for the PR. I think this is helpful. Will take a look next week. Quite swamped recently.

[HyukjinKwon]

gentle ping @actuaryzhang

[actuaryzhang]

@imatiach-msft Thanks for the PR. Added a couple of comments. Sorry for the delayed review.

[HyukjinKwon]

gentle ping @imatiach-msft .

[imatiach-msft]

yes, will update the PR, thanks for the ping

[imatiach-msft]

Jenkins, test this please

[SparkQA]

Test build #78597 has finished for PR 17084 at commit cf59c62.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"2.2.0\") (

[SparkQA]

Test build #78598 has finished for PR 17084 at commit 60fc2a7.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

the pip packaging failing seems to be unrelated to the code... let me try this again

[imatiach-msft]

Jenkins, test this please

[SparkQA]

Test build #78606 has finished for PR 17084 at commit 60fc2a7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

gentle ping @actuaryzhang, thanks!

[imatiach-msft]

Jenkins, test this please

[SparkQA]

Test build #89405 has finished for PR 17084 at commit e89a030.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

gently ping @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley @actuaryzhang @srowen could you please take a look? I've merged with latest code. Thank you!

[SparkQA]

Test build #102046 has finished for PR 17084 at commit 793a284.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"3.0.0\") (

[SparkQA]

Test build #102177 has finished for PR 17084 at commit e78076b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationMetrics @Since(\"3.0.0\") (

[holdenk]

I did a quick first pass review, thanks for working on this :)

[SparkQA]

Test build #102510 has finished for PR 17084 at commit 955b022.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[imatiach-msft]

@holdenk @srowen @actuaryzhang would you be able to take another look at this PR when you have a chance? I've updated to latest, fixed all failing tests and I think I have fixed all of the comments. Thank you!

[srowen]

It's looking OK; does Pyspark need an update, and does the multiclass evaluator need an update to match?

[imatiach-msft]

@srowen multiclass weight columns PR is already merged:
#17086
The pyspark side will need to be updated. Should that be part of this PR or a separate PR?

[srowen]

Ah right, long since lost track. Yeah it would make sense to update it here too. I don't think R has an evaluator?

[SparkQA]

Test build #102621 has finished for PR 17084 at commit 079e114.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #102720 has finished for PR 17084 at commit a571adc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class BinaryClassificationEvaluator(JavaEvaluator, HasLabelCol, HasRawPredictionCol, HasWeightCol,

[SparkQA]

Test build #102733 has finished for PR 17084 at commit d8a5865.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class MulticlassMetrics @Since(\"1.1.0\") (predictionAndLabels: RDD[_ <: Product])

[SparkQA]

Test build #102734 has finished for PR 17084 at commit 00bfec1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class MulticlassMetrics @Since(\"1.1.0\") (predictionAndLabels: RDD[_ <: Product])

[imatiach-msft]

@srowen I think I have fixed all pending comments and the tests are currently passing for this PR. I couldn't find any evaluators in the R code, but then again I am not as familiar with sparkR (however, I've used sparklyr before https://github.com/rstudio/sparklyr). Please let me know if there are any other comments that need to be addressed. Thank you!

[srowen]

Merged to master