[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math #7451

yjshen · 2015-07-16T18:41:54Z

JIRA:
https://issues.apache.org/jira/browse/SPARK-8280
https://issues.apache.org/jira/browse/SPARK-8281

yjshen · 2015-07-16T18:50:23Z

Log family semantic comparision:

Expression	Hive	Spark SQL
ln(0.0)	NULL	-Infinity
ln(-1)	NULL	NaN

Since we disable udf7 to enable log base between (0.0, 1], we do not have log related comparsion with Hive. So, what about other semantics of Log?

yjshen · 2015-07-16T18:52:32Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

@@ -387,27 +370,18 @@ case class Atan2(left: Expression, right: Expression)

  protected override def nullSafeEval(input1: Any, input2: Any): Any = {
    // With codegen, the values returned by -0.0 and 0.0 are different. Handled with +0.0
-    val result = math.atan2(input1.asInstanceOf[Double] + 0.0, input2.asInstanceOf[Double] + 0.0)
-    if (result.isNaN) null else result
+    math.atan2(input1.asInstanceOf[Double] + 0.0, input2.asInstanceOf[Double] + 0.0)


Still need + 0.0 here, math.atan2 is calling java.lang.Math.atan2 inside

yjshen · 2015-07-16T18:57:03Z

cc @rxin

rxin · 2015-07-16T19:27:18Z

Let's follow Hive for log, and make sure we add a unit test.

yjshen · 2015-07-17T01:53:54Z

So we behave like Hive except that we do support log base in between (0.0, 1.0], and add a unit test like udf7

rxin · 2015-07-17T01:54:55Z

Yup.

rxin · 2015-07-17T06:46:57Z

Jenkins, add to whitelist.

rxin · 2015-07-17T06:48:58Z

Jenkins, ok to test.

rxin · 2015-07-17T07:01:36Z

Jenkins, test this please.

rxin · 2015-07-17T07:01:48Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

+          if ($c1 <= 0.0 || $c2 <= 0.0) {
+            ${ev.isNull} = true;
+          } else {
+           ${ev.primitive} = java.lang.Math.log($c2) / java.lang.Math.log($c1);


missed a space here

rxin · 2015-07-17T07:36:31Z

Jenkins, retest this please.

SparkQA · 2015-07-17T08:33:40Z

Test build #37596 has finished for PR 7451 at commit fb73b87.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class UnaryLogExpression(f: Double => Double, name: String)
- case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
- case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
- case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")

SparkQA · 2015-07-17T09:01:46Z

Test build #37605 has finished for PR 7451 at commit 003ab23.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RFormula(override val uid: String)
- abstract class LeafExpression extends Expression
- abstract class UnaryExpression extends Expression
- abstract class BinaryExpression extends Expression
- abstract class UnaryLogExpression(f: Double => Double, name: String)
- case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
- case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
- case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")
- case class Length(child: Expression) extends UnaryExpression with ExpectsInputTypes
- case class FormatNumber(x: Expression, d: Expression)
- abstract class BinaryNode extends LogicalPlan

yjshen · 2015-07-17T10:09:17Z

rebased the code to resolve merge conflicts.

rxin · 2015-07-17T16:32:01Z

Jenkins, retest this please.

SparkQA · 2015-07-17T18:07:27Z

Test build #37646 has finished for PR 7451 at commit a83b47c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class UnsafeProjection extends Projection
- case class FromUnsafeProjection(fields: Seq[DataType]) extends Projection
- abstract class BaseProjection extends Projection
- class SpecificProjection extends $
- class SpecificProjection extends $
- abstract class UnaryLogExpression(f: Double => Double, name: String)
- case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
- case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
- case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")

rxin · 2015-07-17T19:07:24Z

Oops there is another conflict. I tried resolving this locally but it was a big one. Can you resolve it?

yjshen · 2015-07-17T19:09:41Z

yes, I've resolved it locally, push here again in a minute.

rxin · 2015-07-17T19:17:41Z

Jenkins, retest this please.

rxin · 2015-07-17T23:17:32Z

Jenkins, retest this please.

rxin · 2015-07-18T00:33:08Z

Thanks - I've merged this.

SparkQA · 2015-07-18T00:48:07Z

Test build #1097 has finished for PR 7451 at commit 47a529d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class UnaryLogExpression(f: Double => Double, name: String)
- case class Conv(numExpr: Expression, fromBaseExpr: Expression, toBaseExpr: Expression)
- case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
- case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
- case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")

SparkQA · 2015-07-18T00:53:04Z

Test build #37676 has finished for PR 7451 at commit 47a529d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class UnaryLogExpression(f: Double => Double, name: String)
- case class Log(child: Expression) extends UnaryLogExpression(math.log, "LOG")
- case class Log10(child: Expression) extends UnaryLogExpression(math.log10, "LOG10")
- case class Log1p(child: Expression) extends UnaryLogExpression(math.log1p, "LOG1P")

yjshen reviewed Jul 16, 2015
View reviewed changes

yjshen mentioned this pull request Jul 16, 2015

[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math #6835

Closed

rxin reviewed Jul 17, 2015
View reviewed changes

yjshen force-pushed the nan_null2 branch from 003ab23 to a83b47c Compare July 17, 2015 10:07

yjshen added 3 commits July 18, 2015 02:20

null to nan in Math Expression

188be51

handle log expressions similar to Hive

63dee44

style fix

47a529d

yjshen force-pushed the nan_null2 branch from a83b47c to 47a529d Compare July 17, 2015 19:15

asfgit closed this in 529a2c2 Jul 18, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math #7451

[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math #7451

yjshen commented Jul 16, 2015

yjshen commented Jul 16, 2015

yjshen Jul 16, 2015

yjshen commented Jul 16, 2015

rxin commented Jul 16, 2015

yjshen commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin Jul 17, 2015

rxin commented Jul 17, 2015

SparkQA commented Jul 17, 2015

SparkQA commented Jul 17, 2015

yjshen commented Jul 17, 2015

rxin commented Jul 17, 2015

SparkQA commented Jul 17, 2015

rxin commented Jul 17, 2015

yjshen commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 18, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 18, 2015

[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math #7451

[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math #7451

Conversation

yjshen commented Jul 16, 2015

yjshen commented Jul 16, 2015

yjshen Jul 16, 2015

Choose a reason for hiding this comment

yjshen commented Jul 16, 2015

rxin commented Jul 16, 2015

yjshen commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin Jul 17, 2015

Choose a reason for hiding this comment

rxin commented Jul 17, 2015

SparkQA commented Jul 17, 2015

SparkQA commented Jul 17, 2015

yjshen commented Jul 17, 2015

rxin commented Jul 17, 2015

SparkQA commented Jul 17, 2015

rxin commented Jul 17, 2015

yjshen commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 17, 2015

rxin commented Jul 18, 2015

SparkQA commented Jul 18, 2015

SparkQA commented Jul 18, 2015