[SPARK-13332][SQL] Decimal datatype support for SQL pow

[yucai]

Decimal datatype support for SQL pow

• when base is Decimal and exponent is integer(Byte, Short, Int), calculate and return as Decimal
• otherwise, calculate and return as Double

https://issues.apache.org/jira/browse/SPARK-13332

[AmplabJenkins]

Can one of the admins verify this patch?

[yucai]

@adrian-wang could you help review? Much thanks!

[adrian-wang]

keep the test of f(v2, v1)

[adrian-wang]

maybe v.toDouble is better

[rxin]

I think it'd be a lot simpler if we create a separate Pow for Decimal, and handle byte/short/etc to integer in type coercion, rather than in the PowDecimal class.

[yucai]

OK, let me try this solution.

[yucai]

@rxin I tried your suggestion like creating PowDecimal for Decimal specially like below:

case class PowDecimal(left: Expression, right: Expression)
  extends BinaryMathExpression(math.pow, "POWER") {
  override def inputTypes: Seq[AbstractDataType] = Seq(DecimalType, IntegerType)
  ...
}

case class Pow(left: Expression, right: Expression)
  extends BinaryMathExpression(math.pow, "POWER") {
  ...
} 

But one concern is when "select pow(cast(2 as decimal(5,2)), 3)", how to make the "PowDecimal" node created? The current path will create "Pow" node anyway.

So we think of maybe we can still put Decimal processing in "Pow", but byte/short/etc. to integer in type coercion, like below:

case class Pow(left: Expression, right: Expression)
  extends BinaryMathExpression(math.pow, "POWER") {
  override def inputTypes: Seq[AbstractDataType] = Seq(NumericType, NumericType)

  override def dataType: DataType = (left.dataType, right.dataType) match {
    case (dt: DecimalType, ByteType | ShortType | IntegerType) => dt
    case _ => DoubleType
  }
  protected override def nullSafeEval(input1: Any, input2: Any): Any =
    (left.dataType, right.dataType) match {
      case (dt: DecimalType, _) => input1.asInstanceOf[Decimal].pow(input2.asInstanceOf[Int])
      case _ => math.pow(input1.asInstanceOf[Double], input2.asInstanceOf[Double])
    }
  override def genCode(ctx: CodegenContext, ev: ExprCode): String = ...
}

In HiveTypeCoercion:

  object PowCoercion extends Rule[LogicalPlan] {
    def apply(plan: LogicalPlan): LogicalPlan = plan resolveExpressions {
      case e if !e.childrenResolved => e
      case e @ Pow(left, right) =>
        (left.dataType, right.dataType) match {
          case (dt: DecimalType, IntegerType) => e
          case (DoubleType, DoubleType) => e
          case (dt: DecimalType, ByteType | ShortType) => Pow(left, Cast(right, IntegerType))
          case (DoubleType, _) => Pow(left, Cast(right, DoubleType))
          case (_, DoubleType) => Pow(Cast(left, DoubleType), right)
          case _ => Pow(Cast(left, DoubleType), Cast(right, DoubleType))
        }
    }
  }

How do you think of this way?

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.