[SQL] Improve DataFrame API error reporting #4296

rxin · 2015-01-31T06:29:24Z

Throw UnsupportedOperationException if a Column is not computable.
Perform eager analysis on DataFrame so we can catch errors when they happen (not when an action is run).

SparkQA · 2015-01-31T06:32:50Z

Test build #26456 has started for PR 4296 at commit 265fde7.

This patch merges cleanly.

SparkQA · 2015-01-31T07:26:48Z

Test build #26456 has finished for PR 4296 at commit 265fde7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait Column extends DataFrame with ExpressionApi
- class ColumnName(name: String) extends IncomputableColumn(name)
- trait DataFrame extends DataFrameSpecificApi with RDDApi[Row]
- class GroupedDataFrame protected[sql](df: DataFrameImpl, groupingExprs: Seq[Expression])
- protected[sql] class QueryExecution(val logical: LogicalPlan)

AmplabJenkins · 2015-01-31T07:26:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26456/
Test FAILed.

This patch changes Column from a concrete implementation to a trait, and provides two concrete implementations: IncomputableColumn and ComputableColumn.

marmbrus · 2015-02-02T03:12:46Z

sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala

+      testData.groupBy('key).agg(Map("nonExistentName" -> "sum"))
+    }
+    intercept[Exception] {
+      testData.groupBy("nonExistentName").agg(Map("key" -> "sum"))


Why isn't this (String, String)*?

SparkQA · 2015-02-02T04:32:47Z

Test build #26487 has started for PR 4296 at commit 17f6bae.

This patch merges cleanly.

SparkQA · 2015-02-02T04:43:50Z

Test build #26487 has finished for PR 4296 at commit 17f6bae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait Column extends DataFrame with ExpressionApi
- class ColumnName(name: String) extends IncomputableColumn(name)
- trait DataFrame extends DataFrameSpecificApi with RDDApi[Row]
- class GroupedDataFrame protected[sql](df: DataFrameImpl, groupingExprs: Seq[Expression])
- protected[sql] class QueryExecution(val logical: LogicalPlan)

AmplabJenkins · 2015-02-02T04:43:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26487/
Test FAILed.

SparkQA · 2015-02-02T23:07:44Z

Test build #26553 has started for PR 4296 at commit 5afe1ff.

This patch merges cleanly.

SparkQA · 2015-02-03T00:06:56Z

Test build #26553 has finished for PR 4296 at commit 5afe1ff.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-03T00:06:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26553/
Test FAILed.

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala

fix python tests

SparkQA · 2015-02-03T01:03:00Z

Test build #26564 has started for PR 4296 at commit 6527b86.

This patch merges cleanly.

SparkQA · 2015-02-03T02:42:27Z

Test build #26564 has finished for PR 4296 at commit 6527b86.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LogisticGradient(numClasses: Int) extends Gradient
- trait Column extends DataFrame with ExpressionApi
- class ColumnName(name: String) extends IncomputableColumn(name)
- trait DataFrame extends DataFrameSpecificApi with RDDApi[Row]
- class GroupedDataFrame protected[sql](df: DataFrameImpl, groupingExprs: Seq[Expression])
- protected[sql] class QueryExecution(val logical: LogicalPlan)
- case class HiveScriptIOSchema (
- val trimed_class = serdeClassName.split("'")(1)

AmplabJenkins · 2015-02-03T02:42:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26564/
Test FAILed.

pwendell · 2015-02-03T02:53:05Z

Jenkins, test this please

SparkQA · 2015-02-03T02:58:01Z

Test build #26587 has started for PR 4296 at commit 6527b86.

This patch merges cleanly.

pwendell · 2015-02-03T03:01:26Z

@rxin this test failure was spurious. It is safe to merge.

SparkQA · 2015-02-03T04:36:57Z

Test build #26587 has finished for PR 4296 at commit 6527b86.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait Column extends DataFrame with ExpressionApi
- class ColumnName(name: String) extends IncomputableColumn(name)
- trait DataFrame extends DataFrameSpecificApi with RDDApi[Row]
- class GroupedDataFrame protected[sql](df: DataFrameImpl, groupingExprs: Seq[Expression])
- protected[sql] class QueryExecution(val logical: LogicalPlan)

AmplabJenkins · 2015-02-03T04:37:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26587/
Test PASSed.

rxin added 2 commits February 1, 2015 18:18

[SQL][API] ComputableColumn vs IncomputableColumn

e6f00b8

This patch changes Column from a concrete implementation to a trait, and provides two concrete implementations: IncomputableColumn and ComputableColumn.

Added eager analysis for error reporting.

b932e86

marmbrus reviewed Feb 2, 2015
View reviewed changes

Various fixes.

17f6bae

rxin force-pushed the col-computability branch from 265fde7 to 17f6bae Compare February 2, 2015 04:31

Fix scala test.

5afe1ff

Davies Liu and others added 3 commits February 2, 2015 16:57

fix python tests

f79034c

Merge branch 'master' into col-computability

fd92bc7

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala

Merge pull request #8 from davies/col-computability

6527b86

fix python tests

asfgit closed this in 554403f Feb 3, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SQL] Improve DataFrame API error reporting #4296

[SQL] Improve DataFrame API error reporting #4296

rxin commented Jan 31, 2015

SparkQA commented Jan 31, 2015

SparkQA commented Jan 31, 2015

AmplabJenkins commented Jan 31, 2015

marmbrus Feb 2, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 2, 2015

AmplabJenkins commented Feb 2, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

pwendell commented Feb 3, 2015

SparkQA commented Feb 3, 2015

pwendell commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

[SQL] Improve DataFrame API error reporting #4296

[SQL] Improve DataFrame API error reporting #4296

Conversation

rxin commented Jan 31, 2015

SparkQA commented Jan 31, 2015

SparkQA commented Jan 31, 2015

AmplabJenkins commented Jan 31, 2015

marmbrus Feb 2, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 2, 2015

SparkQA commented Feb 2, 2015

AmplabJenkins commented Feb 2, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

pwendell commented Feb 3, 2015

SparkQA commented Feb 3, 2015

pwendell commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015