[SPARK-6994] Allow to fetch field values by name in sql.Row #5573

vidma · 2015-04-18T15:07:57Z

It looked weird that up to now there was no way in Spark's Scala API to access fields of DataFrame/sql.Row by name, only by their index.

This tries to solve this issue.

rxin · 2015-04-19T00:21:47Z

sql/catalyst/src/main/scala/org/apache/spark/sql/RowImplicits.scala

+
+object RowImplicits {
+
+  implicit class RowWithNamedFields[K, V](sqlRow: Row) {


why don't we just add this to Row itself?

👍 if so, I guess I should rename getValue[T] into an overloaded getAs[T](fieldName: String) to be consistent with an existing method getAs? (I remember overloading didn't work with implicits...)

yup sounds good

For the new method to be convenient to use it should go into Row.
However, the schema is not always defined for a Row, so Row.getAs[T](fieldName: String) with throw an UnsupportedOperationException.
GenericRowWithSchema will implement the case when schema exists.

vidma · 2015-04-19T12:22:07Z

moved the changes to the Row and RowWithSchema classes, and also added a fieldIndex to the schema (StructType).

also got rid of dependency on HiveTest.

- add fieldIndex(name: String) - add getAs[T](fieldName: String) - add getValuesMap[T] returning a map of values for the requested fieldNames

vidma · 2015-04-19T12:46:00Z

it's slightly weird that there exists a RowSuite.scala in sql-core and none in sql-catalyst.
Anyway, the one in sql-core looks like integration test, so I added a little test there too, to make sure it works on Rows created via DataFrames.

P.S. isn't jenkins supposted to run tests on this PR?

marmbrus · 2015-04-20T23:33:20Z

ok to test

SparkQA · 2015-04-21T01:30:04Z

Test build #30611 has finished for PR 5573 at commit 6145ae3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

marmbrus · 2015-04-21T21:47:30Z

Thanks, merged to master.

It looked weird that up to now there was no way in Spark's Scala API to access fields of `DataFrame/sql.Row` by name, only by their index. This tries to solve this issue. Author: vidmantas zemleris <[email protected]> Closes apache#5573 from vidma/features/row-with-named-fields and squashes the following commits: 6145ae3 [vidmantas zemleris] [SPARK-6994][SQL] Allow to fetch field values by name on Row 9564ebb [vidmantas zemleris] [SPARK-6994][SQL] Add fieldIndex to schema (StructType)

vidma force-pushed the features/row-with-named-fields branch 2 times, most recently from 496cd36 to 10ae4d8 Compare April 18, 2015 17:00

vidma changed the title ~~Add named field support for sql.Row~~ [SPARK-6994] Allow to fetch field values by name in sql.Row Apr 18, 2015

rxin reviewed Apr 19, 2015
View reviewed changes

vidma force-pushed the features/row-with-named-fields branch 9 times, most recently from d823862 to 49698a4 Compare April 19, 2015 12:17

[SPARK-6994][SQL] Add fieldIndex to schema (StructType)

9564ebb

vidma force-pushed the features/row-with-named-fields branch from 49698a4 to a9a8854 Compare April 19, 2015 12:19

vidma force-pushed the features/row-with-named-fields branch from a9a8854 to 0a2b8b0 Compare April 19, 2015 12:31

[SPARK-6994][SQL] Allow to fetch field values by name on Row

6145ae3

- add fieldIndex(name: String) - add getAs[T](fieldName: String) - add getValuesMap[T] returning a map of values for the requested fieldNames

vidma force-pushed the features/row-with-named-fields branch from 0a2b8b0 to 6145ae3 Compare April 19, 2015 12:41

asfgit closed this in 2e8c6ca Apr 21, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-6994] Allow to fetch field values by name in sql.Row #5573

[SPARK-6994] Allow to fetch field values by name in sql.Row #5573

vidma commented Apr 18, 2015

rxin Apr 19, 2015

vidma Apr 19, 2015

rxin Apr 19, 2015

vidma Apr 19, 2015

vidma commented Apr 19, 2015

vidma commented Apr 19, 2015

marmbrus commented Apr 20, 2015

SparkQA commented Apr 21, 2015

marmbrus commented Apr 21, 2015


		object RowImplicits {

		implicit class RowWithNamedFields[K, V](sqlRow: Row) {

[SPARK-6994] Allow to fetch field values by name in sql.Row #5573

[SPARK-6994] Allow to fetch field values by name in sql.Row #5573

Conversation

vidma commented Apr 18, 2015

rxin Apr 19, 2015

Choose a reason for hiding this comment

vidma Apr 19, 2015

Choose a reason for hiding this comment

rxin Apr 19, 2015

Choose a reason for hiding this comment

vidma Apr 19, 2015

Choose a reason for hiding this comment

vidma commented Apr 19, 2015

vidma commented Apr 19, 2015

marmbrus commented Apr 20, 2015

SparkQA commented Apr 21, 2015

marmbrus commented Apr 21, 2015