Getting Null Values when using SparkSql

[bhavikpatel91]

Hi,

I am loading DynamoDb table and Registering it for querying using SparkSql.

e.g

//Step 1
val TableDf = spark.read.dynamodb("some table")

//Step 2
val TableDf = spark.read.dynamodb("some table").cache()

TableDf.createOrReplaceTempView("someTable")

val dataDF = spark.sql("select * from someTable")
// This shows all the values as saved in Dynamo Table
dataDF.show()

val newDF = spark.sql("select a,b,c from someTable")
// When using Step 1 This shows records of most columns as null
newDF.show()

But when I use Step 2 i.e cached dataframe to register table the data is shown as expected.

I have a big dynamo table and can't cache it!

[mkanchwala]

+1 Facing same issue on Amazon EMR as well.