Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Integration cache test for BroadcastNestedLoopJoin failure #356

Closed
razajafri opened this issue Jul 14, 2020 · 0 comments
Closed

[BUG] Integration cache test for BroadcastNestedLoopJoin failure #356

razajafri opened this issue Jul 14, 2020 · 0 comments
Labels
bug Something isn't working
Milestone
Jul 6 - Jul 17

Comments

@razajafri
Copy link
Collaborator

Describe the bug
When running a crossJoin using BroadcastNestedLoopJoin, the query results in a NPE.

here is the stack trace

src/main/python/cache_test.py::test_cache_broadcast_nested_loop_join[Left-String] 20/07/14 21:34:47 ERROR Executor: Exception in task 5.0 in stage 8.0 (TID 90)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 4.0 in stage 8.0 (TID 89)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 6.0 in stage 8.0 (TID 91)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 9.0 in stage 8.0 (TID 94)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 3.0 in stage 8.0 (TID 88)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 1.0 in stage 8.0 (TID 86)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 10.0 in stage 8.0 (TID 95)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 11.0 in stage 8.0 (TID 96)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 2.0 in stage 8.0 (TID 87)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 8.0 in stage 8.0 (TID 93)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 7.0 in stage 8.0 (TID 92)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 ERROR Executor: Exception in task 0.0 in stage 8.0 (TID 85)
java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/07/14 21:34:47 WARN TaskSetManager: Lost task 11.0 in stage 8.0 (TID 96, 192.168.0.121, executor driver): java.lang.NullPointerException
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.readObject(GpuBroadcastExchangeExec.scala:90)
	at org.apache.spark.sql.rapids.execution.SerializeConcatHostBuffersDeserializeBatch.batch(GpuBroadcastExchangeExec.scala:63)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$2(GpuBroadcastNestedLoopJoinExec.scala:204)
	at com.nvidia.spark.rapids.Arm.withResource(Arm.scala:26)
	at com.nvidia.spark.rapids.Arm.withResource$(Arm.scala:24)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.withResource(GpuBroadcastNestedLoopJoinExec.scala:130)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$lzycompute$1(GpuBroadcastNestedLoopJoinExec.scala:203)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.builtTable$2(GpuBroadcastNestedLoopJoinExec.scala:202)
	at org.apache.spark.sql.rapids.execution.GpuBroadcastNestedLoopJoinExec.$anonfun$doExecuteColumnar$4(GpuBroadcastNestedLoopJoinExec.scala:218)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

20/07/14 21:34:47 ERROR TaskSetManager: Task 11 in stage 8.0 failed 1 times; aborting job

Steps/Code to reproduce bug
Run the cache_test.py#test_cache_broadcast_nested_loop_join

Expected behavior
The cache_test.py#test_cache_broadcast_nested_loop_join should pass
@razajafri razajafri added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jul 14, 2020
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jul 17, 2020
@sameerz sameerz added this to the Jul 6 - Jul 17 milestone Jul 17, 2020
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to fd3cf3e (NVIDIA#356)
2fc6cd4 
Signed-off-by: spark-rapids automation <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
Jul 6 - Jul 17
Development

No branches or pull requests
2 participants
@sameerz @razajafri