Skip to content

Commit

Permalink
[SPARK-43378][CORE] Properly close stream objects in deserializeFromC…
Browse files Browse the repository at this point in the history
…hunkedBuffer

### What changes were proposed in this pull request?

Fixes a that SerializerHelper.deserializeFromChunkedBuffer does not call close on the deserialization stream. For some serializers like Kryo this creates a performance regressions as the kryo instances are not returned to the pool.

### Why are the changes needed?
This causes a performance regression in Spark 3.4.0 for some workloads.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Existing tests.

Closes #41049 from eejbyfeldt/SPARK-43378.

Authored-by: Emil Ejbyfeldt <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
  • Loading branch information
Emil Ejbyfeldt authored and srowen committed May 5, 2023
1 parent 05df8c4 commit cb26ad8
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ private[spark] object SerializerHelper extends Logging {
serializerInstance: SerializerInstance,
bytes: ChunkedByteBuffer): T = {
val in = serializerInstance.deserializeStream(bytes.toInputStream())
in.readObject()
val res = in.readObject()
in.close()
res
}
}

0 comments on commit cb26ad8

Please sign in to comment.