Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Closes some resources in withResource early after being used #11713

Open
thirtiseven opened this issue Nov 12, 2024 · 0 comments
Open

[FEA] Closes some resources in withResource early after being used #11713

thirtiseven opened this issue Nov 12, 2024 · 0 comments
Labels
good first issue Good for newcomers reliability Features to improve reliability or bugs that severly impact the reliability of the plugin

Comments

@thirtiseven
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

When using withResource to manage resources, we'd better close the resource just after it's used, to reduce peak memory usage. For example, the following code (in GpuCast.scala)

withResource(input.getChildColumnView(0)) { childView =>
	withResource(doCast(childView, nestedFrom, nestedTo, options)) { childColumnVector =>
		withResource(input.replaceListChild(childColumnVector))(_.copyToColumnVector())
	}
}

can be rewritten as

val childColumnVector = withResource(input.getChildColumnView(0)) { childView =>
	doCast(childView, nestedFrom, nestedTo, options)
}
withResource(childColumnVector) { childColumnVector =>
	withResource(input.replaceListChild(childColumnVector))(_.copyToColumnVector())
}

to close childView early after being used.

There are some other cases in our codebase we can improve in this way.

Describe the solution you'd like
I wrote a simple checker script to find such cases. It is just based on some regex and hand-written rules, so some cases may be missed and some false positives may be reported.

Here is the report, we can check and fix them manually, to reduce the chance of GPU OOM.

/spark-rapids/sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/GpuWindowGroupLimitExec.scala, line 150: 'inputCB' can be closed early.
/spark-rapids/sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/GpuToPrettyString.scala, line 68: 'evaluatedCol' can be closed early.
/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 186: 'retTab' can be closed early.
/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 361: 'retTab' can be closed early.
/spark-rapids/sql-plugin/src/main/spark334/scala/com/nvidia/spark/rapids/shims/GetSequenceSize.scala, line 87: 'longMin' can be closed early.
/spark-rapids/sql-plugin/src/main/spark334/scala/com/nvidia/spark/rapids/shims/GetSequenceSize.scala, line 87: 'minusOne' can be closed early.
/spark-rapids/sql-plugin/src/main/spark334/scala/com/nvidia/spark/rapids/shims/GetSequenceSize.scala, line 87: 'stepEqualsMinusOne' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala, line 1553: 'listSizes' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala, line 1567: 'extractedWithNulls' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala, line 1970: 'stringsCol' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala, line 1970: 'keysValuesTable' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala, line 2302: 'intPart' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala, line 2302: 'reversedIntegerPart' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeExtractors.scala, line 153: 'zeroS' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 442: 'rhs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 442: 'castRhs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 442: 'castLhs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 611: 'zero' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 611: 'maxDecimal' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala, line 921: 'retTab' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeCreator.scala, line 162: 'deduped' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 616: 'child.isNull()' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 947: 'children' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 1370: 'zero' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 1370: 'counts' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 1396: 'rhs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 1396: 'lhs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 1396: 'boolMaskWithNans' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala, line 1424: 'lhsFlatten' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatDataWriter.scala, line 499: 'outCb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatDataWriter.scala, line 838: 'keysAndGroups' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatDataWriter.scala, line 876: 'keyHostCb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'stripped' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'lengths' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'zero' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'isEmpty' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'isNull' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'emptyRow' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'nullsReplaced' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'literalNull' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 141: 'isLiteralNull' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 154: 'newline' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 215: 'ds' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 215: 'table' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuJsonToStructs.scala, line 215: 'columns' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/predicates.scala, line 104: 'tbl' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/predicates.scala, line 104: 'leftTrueBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/GpuApproximatePercentile.scala, line 170: 'cv' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/GpuApproximatePercentile.scala, line 170: 'childView' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/aggregateFunctions.scala, line 286: 'hcv' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/aggregateFunctions.scala, line 286: 'partialN' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/aggregateFunctions.scala, line 650: 'dataCol' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/aggregateFunctions.scala, line 801: 'dataCol' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala, line 193: 'keys' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala, line 765: 'gatherVec' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala, line 890: 'scb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala, line 890: 'trackerBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuWindowInPandasExecBase.scala, line 162: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuWindowInPandasExecBase.scala, line 349: 'startVal' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/GpuWindowInPandasExecBase.scala, line 381: 'startVal' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/python/BatchGroupUtils.scala, line 580: 'keyBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/GpuHiveFileFormat.scala, line 266: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/GpuHiveFileFormat.scala, line 266: 'table' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/hive/rapids/GpuHiveTableScanExec.scala, line 602: 'regexFiltered' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/SortUtils.scala, line 385: 'toSort' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala, line 261: 'inputBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala, line 488: 'lastRow' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala, line 670: 'batchToGenerate' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala, line 670: 'src' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala, line 702: 'inputBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGenerateExec.scala, line 702: 'table' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala, line 287: 'splits' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/decimalExpressions.scala, line 119: 'maxScalar' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RmmRapidsRetryIterator.scala, line 701: 'src' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RmmRapidsRetryIterator.scala, line 701: 'tbl' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuMapUtils.scala, line 79: 'keyNotExist' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCoalesceBatches.scala, line 850: 'outputBuffers' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsDiskStore.scala, line 123: 'fc' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsDiskStore.scala, line 235: 'fis' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/conditionalExpressions.scala, line 310: 'tbl' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/conditionalExpressions.scala, line 310: 'trueBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/conditionalExpressions.scala, line 310: 'falseBatch' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/conditionalExpressions.scala, line 407: 'thenElseScalars' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuKeyBatchingIterator.scala, line 91: 'appended' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 381: 'childView' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 381: 'falseScalar' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 552: 'arg' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 552: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 552: 'newKeysCol' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 552: 'deduped' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala, line 628: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala, line 2190: 'combinedHmb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/JoinGatherer.scala, line 716: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuHashPartitioningBase.scala, line 83: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGetJsonObject.scala, line 265: 'input' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuMultiFileReader.scala, line 1157: 'buffer' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuTextBasedPartitionReader.scala, line 259: 'regexpFiltered' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuTextBasedPartitionReader.scala, line 523: 'stripped' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 228: 'max' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 396: 'half' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 396: 'doubleMillisPlusHalf' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 423: 'milliseconds' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 423: 'thousand' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 2346: 'combinedBuf' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala, line 2632: 'outhmb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/SamplingUtils.scala, line 216: 'spb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/basicPhysicalOperators.scala, line 1070: 'stepScalar' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/basicPhysicalOperators.scala, line 1070: 'startScalar' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 511: 'inputWithNansToNull' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 581: 'childView' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 745: 'micros' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 818: 'strChildCol' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 980: 'case' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1528: 'kvStructColumn' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1528: 'keyColumn' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1528: 'castKey' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1528: 'valueColumn' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1528: 'castValue' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1554: 'childColumns' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1565: 'dataCol' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala, line 1709: 'maxSecondsS' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/BufferSendState.scala, line 91: 'mtb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleClient.scala, line 318: 'transferReq' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleServer.scala, line 303: 'mtb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleServer.scala, line 439: 'bufferTx' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleServer.scala, line 439: 'buffersToSend' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/replay.scala, line 94: 'fs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/GpuLore.scala, line 99: 'fs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/GpuLore.scala, line 110: 'fs' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuUnboundedToUnboundedAggWindowExec.scala, line 255: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuUnboundedToUnboundedAggWindowExec.scala, line 353: 'rideGrpCB' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuUnboundedToUnboundedAggWindowExec.scala, line 353: 'aggResultsCB' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuUnboundedToUnboundedAggWindowExec.scala, line 946: 'cb' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuUnboundedToUnboundedAggWindowExec.scala, line 946: 'toRepeat' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuBatchedBoundedWindowExec.scala, line 111: 'freshBatchCB' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuBatchedBoundedWindowExec.scala, line 210: 'inputCbSpillable' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuBatchedBoundedWindowExec.scala, line 210: 'inputCB' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuCachedDoublePassWindowExec.scala, line 163: 'parts' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuCachedDoublePassWindowExec.scala, line 277: 'parts' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuWindowExpression.scala, line 1469: 'nulls' can be closed early.
/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/window/GpuWindowExpression.scala, line 1547: 'nulls' can be closed early.
/spark-rapids/sql-plugin/src/main/spark330/scala/org/apache/spark/sql/rapids/shims/intervalExpressions.scala, line 208: 'max' can be closed early.
/spark-rapids/sql-plugin/src/main/spark330/scala/org/apache/spark/sql/rapids/shims/intervalExpressions.scala, line 208: 'z' can be closed early.
/spark-rapids/sql-plugin/src/main/spark330/scala/org/apache/spark/sql/rapids/shims/intervalExpressions.scala, line 208: 'min' can be closed early.
/spark-rapids/sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala, line 126: 'microsPerSecondS' can be closed early.

Describe alternatives you've considered
Ideally the checker can be integrated into the build process, so that we can find such cases automatically. Maybe it should be based on some AST analysis and be a real static analysis tool, but I'm really not an expert in programming language.

@thirtiseven thirtiseven added ? - Needs Triage Need team to review and classify reliability Features to improve reliability or bugs that severly impact the reliability of the plugin labels Nov 12, 2024
@mattahrens mattahrens added good first issue Good for newcomers and removed ? - Needs Triage Need team to review and classify labels Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers reliability Features to improve reliability or bugs that severly impact the reliability of the plugin
Projects
None yet
Development

No branches or pull requests

2 participants