Fix check write for spark 3.5 unit tests #6793
Conversation
| assert(write.isInstanceOf[InsertIntoHadoopFsRelationCommand]) | ||
| checkSort(write.query) | ||
| } | ||
| // TODO: CreateDataSourceTableAsSelectCommand is not supported |
There was a problem hiding this comment.
There was a problem hiding this comment.
why does CreateDataSourceTableAsSelectCommandnot support ? CreateDataSourceTableAsSelectCommand should call InsertIntoHadoopFsRelationCommand internally.
There was a problem hiding this comment.
InsertIntoHadoopFsRelationCommand's catalogTable will be empty
There was a problem hiding this comment.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6793 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 687 687
Lines 42442 42442
Branches 5793 5793
======================================
Misses 42442 42442 ☔ View full report in Codecov by Sentry. |
| def collect(p: LogicalPlan): Seq[RebalancePartitions] = { | ||
| p.flatMap { | ||
| case r: RebalancePartitions => Seq(r) | ||
| case s: LogicalQueryStage => collect(s.logicalPlan) |
There was a problem hiding this comment.
why do we need to match LogicalQueryStage ?
There was a problem hiding this comment.
This test case gets a logical plan like this:
There was a problem hiding this comment.
Is it a Spark issue ? IMO, LogicalQueryStage should not exist in query execution.
There was a problem hiding this comment.
I guess it's because the logical plan is obtained by write.cmd
🔍 Description
Issue References 🔗
KyuubiSparkSQLExtensionTest#withListenerin Spark 3.5 may not work, make the following adjustments:AdaptiveSparkPlanExecwhen collecting writescallbackfunction in main thread to avoid assert failed not workingcollectRebalancePartitionsmethoadDescribe Your Solution 🔧
Types of changes 🔖
Test Plan 🧪
Behavior Without This Pull Request ⚰️
Behavior With This Pull Request 🎉
Related Unit Tests
Checklist 📝
Be nice. Be informative.