[SPARK-12130] Replace shuffleManagerClass with shortShuffleMgrNames in ExternalShuffleBlockResolver #10131
Conversation
| @@ -200,7 +200,7 @@ private[spark] class BlockManager( | |||
| val shuffleConfig = new ExecutorShuffleInfo( | |||
| diskBlockManager.localDirs.map(_.toString), | |||
| diskBlockManager.subDirsPerLocalDir, | |||
| shuffleManager.getClass.getName) | |||
| conf.get("spark.shuffle.manager", "sort").toLowerCase) | |||
There was a problem hiding this comment.
looks OK, but why not add a field in ShuffleManager itself that sets this? So you can just do shuffleManager.name or something here
|
@andrewor14 Now I add shortShuffleMgrName to ShuffleManager for getting its short Name. How do you think? Thanks. |
|
Test build #47197 has finished for PR 10131 at commit
|
| @@ -60,6 +60,9 @@ private[spark] trait ShuffleManager { | |||
| */ | |||
| def shuffleBlockResolver: ShuffleBlockResolver | |||
|
|
|||
| /** Return short name for the ShuffleManager */ | |||
| def shortShuffleMgrName: String | |||
There was a problem hiding this comment.
I would make these vals since we don't ever expect them to change within a shuffle manager
|
Minor comment but this looks good otherwise. |
|
@andrewor14 thanks. I think now it's ok. |
|
Test build #47710 has finished for PR 10131 at commit
|
|
Test build #47711 has finished for PR 10131 at commit
|
|
Test build #47716 has finished for PR 10131 at commit
|
| @@ -28,6 +28,10 @@ import org.apache.spark.{TaskContext, ShuffleDependency} | |||
| * boolean isDriver as parameters. | |||
| */ | |||
| private[spark] trait ShuffleManager { | |||
|
|
|||
| /** Return short name for the ShuffleManager */ | |||
| val shortShuffleMgrName: String | |||
There was a problem hiding this comment.
just call this shortName
|
LGTM, merging into master. I'll fix the comment on merge. |
|
@andrewor14 OK, Thanks. |
Replace shuffleManagerClassName with shortShuffleMgrName is to reduce time of string's comparison. and put sort's comparison on the front. cc @JoshRosen @andrewor14