Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filecache: Modify FileCacheLocalityManager.init to pass in Spark context #8531

Merged
merged 5 commits into from
Jun 28, 2023

Conversation

res-life
Copy link
Collaborator

@res-life res-life commented Jun 8, 2023

This is used by removing a executor form locations when the executor exits:

sc.listenerBus.addToSharedQueue(new SparkListener {

   override def onExecutorRemoved

})

We have to pass in the Spark context.
I tried to get by at the private repo.

SparkContext.getActive.get

Throws the following exception:

23/06/12 09:56:29 ERROR SparkContext: Error initializing SparkContext.
java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:529)
	at scala.None$.get(Option.scala:527)
	at org.apache.spark.sql.rapids.internal.TrampolineUtils$.getListenerBus(TrampolineUtils.scala:48)
	at com.nvidia.spark.rapids.filecache.FileCacheLocalityManager$.init(FileCacheLocalityManager.scala:51)

@res-life res-life force-pushed the filecache-handle-executor-exit branch from bd8bb37 to b585361 Compare June 12, 2023 11:31
@res-life res-life changed the title [WIP] filecache: remove all the locality infomation of a executor when the executor exits [WIP] filecache: Modify FileCacheLocalityManager.init to pass in Spark context Jun 12, 2023
@sameerz sameerz added the task Work required that improves the product but is not user facing label Jun 19, 2023
@res-life res-life marked this pull request as ready for review June 20, 2023 14:20
jlowe
jlowe previously approved these changes Jun 20, 2023
Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK, but we may not need to pass the SparkConf explicitly since it should be trivial to access via getConf on the new SparkContext parameter.

@res-life
Copy link
Collaborator Author

Done. Removed SparkConf from parameters.

@res-life
Copy link
Collaborator Author

build

@jlowe jlowe changed the title [WIP] filecache: Modify FileCacheLocalityManager.init to pass in Spark context filecache: Modify FileCacheLocalityManager.init to pass in Spark context Jun 28, 2023
@jlowe jlowe merged commit d36612e into NVIDIA:branch-23.08 Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants