Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Possible improvements to Qualification tool #73

Closed
tgravescs opened this issue Jan 27, 2023 · 1 comment
Closed

[FEA] Possible improvements to Qualification tool #73

tgravescs opened this issue Jan 27, 2023 · 1 comment
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@tgravescs
Copy link
Collaborator

tgravescs commented Jan 27, 2023

Is your feature request related to a problem? Please describe.

Here are a few things I've found during migrating some jobs we may want to look at more to see if Qualification tool can give us hints about. ie perhaps configs to use or things for user to look for.

  • look for explicit settings of shuffle.partitions and max partition bytes... others? I had a case CPU side has shuffle partitions set and that job was spilling which caused slowness in transitioning to GPU
  • Look for spilling
  • look for heavily skewed tasks
  • summarize IO to total bytes intput/output/shuffle read remote/shuffle read local/shuffle write/spilled
@amahussein
Copy link
Collaborator

duplicate of #1152

@amahussein amahussein closed this as not planned Won't fix, can't repro, duplicate, stale Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants