Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split job and stage level aggregated metrics into different files #1050

Merged
merged 3 commits into from
May 30, 2024
Merged

Split job and stage level aggregated metrics into different files #1050

merged 3 commits into from
May 30, 2024

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented May 30, 2024 •
edited
Loading

Fixes #1017. Please see issue description for more details about sub tasks.

Changes:

  1. Splits the job_+_stage_level_aggregated_task_metrics.csv generated by the Profiling tool into two separate files.
  2. Update spill heuristics method to use the new stage level aggregated metrics.
  3. Updated prediction code to handle the separate metrics. This will be refactored later when updating the code with latest changes.

@parthosa
Split JobStageAggTaskMetrics file into two different files (#1044)
11625d0 
* Split job and stage aggregate metrics

Signed-off-by: Partho Sarthi <[email protected]>

* Update spill heuristics to read correct files

Signed-off-by: Partho Sarthi <[email protected]>

---------

Signed-off-by: Partho Sarthi <[email protected]>
@parthosa
Update prediction code to use separate job and stage level aggregates (…
6dff2e9 
…#1047)

* Update prediction code to use separate job and stage level aggregates

Signed-off-by: Partho Sarthi <[email protected]>

* Refactor to move inside condition

Signed-off-by: Partho Sarthi <[email protected]>

* Fix typo

Signed-off-by: Partho Sarthi <[email protected]>

---------

Signed-off-by: Partho Sarthi <[email protected]>
@parthosa parthosa added feature request New feature or request user_tools Scope the wrapper module running CSP, QualX, and reports (python) core_tools Scope the core module (scala) labels May 30, 2024
@parthosa parthosa requested a review from amahussein May 30, 2024 19:01
@parthosa parthosa self-assigned this May 30, 2024
@parthosa parthosa changed the title Split JobStageAggTaskMetrics file into two different files Split job and stage level aggregated metrics into different files May 30, 2024
@parthosa parthosa mentioned this pull request May 30, 2024
Merged
amahussein
amahussein approved these changes May 30, 2024
View reviewed changes
Copy link
Collaborator

@amahussein amahussein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa
LGTME

@amahussein amahussein merged commit 3dc9446 into dev May 30, 2024
15 checks passed
@parthosa parthosa deleted the spark-rapids-tools-1017-staging branch May 30, 2024 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amahussein amahussein amahussein approved these changes

Assignees

@parthosa parthosa

Labels
core_tools Scope the core module (scala) feature request New feature or request user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Split JobStageAggTaskMetrics file into two different files
2 participants
@parthosa @amahussein