Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Photon-specific SQL Metrics #1390

Merged
merged 5 commits into from
Oct 25, 2024

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented Oct 23, 2024

Fixes #1388

This PR adds support for Photon-specific SQL metrics as:

  1. cumulative time metric that can be used as a replacement for the scan time metric.
  2. shuffle write time can be reconstructed using the following metrics:
    1. time taken waiting on file write IO (part of shuffle file write)
    2. time taken to sort rows by partition ID (part of shuffle file write)
    3. time taken to convert columns to rows (part of shuffle file write)
  3. peak memory usage metric can be used for the peak execution memory metric.

Code Changes

Tests:

  • Added test case to verify calculation of SQL metrics for Photon applications. [1] [2]

@parthosa parthosa added bug Something isn't working core_tools Scope the core module (scala) labels Oct 23, 2024
@parthosa parthosa self-assigned this Oct 23, 2024
@parthosa parthosa marked this pull request as ready for review October 23, 2024 20:35
amahussein
amahussein previously approved these changes Oct 23, 2024
Copy link
Collaborator

@amahussein amahussein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa
just a small nit

cindyyuanjiang
cindyyuanjiang previously approved these changes Oct 23, 2024
Copy link
Collaborator

@cindyyuanjiang cindyyuanjiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa! a minor nit.

Copy link
Collaborator

@cindyyuanjiang cindyyuanjiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa! LGTM.

Copy link
Collaborator

@nartal1 nartal1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa ! LGTM.

@parthosa parthosa merged commit 1504968 into NVIDIA:dev Oct 25, 2024
14 checks passed
@parthosa parthosa deleted the spark-rapids-tools-1388 branch October 25, 2024 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Missing Metrics in Photon Event Logs Affecting QualX Predictions
4 participants