Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] NDS query 64 falls back to CPU only for a shuffle #9092

Closed
jlowe opened this issue Aug 22, 2023 · 2 comments · Fixed by #9134
Closed

[BUG] NDS query 64 falls back to CPU only for a shuffle #9092

jlowe opened this issue Aug 22, 2023 · 2 comments · Fixed by #9134
Assignees
Labels
bug Something isn't working performance A performance related task/issue

Comments

@jlowe
Copy link
Member

jlowe commented Aug 22, 2023

When running NDS query 64 at the 3k scale factor on AWS EMR 6.12, there is an unnecessary fallback to the CPU for a shuffle. The node sequence in the plan UI is:

GpuHashAggregate(partial) ->
  GpuColumnarToRow ->
    AdaptiveSparkPlan ->
      Exchange ->
        ReusedExchange ->
          AQEShuffleRead ->
            GpuRowToColumnar ->
              GpuHashAggregate(final)
@jlowe jlowe added bug Something isn't working ? - Needs Triage Need team to review and classify performance A performance related task/issue labels Aug 22, 2023
@sameerz
Copy link
Collaborator

sameerz commented Aug 22, 2023

Need to check for this in an on-premises cluster. There were no notifications in the driver log.

Marking as high priority for the interim while we investigate.

@jlowe
Copy link
Member Author

jlowe commented Aug 25, 2023

This is also happening in NDS query 23_part1, 23_part2, 24_part1, 24_part2, 47, and 57.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working performance A performance related task/issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants