Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] The result of casting string to decimal is not identical to Apache Spark #2019

Closed
sperlingxx opened this issue Mar 25, 2021 · 3 comments · Fixed by #4081
Closed

[BUG] The result of casting string to decimal is not identical to Apache Spark #2019

sperlingxx opened this issue Mar 25, 2021 · 3 comments · Fixed by #4081
Assignees
Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf

Comments

@sperlingxx
Copy link
Collaborator

sperlingxx commented Mar 25, 2021

Describe the bug
Currently string to decimal type on the GPU might produce results which slightly differed from the correct results when the string
represents any number exceeding the max precision that the operation CAST_STRING_TO_FLOAT can keep. For instance, the GPU returns 99999999999999987 for input string "99999999999999999".

The cause of divergence is that we can not directly cast strings containing scientific notation to decimal via cuDF API. So, we have to cast strings to floats firstly. Then, cast floats to decimals. The first step may lead to precision loss.

Steps/Code to reproduce bug
Here are samples to produce inconsistent results:

  • cast("99999999999999999" to Decimal(18, 0))
    • 99999999999999999(CPU) vs 99999999999999987(GPU)
  • cast("999999999999999" to Decimal(18, -2))
    • 999999999999999.00(CPU) vs 999999999999999.10(GPU)

Expected behavior
The GPU runtime can produce exact same results as Apache Spark.

Additional context
Current issue is related to issue #1625 and PR #1999.

@sperlingxx sperlingxx added bug Something isn't working ? - Needs Triage Need team to review and classify labels Mar 25, 2021
@revans2 revans2 added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Mar 25, 2021
@revans2
Copy link
Collaborator

revans2 commented Mar 25, 2021

Does this depend on rapidsai/cudf#7665

@sameerz
Copy link
Collaborator

sameerz commented Nov 22, 2021

@razajafri @sperlingxx is this resolved since PR #4081 is closed?

@razajafri
Copy link
Collaborator

Yes, the PR even has fixes 2019 in the description, it should've been autoclosed, no?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cudf_dependency An issue or PR with this label depends on a new feature in cudf
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants