Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: Number without scale in Oracle vs bigint in Databricks - Reconcilation is treating all records as "mismatch" #1202

Open
1 task done
Tim-Chowdary opened this issue Nov 13, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@Tim-Chowdary
Copy link

Tim-Chowdary commented Nov 13, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Category of Bug / Issue

ReconcileError

Current Behavior

Problem Statement: When a column with a NUMBER data type (without scale) is involved in a join, mismatched records (both data and record count) are not displayed on the dashboard.
Root Cause: In Oracle, a NUMBER without scale is treated as a decimal, which causes the join criteria to fail, resulting in empty outputs.

Note: By commenting the transformation rule to handle nulls in joins, we were able to display the total number of mismatch records. However, mismatch data is not shown in the detail window.

This is really to not generate false negatives during hash comparisons

 select a ,b, a=b, hash(a), hash(b), hash(a)=hash(b) 
    from (select cast(21.00 AS decimal(38,0))as a, cast(21 AS int) as b);
@Tim-Chowdary Tim-Chowdary added the bug Something isn't working label Nov 13, 2024
@sundarshankar89 sundarshankar89 changed the title [BUG]: [DSG] Number without scale in Oracle vs bigint in Databricks - Reconcilation is treating all records as "mismatch" [BUG]: Number without scale in Oracle vs bigint in Databricks - Reconcilation is treating all records as "mismatch" Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant