Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#2674] fix(jdbc backend): Save millisecond timestamp on mysql soft delete #2685

Merged
merged 2 commits into from
Mar 27, 2024

Conversation

xloya
Copy link
Contributor

@xloya xloya commented Mar 26, 2024

What changes were proposed in this pull request?

The current way of saving second-level timestamps will cause multiple deletion operations fails in the same second because of the duplicate key. This pr will save millisecond timestamp.

Why are the changes needed?

Fix: #2674

@xloya
Copy link
Contributor Author

xloya commented Mar 26, 2024

@yuqi1129 Could you help test this is worked in integration tests? Thanks.

@xloya xloya closed this Mar 26, 2024
@xloya xloya reopened this Mar 26, 2024
@yuqi1129
Copy link
Contributor

@xloya
it works for MetalakeIT, what concerns me is whether this is the right way to solve the problem. Milliseconds are sufficient for most cases, but can we use a larger value to completely avoid the problem?

like current_milliseconds << 18 + auto_increment_value_within_2^17

@xloya
Copy link
Contributor Author

xloya commented Mar 26, 2024

@yuqi1129 I agree that enlarging the value avoids the problem as much as possible.
I have seen some implementations of soft deletion. Some use the Datetime type, which has second-level precision; some use the Timestamp type, which has microsecond-level precision. I haven't seen any implementation of soft deletion using higher precision yet. Maybe a timestamp at the millisecond or microsecond level is enough.
In addition, we need to consider whether multiple databases support the same time function. The current implementation has considered the compatibility of H2 and MySQL time functions. I am not sure if there is a function that can implement the logic you mentioned.

Copy link
Contributor

@yuqi1129 yuqi1129 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yuqi1129 yuqi1129 merged commit 66addc7 into apache:main Mar 27, 2024
7 checks passed
@xloya xloya deleted the fix-mysql-soft-delete branch June 20, 2024 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug report] Duplicate entry 'metalakeA_45a84d84-1711373100' for key 'metalake_meta.uk_mn_del'
2 participants