fix infinite loop in replace with AI collations #2849
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tanscorpio7
changed the title
Pull Request Test Coverage Report for Build 10385862370Details
💛 - Coveralls |
| /* ICU bug, When pattern start with a surrogate pair ICU usearch_next stops moving forward entering an infinite loop */ | ||
| if (u16_pos == pos_prev_loop) | ||
| { | ||
| if (U16_IS_SURROGATE(src_uchar[matched_idx]) && U16_IS_SURROGATE(substr_uchar[0]) && matched_idx + 2 < src_ulen) |
There was a problem hiding this comment.
Does the problem arises only when the first char is surrogate pair? I think better to use U16_IS_SURROGATE(substr_uchar[u16_pos]).
There was a problem hiding this comment.
Yes, it only happens when substring to find start with a surrogate pair.
In this case ICU doesn't seem to set the offset for the next search correctly.
| /* ICU bug, When pattern start with a surrogate pair ICU usearch_next stops moving forward entering an infinite loop */ | ||
| if (pos == pos_prev_loop) | ||
| { | ||
| if ( U16_IS_SURROGATE(src_uchar[matched_idx]) && U16_IS_SURROGATE(from_uchar[0]) && matched_idx + 2 < src_ulen) |
Deepesh125
requested changes
Aug 14, 2024
| @@ -1491,6 +1492,8 @@ pltsql_strpos_non_determinstic(text *src_text, text *substr_text, Oid collid, in | |||
| src_ulen = icu_to_uchar(&src_uchar, VARDATA_ANY(src_text), src_len_utf8); | |||
| substr_ulen = icu_to_uchar(&substr_uchar, VARDATA_ANY(substr_text), substr_len_utf8); | |||
|
|
|||
| is_substr_starts_with_surrogate = U16_IS_SURROGATE(substr_uchar[0]); | |||
There was a problem hiding this comment.
What if search str is empty?
There was a problem hiding this comment.
empty string never reaches pltsql_strpos_non_determinstic.
Even if it did we will error out before we reach here. Confirmed by setting values using GDB.
There was a problem hiding this comment.
We also have test cases for null inputs/empty string inputs in charindex, replace and patindex.
|
|
||
| if (is_substr_starts_with_surrogate && next_char_idx < src_ulen) | ||
| { | ||
| usearch_setOffset(usearch, next_char_idx, &status); |
There was a problem hiding this comment.
Do we need error checking?
tanscorpio7
force-pushed
the
BABEL_5169
branch
2 times, most recently
from
02e5141 to
2606dce
Compare
Signed-off-by: Tanzeel Khan <[email protected]>
tanscorpio7
force-pushed
the
BABEL_5169
branch
from
2606dce to
3589eef
Compare
Signed-off-by: Tanzeel Khan <[email protected]>
Signed-off-by: Tanzeel Khan <[email protected]>
Signed-off-by: Tanzeel Khan <[email protected]>
Deepesh125
approved these changes
Aug 14, 2024
Deepesh125
merged commit 4217dbf
into
babelfish-for-postgresql:BABEL_4_X_DEV
43 checks passed
tanscorpio7
added a commit
to tanscorpio7/babelfish_extensions
that referenced
this pull request
Aug 20, 2024
…esql#2849) ICU usearch_next() goes into infinite loop when pattern to search starts with a surrogate pair. To get around this we check if output of usearch_next() is stuck and not proceeding forwards and set the offset for next search ourselves. The next offset is simply the next character after the current char in source string. SRC STRING - 'abc🙂defghi🙂🙂' PATTERN TO FIND = '🙂def' usearch_next() gets stuck on "🙂" idx = 3 and repeatedly returns this index. We will intervene and set the offset to "d" idx = 4. So that usearch_next only starts looking from this character. Taks: BABEL-5167 Signed-off-by: Tanzeel Khan <[email protected]>
1 task
sharathbp
pushed a commit
to amazon-aurora/babelfish_extensions
that referenced
this pull request
Aug 20, 2024
…esql#2849) ICU usearch_next() goes into infinite loop when pattern to search starts with a surrogate pair. To get around this we check if output of usearch_next() is stuck and not proceeding forwards and set the offset for next search ourselves. The next offset is simply the next character after the current char in source string. SRC STRING - 'abc🙂defghi🙂🙂' PATTERN TO FIND = '🙂def' usearch_next() gets stuck on "🙂" idx = 3 and repeatedly returns this index. We will intervene and set the offset to "d" idx = 4. So that usearch_next only starts looking from this character. Taks: BABEL-5167 Signed-off-by: Tanzeel Khan <[email protected]>
jsudrik
pushed a commit
that referenced
this pull request
Aug 20, 2024
ICU usearch_next() goes into infinite loop when pattern to search starts with a surrogate pair. To get around this we check if output of usearch_next() is stuck and not proceeding forwards and set the offset for next search ourselves. The next offset is simply the next character after the current char in source string. SRC STRING - 'abc🙂defghi🙂🙂' PATTERN TO FIND = '🙂def' usearch_next() gets stuck on "🙂" idx = 3 and repeatedly returns this index. We will intervene and set the offset to "d" idx = 4. So that usearch_next only starts looking from this character. Taks: BABEL-5167 Signed-off-by: Tanzeel Khan <[email protected]>
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
ICU usearch_next() goes into infinite loop when pattern to search starts with a surrogate pair.
To get around this we check if output of usearch_next() is stuck and not proceeding forwards
and set the offset for next search ourselves.
The next offset is simply the next character after the current char in source string.
Issues Resolved
[BABEL-5169]
Sign Off
Signed-off-by: Tanzeel Khan [email protected]
Check List
By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.