fix conversion of varchar to binary varbinary and vice versa #1957

tanscorpio7 · 2023-10-26T04:49:38Z

Description

Binary data type simply stores hex codes but when transformed to and from varchar, use of correct encoding becomes important. For example the symbol '™' is stored as
0xE284A2 in UTF-8 enc. while
0x99 in Win encoding
From users perspective they must see the hex value which is congruent to their server encoding.

To fix this we do necessary encoding to source data in varchar <--> varbinary internal functions.
Length checks should be done when data (hex string) is in server encoding.

We must also handle the case for string literal to binary types. So we explicitly call the varcharvarbinary conversion when a string literal is being casted to binary data type.

Also added CAST functions for sys.BBF_VARBINARY to sys.BBF_BINARY and vice versa
these cast are being used in geography and geometry cast functions
made necessary changes to geography and geometry casts as well
i.e. CAST (CAST ($1 AS sys.VARCHAR) AS sys.bbf_varbinary) --> CAST ($1 AS sys.bbf_varbinary)

Engine PR: babelfish-for-postgresql/postgresql_modified_for_babelfish#248

Extension PR: #1957

Issues Resolved

[BABEL-1940]

Sign-off

Signed-off-by: Tanzeel Khan [email protected]
Co-authored-by: Rohit Bhagat [email protected]

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tanzeel Khan <[email protected]>

sumitj824 · 2023-10-26T09:20:59Z

Please add tests.

Signed-off-by: Tanzeel Khan <[email protected]>

contrib/babelfishpg_common/src/varbinary.c

test/JDBC/expected/BABEL_1940.out

Signed-off-by: Tanzeel Khan <[email protected]>

contrib/babelfishpg_common/src/encoding/mb/conv.c

contrib/babelfishpg_tsql/src/pltsql_coerce.c

test/JDBC/input/babel_function_string-before-15-5-or-14-10-vu-verify.sql

Signed-off-by: Tanzeel Khan <[email protected]>

test/JDBC/expected/BABEL_1940.out

rohit01010 · 2023-11-03T12:02:26Z

1> SELECT CONVERT(VARBINARY(10), CONVERT(VARCHAR(10), 0x330033))
2> go
Msg 33557097, Level 16, State 1, Server BABELFISH, Line 1
invalid byte sequence for encoding "UTF8": 0x00

Getting this above error with the changes from this PR, Kindly resolve and add this query to test-cases

Signed-off-by: Tanzeel Khan <[email protected]>

contrib/babelfishpg_common/src/varbinary.c

test/JDBC/expected/BABEL_1940.out

Signed-off-by: Tanzeel Khan <[email protected]>

contrib/babelfishpg_common/src/varbinary.c

Signed-off-by: Tanzeel Khan <[email protected]>

Release syscache reference when returning not null nodes from string literal hook to avoid cache reference leak Engine PR: #248 Extension PR: babelfish-for-postgresql/babelfish_extensions#1957 Task: 1940 Signed-off-by: Tanzeel Khan <[email protected]>

…sh-for-postgresql#248) Release syscache reference when returning not null nodes from string literal hook to avoid cache reference leak Engine PR: babelfish-for-postgresql#248 Extension PR: babelfish-for-postgresql/babelfish_extensions#1957 Task: 1940 Signed-off-by: Tanzeel Khan <[email protected]>

Release syscache reference when returning not null nodes from string literal hook to avoid cache reference leak Engine PR: #248 Extension PR: babelfish-for-postgresql/babelfish_extensions#1957 Task: 1940 Signed-off-by: Tanzeel Khan <[email protected]>

…sh-for-postgresql#248) Release syscache reference when returning not null nodes from string literal hook to avoid cache reference leak Engine PR: babelfish-for-postgresql#248 Extension PR: babelfish-for-postgresql/babelfish_extensions#1957 Task: 1940 Signed-off-by: Tanzeel Khan <[email protected]>

KushaalShroff · 2023-11-28T06:39:08Z

contrib/babelfishpg_common/sql/binary.sql

+CREATE CAST (sys.BBF_BINARY AS sys.BBF_VARBINARY)
+WITHOUT FUNCTION AS IMPLICIT;
+
+CREATE CAST (sys.BBF_VARBINARY AS sys.BBF_BINARY)
+WITHOUT FUNCTION AS IMPLICIT;
+


No upgrade script changes for these?

…sh-for-postgresql#248) Release syscache reference when returning not null nodes from string literal hook to avoid cache reference leak Engine PR: babelfish-for-postgresql#248 Extension PR: babelfish-for-postgresql/babelfish_extensions#1957 Task: 1940 Signed-off-by: Tanzeel Khan <[email protected]>

…sh-for-postgresql#1957) Binary data type simply stores hex codes but when transformed to and from varchar, use of correct encoding becomes important when we cast to and from string data types. For example the symbol '™' is stored as 0xE284A2 in UTF-8 encoding while 0x99 in WIN1252 encoding. From users perspective they must see the hex value which is congruent to their server encoding. To fix this we do necessary encoding to source data in varchar <--> varbinary internal functions. Length checks should be done when data (hex string) is in server encoding. We must also handle the case for string literal to binary types. So we explicitly call the varcharvarbinary conversion when a string literal is being casted to binary data type. Also added CAST functions for sys.BBF_VARBINARY to sys.BBF_BINARY and vice versa. These cast are being used in geography and geometry data types cast functions. Made necessary changes to geography and geometry casts as well i.e. CAST (CAST ($1 AS sys.VARCHAR) AS sys.bbf_varbinary) --> CAST ($1 AS sys.bbf_varbinary) Engine PR: babelfish-for-postgresql/postgresql_modified_for_babelfish#248 Extension PR: babelfish-for-postgresql#1957 Task: BABEL-1940 Signed-off-by: Tanzeel Khan [email protected] Co-authored-by: Rohit Bhagat [email protected]

tanscorpio7 added 3 commits October 26, 2023 04:43

initial commit

982793f

Signed-off-by: Tanzeel Khan <[email protected]>

fix indent

745aaeb

Signed-off-by: Tanzeel Khan <[email protected]>

allow 0x00 when encoding to UTF-8

d9391a2

Signed-off-by: Tanzeel Khan <[email protected]>

tanscorpio7 requested review from Deepesh125 and sumitj824 October 26, 2023 09:16

tanscorpio7 added 2 commits October 26, 2023 14:00

add tests

cd3bbfc

Signed-off-by: Tanzeel Khan <[email protected]>

correct upgrade tests for babel_function_string

d2db3fb

Signed-off-by: Tanzeel Khan <[email protected]>

Deepesh125 reviewed Oct 27, 2023

View reviewed changes

contrib/babelfishpg_common/src/varbinary.c Outdated Show resolved Hide resolved

Deepesh125 reviewed Oct 27, 2023

View reviewed changes

test/JDBC/expected/BABEL_1940.out Outdated Show resolved Hide resolved

sumitj824 reviewed Oct 27, 2023

View reviewed changes

test/JDBC/expected/BABEL_1940.out Outdated Show resolved Hide resolved

update tests

3011a28

Signed-off-by: Tanzeel Khan <[email protected]>

Deepesh125 requested changes Nov 2, 2023

View reviewed changes

tanscorpio7 added 3 commits November 2, 2023 14:38

Merge branch 'BABEL_3_X_DEV' into BABEL_1940

a8ddfae

use correct func to build typename

c05919d

Signed-off-by: Tanzeel Khan <[email protected]>

empty commit rerun actions

7f9da36

Signed-off-by: Tanzeel Khan <[email protected]>

tanscorpio7 requested a review from Deepesh125 November 2, 2023 16:25

Deepesh125 requested review from rohit01010 and sumitj824 November 3, 2023 07:15

release syscache tuples

e033ba4

Signed-off-by: Tanzeel Khan <[email protected]>

tanscorpio7 mentioned this pull request Nov 3, 2023

release sys cache reference after string literal hook babelfish-for-postgresql/postgresql_modified_for_babelfish#248

Merged

1 task

tanscorpio7 added 3 commits November 3, 2023 09:30

empty commit rerun actions post engine changes

e948fd0

Signed-off-by: Tanzeel Khan <[email protected]>

rerun actions with empty commit

cdd4e35

Signed-off-by: Tanzeel Khan <[email protected]>

add few more tests

62729cf

Signed-off-by: Tanzeel Khan <[email protected]>

Deepesh125 reviewed Nov 3, 2023

View reviewed changes

test/JDBC/expected/BABEL_1940.out Outdated Show resolved Hide resolved

tanscorpio7 added 4 commits November 7, 2023 09:41

rerun actions with empty commit

a193211

Signed-off-by: Tanzeel Khan <[email protected]>

Merge remote-tracking branch 'upstream/BABEL_3_X_DEV' into BABEL_1940

d2e85f6

empty commit

d16c126

Signed-off-by: Tanzeel Khan <[email protected]>

reject null byte

5f4507c

Signed-off-by: Tanzeel Khan <[email protected]>

Deepesh125 reviewed Nov 9, 2023

View reviewed changes

contrib/babelfishpg_common/src/varbinary.c Show resolved Hide resolved

Deepesh125 requested changes Nov 9, 2023

View reviewed changes

contrib/babelfishpg_common/src/varbinary.c Show resolved Hide resolved

contrib/babelfishpg_common/src/varbinary.c Outdated Show resolved Hide resolved

test/JDBC/expected/BABEL_1940.out Show resolved Hide resolved

tanscorpio7 added 5 commits November 9, 2023 20:08

fix typmod for binary data type

25fda37

Signed-off-by: Tanzeel Khan <[email protected]>

remove typmod fix for now

b768e82

Signed-off-by: Tanzeel Khan <[email protected]>

push correct test out file

d944eb0

Signed-off-by: Tanzeel Khan <[email protected]>

just comments

f2e7f81

Signed-off-by: Tanzeel Khan <[email protected]>

add some varbinary binary test case

278c0d5

Signed-off-by: Tanzeel Khan <[email protected]>

Deepesh125 previously approved these changes Nov 10, 2023

View reviewed changes

contrib/babelfishpg_common/src/varbinary.c Outdated Show resolved Hide resolved

contrib/babelfishpg_common/src/varbinary.c Outdated Show resolved Hide resolved

some minor refactoring

0bbf3c8

Signed-off-by: Tanzeel Khan <[email protected]>

tanscorpio7 dismissed Deepesh125’s stale review via 0bbf3c8 November 10, 2023 07:44

tanscorpio7 changed the title ~~Varchar Varbinary inter conversion~~ fix conversion of varchar to binary varbinary and vice versa Nov 10, 2023

tanscorpio7 requested a review from Deepesh125 November 10, 2023 09:10

Deepesh125 approved these changes Nov 10, 2023

View reviewed changes

tanscorpio7 changed the title ~~fix conversion of varchar to binary varbinary and vice versa~~ proper release sys cache reference after string literal hook Nov 10, 2023

tanscorpio7 changed the title ~~proper release sys cache reference after string literal hook~~ release sys cache reference after string literal hook Nov 10, 2023

tanscorpio7 changed the title ~~release sys cache reference after string literal hook~~ fix conversion of varchar to binary varbinary and vice versa Nov 10, 2023

Deepesh125 merged commit 62546c5 into babelfish-for-postgresql:BABEL_3_X_DEV Nov 10, 2023
28 checks passed

tanscorpio7 mentioned this pull request Nov 10, 2023

release sys cache reference after string literal hook babelfish-for-postgresql/postgresql_modified_for_babelfish#254

Merged

1 task

tanscorpio7 mentioned this pull request Nov 10, 2023

fix upgrade script for new bbf_binary casts #2010

Merged

1 task

KushaalShroff reviewed Nov 28, 2023

View reviewed changes

tanscorpio7 deleted the BABEL_1940 branch December 12, 2023 08:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix conversion of varchar to binary varbinary and vice versa #1957

fix conversion of varchar to binary varbinary and vice versa #1957

tanscorpio7 commented Oct 26, 2023 •

edited

Loading

sumitj824 commented Oct 26, 2023

rohit01010 commented Nov 3, 2023

KushaalShroff Nov 28, 2023

fix conversion of varchar to binary varbinary and vice versa #1957

fix conversion of varchar to binary varbinary and vice versa #1957

Conversation

tanscorpio7 commented Oct 26, 2023 • edited Loading

Description

Engine PR: babelfish-for-postgresql/postgresql_modified_for_babelfish#248

Extension PR: #1957

Issues Resolved

Sign-off

Check List

sumitj824 commented Oct 26, 2023

rohit01010 commented Nov 3, 2023

KushaalShroff Nov 28, 2023

Choose a reason for hiding this comment

tanscorpio7 commented Oct 26, 2023 •

edited

Loading