Fixed error code difference occurring due to parallel worker enforced #260

Deepesh125 · 2023-11-21T13:05:15Z

Description

When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error.

In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error
message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context
about Babelfish

Issues Resolved

BABEL-4539

Check List

Existing JDBC test cases should be sufficient to test these changes.
Also, regression test suite provided by vanilla Postgres runs fine with these changes.

Signed-off-by: Dipesh Dhameliya [email protected]

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the PostgreSQL license, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.

surendravishnoi · 2023-11-24T09:55:39Z

src/backend/libpq/pqmq.c

+				}
+				else
+				{
+					elog(ERROR, "Unexpected message_id found");


why are we giving error for APG here? also does this code applies to clients as well? they are not aware of SQL dialect

For PG anyway we were throwing elog(ERROR, "unrecognized error field code: %d", (int) code); before

which clients are you referring to? Client here would be leader node (not psql that you might be referring here)

we should validate here as well that this is indeed in context of parallel worker (I am not sure if that is even possible). Also, code treats message id as static strings. If we are using str dup, we need to make sure that assumption is not broken when we simply assign pointers.

surendravishnoi · 2023-11-24T09:56:42Z

src/backend/utils/error/elog.c

@@ -1703,6 +1704,8 @@ ThrowErrorData(ErrorData *edata)
 	if (edata->backtrace)
 		newedata->backtrace = pstrdup(edata->backtrace);
 	/* assume message_id is not available */
+	if (edata->message_id)


why this change? message id is global at-least for given postgres worker

message_id is not global, its part of edata

message ids are static strings, so there is one copy at process level. that is why we do not copy them, assigning pointer is sufficient. What is reasoning behind this change?

First copy is happening inside temporary context (which will be resetted at the end of handling parallel message) and second copy is happening inside error context which would be preserved at process level and will be used everywhere for error manipulation. So there is no harm in keeping this.

surendravishnoi · 2023-11-24T10:07:22Z

src/backend/utils/error/elog.c

@@ -3250,6 +3253,15 @@ send_message_to_frontend(ErrorData *edata)
 			err_sendstring(&msgbuf, edata->funcname);
 		}

+		if (sql_dialect == SQL_DIALECT_TSQL)
+		{
+			if (edata->message_id)


how do we make sure that all clients will be able to handle additional message id in error messgage

I think I will add another check (IsParallelWorker()) here to ensure that only parallel worker sends this message_id and client (leader node in this case) will be able to handle it through dialect.

are we sure that parallel workers have consistent mapping with respect to dialect with leader node? what if error happens inside a procedure executed with PG dialect or vice versa and leader has different dialect?

As discussed offline, we can keep this message_id irrespective of dialect.

…ira-babel-4359

…is Babelfish

…ira-babel-4359

surendravishnoi · 2023-12-18T14:49:40Z

src/include/libpq/libpq-be.h

@@ -250,6 +250,7 @@ typedef struct Port
 	SSL		   *ssl;
 	X509	   *peer;
 #endif
+	bool		is_tds_conn;


this information is available on TSQL side, we should add hook rather than adding a new member here.

We rather keep it as it is for some other reasons.

surendravishnoi · 2023-12-18T14:52:12Z

src/backend/access/transam/parallel.c

@@ -101,6 +101,9 @@ typedef struct FixedParallelState

 	/* Maximum XactLastRecEnd of any worker. */
 	XLogRecPtr	last_xlog_end;
+
+	/* Track if parallel worker is spawned in the context of Babelfish */
+	bool babelfish_context;


if this is not in the context of mutex, move it before mutex

Thats good point, updated accordingly. Thanks

With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…babelfish-for-postgresql#260) When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error. In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context about Babelfish Extension changes: babelfish-for-postgresql/babelfish_extensions#2047 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…l#2047) With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…babelfish-for-postgresql#260) When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error. In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context about Babelfish Extension changes: babelfish-for-postgresql/babelfish_extensions#2047 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…l#2047) With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…babelfish-for-postgresql#260) When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error. In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context about Babelfish Extension changes: babelfish-for-postgresql/babelfish_extensions#2047 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…l#2047) With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

@key

* TSQL connection ps status always shows idle. (babelfish-for-postgresql#2107) This causes confusion because it would appear as if there is a memory leak issue because ps status shows idle while memory usage increases. This commit fixes that issue and will set the ps status to active when it is really active. PG engine code resets it back to idle. Task: BABEL-4604 Signed-off-by: Kristian Lejao <[email protected]> * Run MVU tests in release mode (babelfish-for-postgresql#2113) Task: BABEL-4449 Signed-off-by: Nirmit Shah <[email protected]> * Create and use composite actions for code coverage workflow to improve maintainability (babelfish-for-postgresql#2100) Github actions for running jdbc, odbc, python and dotnet were copied from the respective workflow files into code-coverage workflow file. Now any changes in the original action demands a similar change in the code coverage file as well increasing redundant effort. To improve maintainability we move these actions to new composite actions and reuse them in workflows. Also added a new optional variable in build-modified-postgres/action.yml to optionally build postgres with the enable code coverage flag. Issues Resolved [BABEL-4605] Signed-off-by: Tanzeel Khan [email protected] * Fix the issue that inserted table refered in update join will cause crash (babelfish-for-postgresql#2140) Previously when we implement update join, we missed a corner case that one of the join relations can be a named tuple store instead of RTE relation. And it'll lead analyzer to wrongly set resultRelation for the Query. This fix resolved the corner case by add a if condition to exclude RTE_NAMEDTUPLESTORE for being resultRelation. Task: BABEL-4606 Signed-off-by: Zhibai Song <[email protected]> * Fix system procedure sp_pkeys which may fail to return any row if parallel mode is enforced (babelfish-for-postgresql#2147) It is Postgres peculiarity with Hash cond (with operator text = name and text = name) that hash functions being used to hash text and name data type values are totally different, for example, hashtext would consider explicit collation whereas hashname does not care about explicit collation. That is the reason why hash condition would fail every time. (hashtext and hashname may produce different hash value for same input under collation other than "C"). This means that whenever there is such hash condition for Hash join then it would always fail and will not return any row. And optimiser can choose such hash join based on stats. Such issue is observed with helper view sys.sp_pkeys_view which is being used by system procedure sp_pkeys causing procedure to return zero rows when optimiser chooses certain hash join. This commit fixes such issue by casting name data type to appropriate T-SQL data type. Task: BABEL-4603 Signed-off-by: Dipesh Dhameliya [email protected] * Add Support for Uploading Coredumps in Github Actions (babelfish-for-postgresql#2148) In case of crash during the Tests run , if there is a server crash this action will create the text backtrace as well as upload the corefile as an artifact of Github action run. Example of workflow where coredump generated -https://github.com/babelfish-for-postgresql/babelfish_extensions/actions/runs/7194251970 Signed-off-by: Nirmit Shah [email protected] * Unblock test cases blocked due to BABEL-4539 (babelfish-for-postgresql#2047) With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]> * Blocked CS_AS collation for server_collation_name as it is not supported in babelfish (babelfish-for-postgresql#2141) Babelfish currently supports only CI_AS type of collation for server collation and other collations such as CI_AI, CS_AI and CS_AS should be blocked. Currently CI_AI and CS_AI collation for server collation name is blocked but CS_AS is not blocked. It should also be blocked. This changes will block CS_AS collation for server collation name Task: BABEL-4632 Signed-off-by: Rohit Bhagat <[email protected]> * Unblock few more test cases for parallel worker testing (babelfish-for-postgresql#2156) After fixing few jiras related to parallel worker support for Babelfish, some new test cases are now started passing. This commit unblocks all such cases. Issues Resolved Task: BABEL-4392 Signed-off-by: Dipesh Dhameliya [email protected] * Add PR workflow for Static Code Analyzer and fix existing errors and warnings (babelfish-for-postgresql#2143) This commit integrates Static Code Analyzer with PR workflow to improve code quality and catch errors quickly, along with that this commit also fixes the existing errors and warnings. Signed-off-by: Sumit Jaiswal [email protected] * Fixed definition of parsename, session_context and sp_set_session_context (babelfish-for-postgresql#2152) Following is the current definition of PARSENAME, session_context and sp_set_session_context -- parsename CREATE OR REPLACE FUNCTION sys.parsename(object_name sys.VARCHAR, object_piece int) RETURNS sys.SYSNAME AS 'babelfishpg_tsql', 'parsename' LANGUAGE C IMMUTABLE STRICT; -- session_context CREATE OR REPLACE FUNCTION sys.session_context ("@key" sys.sysname) RETURNS sys.SQL_VARIANT AS 'babelfishpg_tsql', 'session_context' LANGUAGE C; GRANT EXECUTE ON FUNCTION sys.session_context TO PUBLIC; -- sp_set_session_context CREATE OR REPLACE PROCEDURE sys.sp_set_session_context ("@key" sys.sysname, "@value" sys.SQL_VARIANT, "@read_only" sys.bit = 0) AS 'babelfishpg_tsql', 'sp_set_session_context' LANGUAGE C; GRANT EXECUTE ON PROCEDURE sys.sp_set_session_context TO PUBLIC; But the correct definitions should be as follows, in these definitions sys.NVARCHAR should be used instead of sys.VARCHAR and sys.NVARCHAR(128) instead of sys.SYSNAME. This commit fixes such issues. --parsename CREATE OR REPLACE FUNCTION sys.parsename(object_name sys.NVARCHAR(128), object_piece int) RETURNS sys.NVARCHAR(128) AS 'babelfishpg_tsql', 'parsename' LANGUAGE C IMMUTABLE STRICT; -- session_context CREATE OR REPLACE FUNCTION sys.session_context ("@key" sys.NVARCHAR(128)) RETURNS sys.SQL_VARIANT AS 'babelfishpg_tsql', 'session_context' LANGUAGE C; GRANT EXECUTE ON FUNCTION sys.session_context TO PUBLIC; -- sp_set_session_context CREATE OR REPLACE PROCEDURE sys.sp_set_session_context ("@key" sys.NVARCHAR(128), "@value" sys.SQL_VARIANT, "@read_only" sys.bit = 0) AS 'babelfishpg_tsql', 'sp_set_session_context' LANGUAGE C; GRANT EXECUTE ON PROCEDURE sys.sp_set_session_context TO PUBLIC; Task: BABEL-4583 Signed-off-by: Rohit Bhagat <[email protected]> * Fix typmod for sys.binary datatype (babelfish-for-postgresql#2037) sys.binary datatype is created as a domain over sys.bbf_binary. In babelfish we assume hex code as varbinary which must then be casted to binary. For casting we fetch rule from hashmap after coercing both source and target to base type. So binary --> bbf_binary && varbinary-->bbf_varbinary This cast is currently defined as binary coercible, that is NO modification needed. https://github.com/babelfish-for-postgresql/babelfish_extensions/blob/c57e8bc7820e1ef697dd9898b3aa433ba504a7ab/contrib/babelfishpg_tsql/src/pltsql_coerce.c#L114 We fix it by adding proper function for varbinary --> binary cast -> The default length for local variables defined for binary and varbinary should be 1 DECLARE @A binary should be equivalent to DECLARE @A binary(1) DECLARE @A varbinary should be equivalent to DECLARE @A varbinary(1) ->Problems during dump and restore done during upgrade. The new function created for the cast may not exists in the source version and the ugrade scripts are run only after pg dump and restore carried out during version upgrade. So ultimately during dump and restore no cast exists for varbinary to binary, which is a necessary requirement or upgrade will fail with error : _"pg_restore: error: could not execute query: ERROR: argument of DEFAULT must be type sys."binary", not type sys.varbinary"_. if we are dumping a procedure with sys.binary arg and a default value -> binary to varbinary we do not need to modify binary to varbinary cast since we have a type modifer defined for base type of sys.varbinary i.e. CAST(sys.bbf_varbiniary as sys.bbf_varbiniary). Task: BABEL-4544 Signed-off-by: Tanzeel Khan <[email protected]> --------- Signed-off-by: Kristian Lejao <[email protected]> Signed-off-by: Nirmit Shah <[email protected]> Signed-off-by: Tanzeel Khan [email protected] Signed-off-by: Zhibai Song <[email protected]> Signed-off-by: Dipesh Dhameliya [email protected] Signed-off-by: Nirmit Shah [email protected] Signed-off-by: Rohit Bhagat <[email protected]> Signed-off-by: Sumit Jaiswal [email protected] Signed-off-by: Tanzeel Khan <[email protected]> Co-authored-by: Kristian Lejao <[email protected]> Co-authored-by: Nirmit Shah <[email protected]> Co-authored-by: Tanzeel Khan <[email protected]> Co-authored-by: Zhibai Song <[email protected]> Co-authored-by: Dipesh Dhameliya <[email protected]> Co-authored-by: Rohit bhagat <[email protected]> Co-authored-by: Sumit Jaiswal <[email protected]>

…l#2047) With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…babelfish-for-postgresql#260) When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error. In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context about Babelfish Extension changes: babelfish-for-postgresql/babelfish_extensions#2047 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…l#2047) With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…#260) When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error. In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context about Babelfish Extension changes: babelfish-for-postgresql/babelfish_extensions#2047 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

With engine changes babelfish-for-postgresql/postgresql_modified_for_babelfish#260, we introduced new field in Port data structure to track whether given backend is Babelfish backend or not. This commit changes extension part of code to initialise it properly. Additionally, it also unblocks test cases blocked due to BABEL-4539. Engine changes: babelfish-for-postgresql/postgresql_modified_for_babelfish#260 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

…nforced (babelfish-for-postgresql#260) When any error raised from parallel worker then Postgres by default does not share message_id (part of edata) to leader worker. Babelfish's error mapping framework relies on message_id to map to correct T-SQL error code. But since it is no more available to leader worker, T-SQL error code would not be found and default error code will be used. This will also affect the transactional behaviour of given error. In order to fix this issue, we need to store if given worker is spawned in the context of Babelfish or not to share error message_id. Hence, this changes made changes in two data structure namely Port and FixedParallelState to store context about Babelfish Extension changes: babelfish-for-postgresql/babelfish_extensions#2047 Task: BABEL-4539 Signed-off-by: Dipesh Dhameliya <[email protected]>

Fixed error code difference occurring due to parallel worker enforced

3cde6f2

surendravishnoi reviewed Nov 24, 2023

View reviewed changes

Dipesh Dhameliya and others added 8 commits November 30, 2023 14:55

Add additional checks for parallel worker

7558916

Merge branch 'babelfish-for-postgresql:BABEL_3_X_DEV__PG_15_X' into j…

f3497aa

…ira-babel-4359

Address comments

8b84e8d

Introduce new field in port structure to track whether given backend …

6b02461

…is Babelfish

Fixed typo

e1d3d30

Address comments

a67cf1e

Remove unnecessary header files

290dec2

Merge branch 'babelfish-for-postgresql:BABEL_3_X_DEV__PG_15_X' into j…

c32c131

…ira-babel-4359

Deepesh125 mentioned this pull request Dec 18, 2023

Unblock test cases blocked due to BABEL-4539 babelfish-for-postgresql/babelfish_extensions#2047

Merged

1 task

surendravishnoi reviewed Dec 18, 2023

View reviewed changes

Addressed review comments

85fb25d

surendravishnoi approved these changes Dec 18, 2023

View reviewed changes

Deepesh125 merged commit 4b6d741 into babelfish-for-postgresql:BABEL_3_X_DEV__PG_15_X Dec 18, 2023
2 checks passed

Deepesh125 deleted the jira-babel-4359 branch December 18, 2023 15:50

Deepesh125 mentioned this pull request Dec 19, 2023

Unblock test cases blocked due to BABEL-4539 babelfish-for-postgresql/babelfish_extensions#2155

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed error code difference occurring due to parallel worker enforced #260

Fixed error code difference occurring due to parallel worker enforced #260

Deepesh125 commented Nov 21, 2023 •

edited

Loading

surendravishnoi Nov 24, 2023

KushaalShroff Nov 30, 2023

Deepesh125 Nov 30, 2023

surendravishnoi Dec 1, 2023

surendravishnoi Nov 24, 2023

Deepesh125 Nov 30, 2023

surendravishnoi Dec 1, 2023

Deepesh125 Dec 12, 2023

surendravishnoi Nov 24, 2023

Deepesh125 Nov 30, 2023

surendravishnoi Dec 1, 2023

Deepesh125 Dec 12, 2023

surendravishnoi Dec 18, 2023

Deepesh125 Dec 18, 2023

surendravishnoi Dec 18, 2023

Deepesh125 Dec 18, 2023

Fixed error code difference occurring due to parallel worker enforced #260

Fixed error code difference occurring due to parallel worker enforced #260

Conversation

Deepesh125 commented Nov 21, 2023 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Deepesh125 commented Nov 21, 2023 •

edited

Loading