[YSQL] Segmentation faults observed while restarting the postmaster process

[shantanugupta-yb]

Jira Link: DB-2732

Description

Segmentation fault observed when postmaster process restarted post killing of all the postgres backend process

2022-06-21 18:51:09.881 UTC [18312] LOG: server process (PID 22487) was terminated by signal 9: Killed
2022-06-21 18:51:09.902 UTC [18312] LOG: terminating any other active server processes
2022-06-21 18:51:09.937 UTC [22478] WARNING: terminating connection because of crash of another server process
2022-06-21 18:51:09.937 UTC [22478] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-06-21 18:51:09.962 UTC [22485] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-06-21 18:51:09.962 UTC [22485] HINT: In a moment you should be able to reconnect to the database and repeat your command.
W0621 18:51:09.982143 22524 outbound_call.cc:131] Failed to schedule invoking callback on response for request yb.tserver.PgClientService.Heartbeat to 172.151.33.147: Aborted (yb/rpc/thread_pool.cc:260): Service is shutting down
W0621 18:51:09.982344 22526 outbound_call.cc:131] Failed to schedule invoking callback on response for request yb.tserver.PgClientService.Heartbeat to 172.151.33.147: Aborted (yb/rpc/thread_pool.cc:260): Service is shutting down
Wed Jun 22 03:18:29 UTC 2022

Core file trace:

Core was generated by `postgres: yugabyte yugabyte 172.151.50.19(44830) startup '.
Program terminated with signal 11, Segmentation fault.
#0 pfree (pointer=)
at /nfusr/alma8-gcp-cloud/jenkins-worker-9ctm6t/jenkins/jenkins-github-yugabyte-db-alma8-master-clang12-release-526/yugabyte-db/src/postgres/src/backend/utils/mmgr/../../../../../../../src/postgres/src/backend/utils/mmgr/mcxt.c:1119
1119 /nfusr/alma8-gcp-cloud/jenkins-worker-9ctm6t/jenkins/jenkins-github-yugabyte-db-alma8-master-clang12-release-526/yugabyte-db/src/postgres/src/backend/utils/mmgr/../../../../../../../src/postgres/src/backend/utils/mmgr/mcxt.c: No such file or directory.
(gdb) bt
#0 pfree (pointer=)
at /nfusr/alma8-gcp-cloud/jenkins-worker-9ctm6t/jenkins/jenkins-github-yugabyte-db-alma8-master-clang12-release-526/yugabyte-db/src/postgres/src/backend/utils/mmgr/../../../../../../../src/postgres/src/backend/utils/mmgr/mcxt.c:1119
#1 errfinish (dummy=) at ../../../../../../../src/postgres/src/backend/utils/error/elog.c:523
#2 0x000000000094c912 in quickdie (postgres_signal_arg=) at ../../../../../../src/postgres/src/backend/tcop/postgres.c:2685
#3
#4 AllocSetAlloc (context=0x1f44000, size=) at ../../../../../../../src/postgres/src/backend/utils/mmgr/aset.c:961
#5 0x0000000000b2bcf1 in palloc (size=14) at ../../../../../../../src/postgres/src/backend/utils/mmgr/mcxt.c:1022
#6 0x00000000005ca3bb in cstring_to_text_with_len (s=0x231d4ee "''::bpchar", len=10)
at /nfusr/alma8-gcp-cloud/jenkins-worker-9ctm6t/jenkins/jenkins-github-yugabyte-db-alma8-master-clang12-release-526/yugabyte-db/src/postgres/src/backend/utils/adt/../../../../../../../src/postgres/src/backend/utils/adt/varlena.c:163
#7 YbTextToDatum (data=0x231d4ee "''::bpchar", bytes=10, type_attrs=) at ../../../../../../../src/postgres/src/backend/catalog/yb_catalog/yb_type.c:260
#8 0x00007f504fec3098 in yb::pggate::(anonymous namespace)::TranslateText(yb::Slice*, yb::pggate::PgWireDataHeader const&, int, PgTypeEntity const*, PgTypeAttrs const*, yb::pggate::PgTuple*) (yb_cursor=0x4dd07a8,
header=..., index=3, type_entity=0x360648 <YbTypeEntityTable+360>, type_attrs=0xb3eb864, pg_tuple=0x7fffc98aad08) at ../../src/yb/yql/pggate/pg_expr.cc:250
#9 0x00007f504fec5897 in yb::pggate::PgDocResult::WritePgTuple(std::__1::vector<yb::pggate::PgExpr*, std::__1::allocatoryb::pggate::PgExpr* > const&, yb::pggate::PgTuple*, long*) (this=0x4dd0790, targets=...,
pg_tuple=0x7fffc98aad08, row_order=0x7fffc98aac98) at ../../src/yb/yql/pggate/pg_doc_op.cc:139
#10 0x00007f504feb6c2a in yb::pggate::PgDml::GetNextRow(yb::pggate::PgTuple*) (this=, pg_tuple=0x7fffc98aad08) at ../../src/yb/yql/pggate/pg_dml.cc:433
#11 0x00007f504feb697e in yb::pggate::PgDml::Fetch(int, unsigned long*, bool*, PgSysColumns*, bool*) (this=0x9bbdba0, natts=, values=0x3abbeb8, isnulls=0xb089fc8, syscols=,
has_data=0x7fffc98ab177) at ../../src/yb/yql/pggate/pg_dml.cc:370
#12 0x00007f504fe8026a in DmlFetch (this=, handle=0xe, natts=2330328, values=0xb1ab148, isnulls=0xb3eb864, syscols=0xabe4000, has_data=) at ../../src/yb/yql/pggate/pggate.cc:1107
#13 YBCPgDmlFetch (handle=0xe, natts=2330328, values=0xb1ab148, isnulls=0xb3eb864, syscols=0x7fffc98aad08, has_data=) at ../../src/yb/yql/pggate/ybc_pggate.cc:657
#14 0x00000000005bdda0 in ybc_getnext_heaptuple (ybScan=0x4eb4558, is_forward_scan=, recheck=) at ../../../../../../../src/postgres/src/backend/access/yb_access/yb_scan.c:300
#15 0x000000000055944f in ybc_systable_getnext (scan_desc=0xb089fe0)
at /nfusr/alma8-gcp-cloud/jenkins-worker-9ctm6t/jenkins/jenkins-github-yugabyte-db-alma8-master-clang12-release-526/yugabyte-db/src/postgres/src/backend/access/yb_access/../../../../../../../src/postgres/src/backend/access/yb_access/yb_scan.c:1755
#16 systable_getnext (sysscan=0xb089fe0) at ../../../../../../../src/postgres/src/backend/access/index/genam.c:438
#17 0x0000000000ad7d6c in AttrDefaultFetch (relation=0x4194598) at ../../../../../../../src/postgres/src/backend/utils/cache/relcache.c:4763
#18 0x0000000000ad7403 in YbCompleteProcessing (state=0x7fffc98ab370) at ../../../../../../../src/postgres/src/backend/utils/cache/relcache.c:1589
#19 0x0000000000ad6b93 in YBUpdateRelationsAttributes (sys_relations_update_required=) at ../../../../../../../src/postgres/src/backend/utils/cache/relcache.c:1666
#20 0x0000000000ad4bc2 in YBPreloadRelCache () at ../../../../../../../src/postgres/src/backend/utils/cache/relcache.c:1806
#21 0x0000000000add96f in RelationCacheInitializePhase3 () at ../../../../../../../src/postgres/src/backend/utils/cache/relcache.c:4379
#22 0x0000000000b02c99 in InitPostgres (in_dbname=, dboid=0, username=, useroid=0, out_dbname=0x0, override_allow_connections=)
at ../../../../../../../src/postgres/src/backend/utils/init/postinit.c:1074
#23 0x000000000094de90 in PostgresMain (argc=1, argv=0x1d11838, dbname=0x1f44000 <incomplete sequence \325>, username=0x1d117f8 "yugabyte") at ../../../../../../src/postgres/src/backend/tcop/postgres.c:4832
#24 0x00000000008ab4ab in BackendRun (port=0x1b9a1e0) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4555
#25 0x00000000008aabba in ServerLoop () at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4198
#26 0x00000000008a7211 in PostmasterMain (argc=, argv=0x1bb0780) at ../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1423
#27 0x00000000007c7b43 in PostgresServerProcessMain (argc=23, argv=0x1bb0780) at ../../../../../../src/postgres/src/backend/main/main.c:234
#28 0x00000000004f5602 in main ()