sql/pgwire: TestPGTest failed #76655

cockroach-teamcity · 2022-02-16T10:19:25Z

sql/pgwire.TestPGTest failed with artifacts on master @ 512320f0d3c6230edcbe8ca6eab33ed9183bfbfb:

        Query {"String": "DISCARD ALL"}
        ----
RECV *pgproto3.ParameterStatus: &{Name:application_name Value:}
RECV *pgproto3.ParameterStatus: &{Name:DateStyle Value:ISO, MDY}
RECV *pgproto3.ParameterStatus: &{Name:IntervalStyle Value:postgres}
RECV *pgproto3.ParameterStatus: &{Name:TimeZone Value:UTC}
RECV *pgproto3.CommandComplete: &{CommandTag:[68 73 83 67 65 82 68]}
RECV *pgproto3.ReadyForQuery: &{TxStatus:73}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/182/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/pgjdbc:45:
        until [2 args]
        ReadyForQuery
        ----
        {"Type":"CommandComplete","CommandTag":"DISCARD"}
        {"Type":"ReadyForQuery","TxStatus":"I"}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/182/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/pgjdbc:51:
        until [2 args]
        ReadyForQuery
        ----
        {"Type":"CommandComplete","CommandTag":"DISCARD ALL"}
        {"Type":"ReadyForQuery","TxStatus":"I"}
SEND *pgproto3.Parse: &{Name:S_3 Query:BEGIN ParameterOIDs:[]}
SEND *pgproto3.Bind: &{DestinationPortal: PreparedStatement:S_3 ParameterFormatCodes:[] Parameters:[] ResultFormatCodes:[]}
SEND *pgproto3.Execute: &{Portal: MaxRows:0}
SEND *pgproto3.Query: &{String:SAVEPOINT PGJDBC_AUTOSAVE}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/182/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/pgjdbc:59:
        send [0 args]
        Parse {"Name": "S_3", "Query": "BEGIN"}
        Bind {"PreparedStatement": "S_3"}
        Execute
        Query {"String": "SAVEPOINT PGJDBC_AUTOSAVE"}
        ----
RECV *pgproto3.ParseComplete: &{}
RECV *pgproto3.BindComplete: &{}
RECV *pgproto3.CommandComplete: &{CommandTag:[66 69 71 73 78]}
RECV *pgproto3.CommandComplete: &{CommandTag:[83 65 86 69 80 79 73 78 84]}
RECV *pgproto3.ReadyForQuery: &{TxStatus:84}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/182/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/pgjdbc:66:
        until [0 args]
        CommandComplete
        ReadyForQuery
        ----
        {"Type":"ParseComplete"}
        {"Type":"BindComplete"}
        {"Type":"CommandComplete","CommandTag":"BEGIN"}
        {"Type":"CommandComplete","CommandTag":"SAVEPOINT"}
        {"Type":"ReadyForQuery","TxStatus":"T"}

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

TAGS=bazel,gss

/cc @cockroachdb/sql-experience _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

rafiss · 2022-02-16T16:42:35Z

error is

    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/182/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/pgjdbc:66:
        until [0 args]
        CommandComplete
        ReadyForQuery
        ----
        {"Type":"ParseComplete"}
        {"Type":"BindComplete"}
        {"Type":"CommandComplete","CommandTag":"BEGIN"}
        {"Type":"CommandComplete","CommandTag":"SAVEPOINT"}
        {"Type":"ReadyForQuery","TxStatus":"T"}
panic: unexpected Stopped processor

goroutine 25887 [running]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).registerWithRangefeedRaftMuLocked(0xc00545b500, {0x9eeda48, 0xc00281d8c0}, {{0xc00263e728, 0x1, 0x8}, {0xc00263e738, 0x1, 0x8}}, {0x16d43cef41af5247, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:417 +0xbdd
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc00545b500, 0x1c, 0xc0036d2c60, {0x9f50100, 0xc005a93480})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:240 +0xafb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_rangefeed.go:148
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc005bbf200, 0xc0036d2c60, {0x9f50100, 0xc005a93480})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3036 +0x1bf
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0x0, 0xc0036d2c60, {0x9f50100, 0xc005a93480})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:216 +0x185
github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0xc003e4a000, 0xc006a48d00, {0x9f50100, 0xc005a93480})
	github.com/cockroachdb/cockroach/pkg/server/node.go:1194 +0x79
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:702 +0x194
created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:699 +0x28c
I220216 10:19:23.768312 1 (gostd) testmain.go:256  [-] 1  Test //pkg/sql/pgwire:pgwire_test exited with error code 2

I am able to repro with

make stress PKG=./pkg/sql/pgwire TESTS="TestPGTest/pgjdbc" TESTTIMEOUT=15m0s STRESSFLAGS="-stderr -maxfails 1 -maxtime 20m0s -p 16"

i wonder if there's any chance this relates to #76279 ( cc @ajwerner that's just a wild guess, WDYT?)

rafiss · 2022-02-16T17:23:59Z

shouldn't have been closed

rafiss · 2022-02-16T21:29:39Z

Bisected this to #76445

cc @RaduBerinde any ideas how that could affect the following?

# Send a simple query in the middle of extended protocol, which is apparently
# allowed. (See #41511, #33693)
send
Parse {"Name": "S_3", "Query": "BEGIN"}
Bind {"PreparedStatement": "S_3"}
Execute
Query {"String": "SAVEPOINT PGJDBC_AUTOSAVE"}
----

until
CommandComplete
ReadyForQuery
----
{"Type":"ParseComplete"}
{"Type":"BindComplete"}
{"Type":"CommandComplete","CommandTag":"BEGIN"}
{"Type":"CommandComplete","CommandTag":"SAVEPOINT"}
{"Type":"ReadyForQuery","TxStatus":"T"}

RaduBerinde · 2022-02-17T16:44:37Z

CC @ajwerner - this seems like a bug in the rangefeed infrastructure

cockroach/pkg/kv/kvserver/replica_rangefeed.go

Line 417 in 4b8258c

panic("unexpected Stopped processor")

RaduBerinde · 2022-02-17T16:45:31Z

CC @nvanbenschoten

cockroach-teamcity · 2022-02-20T10:04:28Z

sql/pgwire.TestPGTest failed with artifacts on master @ ae6d0339ced65e63f2baca0ace8faf9ad171bbf6:

        Sync
        ----
RECV *pgproto3.ParseComplete: &{}
RECV *pgproto3.BindComplete: &{}
RECV *pgproto3.RowDescription: &{Fields:[{Name:[109 97 120] TableOID:0 TableAttributeNumber:0 DataTypeOID:23 DataTypeSize:4 TypeModifier:-1 Format:0} {Name:[109 105 110] TableOID:0 TableAttributeNumber:0 DataTypeOID:700 DataTypeSize:4 TypeModifier:-1 Format:0}]}
RECV *pgproto3.DataRow: &{Values:[[49 48 48 48] [49 49 46 49]]}
RECV *pgproto3.CommandComplete: &{CommandTag:[83 69 76 69 67 84 32 49]}
RECV *pgproto3.ReadyForQuery: &{TxStatus:73}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1452/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/data_type_size:57:
        until [1 args]
        ReadyForQuery
        ----
        {"Type":"ParseComplete"}
        {"Type":"BindComplete"}
        {"Type":"RowDescription","Fields":[{"Name":"max","TableOID":0,"TableAttributeNumber":0,"DataTypeOID":23,"DataTypeSize":4,"TypeModifier":-1,"Format":0},{"Name":"min","TableOID":0,"TableAttributeNumber":0,"DataTypeOID":700,"DataTypeSize":4,"TypeModifier":-1,"Format":0}]}
        {"Type":"DataRow","Values":[{"text":"1000"},{"text":"11.1"}]}
        {"Type":"CommandComplete","CommandTag":"SELECT 1"}
        {"Type":"ReadyForQuery","TxStatus":"I"}
SEND *pgproto3.Parse: &{Name:s2 Query:SELECT max(b), min(c) FROM minmax_test ParameterOIDs:[]}
SEND *pgproto3.Bind: &{DestinationPortal:p2 PreparedStatement:s2 ParameterFormatCodes:[] Parameters:[] ResultFormatCodes:[1 1]}
SEND *pgproto3.Describe: &{ObjectType:80 Name:p2}
SEND *pgproto3.Execute: &{Portal:p2 MaxRows:0}
SEND *pgproto3.Sync: &{}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1452/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/data_type_size:69:
        send [0 args]
        Parse {"Name": "s2", "Query": "SELECT max(b), min(c) FROM minmax_test"}
        Bind {"DestinationPortal": "p2", "PreparedStatement": "s2", "ResultFormatCodes": [1,1]}
        Describe {"ObjectType": "P", "Name": "p2"}
        Execute {"Portal": "p2"}
        Sync
        ----
RECV *pgproto3.ParseComplete: &{}
RECV *pgproto3.BindComplete: &{}
RECV *pgproto3.RowDescription: &{Fields:[{Name:[109 97 120] TableOID:0 TableAttributeNumber:0 DataTypeOID:23 DataTypeSize:4 TypeModifier:-1 Format:1} {Name:[109 105 110] TableOID:0 TableAttributeNumber:0 DataTypeOID:700 DataTypeSize:4 TypeModifier:-1 Format:1}]}
RECV *pgproto3.DataRow: &{Values:[[0 0 3 232] [65 49 153 154]]}
RECV *pgproto3.CommandComplete: &{CommandTag:[83 69 76 69 67 84 32 49]}
RECV *pgproto3.ReadyForQuery: &{TxStatus:73}
    datadriven.go:75: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1452/execroot/cockroach/bazel-out/k8-dbg/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/cockroach/pkg/sql/pgwire/testdata/pgtest/data_type_size:77:
        until [1 args]
        ReadyForQuery
        ----
        {"Type":"ParseComplete"}
        {"Type":"BindComplete"}
        {"Type":"RowDescription","Fields":[{"Name":"max","TableOID":0,"TableAttributeNumber":0,"DataTypeOID":23,"DataTypeSize":4,"TypeModifier":-1,"Format":1},{"Name":"min","TableOID":0,"TableAttributeNumber":0,"DataTypeOID":700,"DataTypeSize":4,"TypeModifier":-1,"Format":1}]}
        {"Type":"DataRow","Values":[{"binary":"000003e8"},{"text":"A1\ufffd\ufffd"}]}
        {"Type":"CommandComplete","CommandTag":"SELECT 1"}
        {"Type":"ReadyForQuery","TxStatus":"I"}

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

TAGS=bazel,gss

_{This test on roachdash | Improve this report!}

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Feb 16, 2022

blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Feb 16, 2022

otan mentioned this issue Feb 16, 2022

roachtest/test: fix ORM testing due to removed cluster setting #76667

Merged

craig bot closed this as completed in 601d840 Feb 16, 2022

rafiss reopened this Feb 16, 2022

rafiss mentioned this issue Feb 17, 2022

sql/sem/builtins: TestCrdbInternalDatumsToBytesIllegalType failed #76724

Closed

rafiss mentioned this issue Feb 18, 2022

server: TestUpgradeHappensAfterMigrations failed [unexpected stopped rangefeed processor] #76649

Closed

erikgrinaker mentioned this issue Feb 20, 2022

rangefeed: fix panic due to rangefeed stopper race #76825

Merged

craig bot closed this as completed in 4f3e5e0 Feb 20, 2022

This was referenced Feb 20, 2022

release-21.2: rangefeed: fix panic due to rangefeed stopper race #76827

Merged

release-21.1: rangefeed: fix panic due to rangefeed stopper race #76828

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql/pgwire: TestPGTest failed #76655

sql/pgwire: TestPGTest failed #76655

cockroach-teamcity commented Feb 16, 2022

rafiss commented Feb 16, 2022

rafiss commented Feb 16, 2022

rafiss commented Feb 16, 2022

RaduBerinde commented Feb 17, 2022

RaduBerinde commented Feb 17, 2022

cockroach-teamcity commented Feb 20, 2022

sql/pgwire: TestPGTest failed #76655

sql/pgwire: TestPGTest failed #76655

Comments

cockroach-teamcity commented Feb 16, 2022

rafiss commented Feb 16, 2022

rafiss commented Feb 16, 2022

rafiss commented Feb 16, 2022

RaduBerinde commented Feb 17, 2022

RaduBerinde commented Feb 17, 2022

cockroach-teamcity commented Feb 20, 2022