Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql/logictest: TestLogic failed under stress #36570

Closed
cockroach-teamcity opened this issue Apr 5, 2019 · 8 comments · Fixed by #39350
Closed

sql/logictest: TestLogic failed under stress #36570

cockroach-teamcity opened this issue Apr 5, 2019 · 8 comments · Fixed by #39350
Assignees
@vivekmenezes
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
19.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/af621b39a13732f95dde42579b5d6fb55c9e2da7

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1223420&tab=buildLog

=== RUN   TestLogic
--- FAIL: TestLogic (718.60s)
    test_log_scope.go:81: test logs captured to: /tmp/logTestLogic274151351
    test_log_scope.go:62: use -show-logs to present logs inline
=== RUN   TestLogic/local-parallel-stmts
    --- FAIL: TestLogic/local-parallel-stmts (0.25s)
=== RUN   TestLogic/local-parallel-stmts/table
=== PAUSE TestLogic/local-parallel-stmts/table
=== CONT  TestLogic/local-parallel-stmts/table
--- progress: testdata/logic_test/aggregate: 147 statements/queries
        --- FAIL: TestLogic/local-parallel-stmts/table (307.79s)
            logic.go:2366: 
                
                testdata/logic_test/table:426: 
                expected success, but found
                (XX000) internal error: job not found for table id 74, mutation 2
                sql/backfill.go:506: in getJobIDForMutationWithDescriptor()
                DETAIL: stack trace:
                github.com/cockroachdb/cockroach/pkg/sql/backfill.go:506: in getJobIDForMutationWithDescriptor()
                github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1329: in markJobFailed()
                github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1348: in createRollbackJob()
                github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1279: in func2()
                github.com/cockroachdb/cockroach/pkg/sql/lease.go:421: in func1()
                github.com/cockroachdb/cockroach/pkg/internal/client/db.go:598: in func1()
                github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:689: in exec()
                github.com/cockroachdb/cockroach/pkg/internal/client/db.go:597: in Txn()
                github.com/cockroachdb/cockroach/pkg/sql/lease.go:371: in Publish()
                github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1219: in reverseMutations()
                github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:952: in rollbackSchemaChange()
                github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:894: in exec()
                github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:1028: in execSchemaChanges()
                github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2038: in txnStateTransitionsApplyWrapper()
                github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1347: in execCmd()
                github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1157: in run()
                github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:437: in ServeConn()
                github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:581: in func1()
                
                NOTE: internal errors may have more details in logs. Use -show-logs.
            logic.go:2382: 
                testdata/logic_test/table:426: error while processing
            logic.go:2383: pq: internal error: job not found for table id 74, mutation 2
@cockroach-teamcity cockroach-teamcity added this to the 19.1 milestone Apr 5, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 5, 2019
@jordanlewis
Copy link
Member

@vivekmenezes can your team PTAL?

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b1d5ae29d52698485b5398b57178c2ee02b3d49a

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1246064&tab=buildLog


panic: txn: unexpected 10240 leftover bytes [recovered]
	panic: panic while executing 1 statements: COMMIT TRANSACTION; caused by txn: unexpected 10240 leftover bytes

goroutine 1044569 [running]:
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc003b8db00, 0x3222300, 0xc00deadf40, 0x2778d80, 0xc00091a290)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:723 +0x36d
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc003b8db00, 0x3222300, 0xc00deadf40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:435 +0x61
panic(0x2778d80, 0xc00091a290)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/cockroachdb/cockroach/pkg/util/log.ReportOrPanic(0x3222300, 0xc0049c13c0, 0xc0099f3980, 0xc003bd60c0, 0x24, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:551 +0x219
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc005060f00, 0x3222300, 0xc0049c13c0, 0xc004038001)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:396 +0x25d
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).Stop(0xc005060f00, 0x3222300, 0xc0049c13c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:383 +0x44
github.com/cockroachdb/cockroach/pkg/sql.(*txnState).finishSQLTxn(0xc003b8db90)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/txn_state.go:234 +0x59
github.com/cockroachdb/cockroach/pkg/sql.cleanupAndFinish(0x32223c0, 0xc008c364e0, 0x31f5100, 0xc000274600, 0x2a16ea0, 0xc003b8db90, 0x31f4fa0, 0xc00091a1f0, 0x28e9120, 0xc00091a200, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_fsm.go:525 +0xd8
github.com/cockroachdb/cockroach/pkg/util/fsm.Transitions.apply(0xc0001c61e0, 0x32223c0, 0xc008c364e0, 0x31f5100, 0xc000274600, 0x2a16ea0, 0xc003b8db90, 0x31f4fa0, 0xc00091a1f0, 0x28e9120, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/fsm/fsm.go:105 +0x184
github.com/cockroachdb/cockroach/pkg/util/fsm.(*Machine).ApplyWithPayload(0xc003b8db68, 0x32223c0, 0xc008c364e0, 0x31f4fa0, 0xc00091a1f0, 0x28e9120, 0xc00091a200, 0x0, 0xc00ee51588)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/fsm/fsm.go:134 +0x10c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).txnStateTransitionsApplyWrapper(0xc003b8db00, 0x31f4fa0, 0xc00091a1f0, 0x28e9120, 0xc00091a200, 0x1488eddd52e0, 0xc000788370, 0x36, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1996 +0x138
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc003b8db00, 0x32223c0, 0xc008c36420, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1347 +0x9d9
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc003b8db00, 0x3222300, 0xc00deadf40, 0xc0096ed0b8, 0x5400, 0x15000, 0xc0096ed150, 0xc0081c0310, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1157 +0x1b4
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00c158dc0, 0x3222300, 0xc00deadf40, 0xc003b8db00, 0x5400, 0x15000, 0xc0096ed150, 0xc0081c0310, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:437 +0xce
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc00be5153d, 0xc0097aad40, 0x3222300, 0xc00deadf40, 0xc0081c0310, 0xc00c158dc0, 0xc00e47b600, 0x3225d00, 0xc0097aad20, 0xc005b049c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:581 +0x21f
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:516 +0x17b

goroutine 1 [chan receive, 2 minutes]:
testing.(*T).Run(0xc00078a300, 0x2b64129, 0x9, 0x2c9c8e8, 0x922926)
	/usr/local/go/src/testing/testing.go:879 +0x383
testing.runTests.func1(0xc00078a200)
	/usr/local/go/src/testing/testing.go:1119 +0x78
testing.tRunner(0xc00078a200, 0xc00065fde8)
	/usr/local/go/src/testing/testing.go:827 +0xbf
testing.runTests(0xc000274560, 0x4961b00, 0x5, 0x5, 0x32140af)
	/usr/local/go/src/testing/testing.go:1117 +0x2aa
testing.(*M).Run(0xc00017e600, 0x0)
	/usr/local/go/src/testing/testing.go:1034 +0x165

@jordanlewis
Copy link
Member

Huh. This is interesting. We need to sit down and try to fix all the failures in here soon.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/be35db9bddf0739a3971d4095ce19d896e251738

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1262522&tab=buildLog

=== RUN   TestLogic
--- FAIL: TestLogic (435.53s)
    test_log_scope.go:81: test logs captured to: /tmp/logTestLogic513954828
    test_log_scope.go:62: use -show-logs to present logs inline
=== RUN   TestLogic/fakedist-metadata/rename_table
=== PAUSE TestLogic/fakedist-metadata/rename_table
=== CONT  TestLogic/fakedist-metadata/rename_table
        --- FAIL: TestLogic/fakedist-metadata/rename_table (1.07s)
            logic.go:2330: 
                
                testdata/logic_test/rename_table:230: 
                expected success, but found
                (40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE): "sql txn" id=8962780c key=/Table/SystemConfigSpan/Start rw=true pri=0.00909255 stat=PENDING epo=0 ts=1556275732.589792607,1 orig=1556275732.586635050,0 max=1556275732.586635050,0 wto=false seq=19
                sql/sqlbase/errors.go:36: in NewRetryError()
            logic.go:2346: 
                testdata/logic_test/rename_table:230: error while processing
            logic.go:2347: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE): "sql txn" id=8962780c key=/Table/SystemConfigSpan/Start rw=true pri=0.00909255 stat=PENDING epo=0 ts=1556275732.589792607,1 orig=1556275732.586635050,0 max=1556275732.586635050,0 wto=false seq=19
=== RUN   TestLogic/fakedist-metadata
    --- FAIL: TestLogic/fakedist-metadata (0.41s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/3f2470f358e638f526a07495bfd11f1805c135a8

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1326901&tab=buildLog

=== RUN   TestLogic
--- FAIL: TestLogic (370.20s)
    test_log_scope.go:79: test logs captured to: /tmp/logTestLogic238692849
    test_log_scope.go:60: use -show-logs to present logs inline
=== RUN   TestLogic/fakedist-metadata/crdb_internal
=== PAUSE TestLogic/fakedist-metadata/crdb_internal
=== CONT  TestLogic/fakedist-metadata/crdb_internal
--- done: testdata/logic_test/deep_interleaving with config fakedist-metadata: 13 tests, 0 failures
        --- FAIL: TestLogic/fakedist-metadata/crdb_internal (0.55s)
=== RUN   TestLogic/fakedist-metadata
    --- FAIL: TestLogic/fakedist-metadata (0.38s)
=== RUN   TestLogic/fakedist-metadata/crdb_internal/max_retry_counter
--- done: testdata/logic_test/crdb_internal with config fakedist-metadata: 78 tests, 1 failures
            --- FAIL: TestLogic/fakedist-metadata/crdb_internal/max_retry_counter (0.02s)
                logic.go:2336: 
                     
                    testdata/logic_test/crdb_internal:434: SELECT key, (max_retries > 1), flags LIKE '!%' AS f
                      FROM crdb_internal.node_statement_statistics
                     WHERE application_name = 'test_max_retry'
                    ORDER BY key, f
                    expected:
                        SELECT crdb_internal.force_retry(_)  true  false
                        SELECT crdb_internal.force_retry(_)  true  true
                        SET application_name = DEFAULT       false false
                    but found (query options: "") :
                        SELECT crdb_internal.force_retry(_)  true   false
                        SELECT crdb_internal.force_retry(_)  false  true
                        SET application_name = DEFAULT       false  false

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/3bc1328efa85e2fa4bbde2f02d306be2fd5d777a

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1363058&tab=buildLog

=== RUN   TestLogic
--- FAIL: TestLogic (364.65s)
    test_log_scope.go:77: test logs captured to: /tmp/logTestLogic285534448
    test_log_scope.go:58: use -show-logs to present logs inline
=== RUN   TestLogic/5node-dist-vec/dist_vectorize
=== PAUSE TestLogic/5node-dist-vec/dist_vectorize
=== CONT  TestLogic/5node-dist-vec/dist_vectorize
--- done: testdata/logic_test/dist_vectorize with config 5node-dist-vec: 11 tests, 1 failures
        --- FAIL: TestLogic/5node-dist-vec/dist_vectorize (1.52s)
            logic.go:2338: 
                 
                testdata/logic_test/dist_vectorize:53: SELECT url FROM [EXPLAIN ANALYZE SELECT count(*) FROM kv]
                expected:
                    https://cockroachdb.github.io/distsqlplan/decode.html#eJzMlcGLm0AUxu_9K-SdtjBWR002O6ctPeXQpCRZeiiyzOrDSo0jM-Oyy5L_vaiFGmlnBgIbjz7z5fs-fg_fG9Qixw0_ogL2AygQiIBADAQSILCAlEAjRYZKCdn9ZBCs8xdgIYGyblrdjVMCmZAI7A10qSsEBgf-VOEOeY4yCIFAjpqXVW_TyPLI5ev9r2cgsG94rZjnB53xttXM24gagcAT19lPVJ5oddONu3C6barJSGGFmS6fS_3KvPBT2HkpzavK0-URmRcqSE8EBsmfrErzAoHRE3Hv87koJBZcCxkszut82T5sDo-77ff9zcfLcuMLZq0uRW3PHv03-9_IbS1kjhLzs7zpydyOTmjtH74-rjeHm3v6fu3is3bUfdOofdOCyA_iK--apdGIxnJ2uxa504gcaMR-kFyZhqXRiMbt7GjE7jRiBxqJ33_drknD0mhEYzU7Gok7jcSBxsK_MgtLnxGLu9mxsNzzHapG1Aont_Hf_xx2NxPzAocDq0QrM_wmRdbbDI_bXtcPclR6eEuHh3U9vOoCjsXUKI7OxHQqjszOFuvYqE7M4uSS3AujeGl2Xl7ifGsUr8zOq0uc78ysQsuamJds6p2ePvwOAAD__0aN3zM=
                    
                but found (query options: "") :
                    https://cockroachdb.github.io/distsqlplan/decode.html#eJzcVUFr2zAYve9XmO_UgTJbttOmOnXslMOSkaTsMExRrQ_XzJGMJJeWkv8-bA-WmE3y8FhDj_rk5_ce76HvBaQSuOJ7NMC-AQUCMRBIgEAKBOaQEai1ytEYpdtPesBSPAGLCJSybmw7zgjkSiOwF7ClrRAY7Ph9hRvkAnUYAQGBlpdVR1Prcs_18833RyCwrbk0LJiFLfG6sSxYKYlA4J7b_AFNoBpbt-NWnG3qajAyWGFuy8fSPrMg-hC1XMbyqgpsuUcWRAayA4Ee8lOrsbxAYPRAxvv5WBQaC26VDuendj6tb1e7u8366_bi_TTd-IR5Y0sl_drjP2r_JbmRSgvUKE70Zge3OzpIa3v7-W652l3c0P_nLjlxR8c3LfY3LYxnYRJwKQIaKPuA-u97F__L3nncHSVzdXa9i990Mh53Z51MMj6ZZEQy6ax79V7zffY4OkpjcXZppOPTSEekMZ-9chYeP0dZXJ9dFp49v0FTK2lwsDN__-eo3aUoCuwXr1GNzvGLVnlH0x_XHa4bCDS2v6X9YSn7q1bgMZg6wfEJmA7BsZvZQ5040ZducDpF93wK8-UU5isneOFmXkxhvnZnFXlq4i7ZkDs7vPsRAAD__91L5wM=
                    
                
            logic.go:2370: 
                testdata/logic_test/dist_vectorize:58: error while processing
            logic.go:2371: testdata/logic_test/dist_vectorize:58: too many errors encountered, skipping the rest of the input
=== RUN   TestLogic/5node-dist-vec
    --- FAIL: TestLogic/5node-dist-vec (0.00s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4bc081bee5a52c4b8116910025c7c04519872e99

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1420659&tab=buildLog


panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x20b2bcc]

goroutine 3538787 [running]:
github.com/cockroachdb/cockroach/pkg/sql/row.(*CFetcher).GetRangesInfo(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/cfetcher.go:1119
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*colBatchScan).DrainMeta(0xc00e5df810, 0x3906c20, 0xc021767470, 0xc06db0bc30, 0x9b7e71, 0x2feefc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/colbatch_scan.go:76 +0x1fc
github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc.(*Outbox).sendMetadata(0xc017010d80, 0x3906c20, 0xc021767470, 0x147febdcb940, 0xc02c6bc2b0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc/outbox.go:249 +0xd2
github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc.(*Outbox).runWithStream(0xc017010d80, 0x3906c20, 0xc021767470, 0x147febdcb940, 0xc02c6bc2b0, 0xc02c6bc0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc/outbox.go:289 +0x182
github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc.(*Outbox).Run(0xc017010d80, 0x3906c20, 0xc021767140, 0x38a1180, 0xc029520480, 0x1a5fbc6800000001, 0x2f9b45b27e455148, 0x97e9d024, 0x3, 0xc02c6bc0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc/outbox.go:156 +0x555
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*vectorizedFlowCreator).setupRemoteOutputStream.func1(0x3906c20, 0xc021767140, 0xc02c6bc0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/column_exec_setup.go:1050 +0x97
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1(0xc0513891c0, 0x3906c20, 0xc021767140, 0xc02c6bc0c0, 0xc05fb24380)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/column_exec_setup.go:1485 +0x44
created by github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/column_exec_setup.go:1484 +0x6f

goroutine 1 [chan receive, 8 minutes]:
testing.(*T).Run(0xc000137600, 0x3190074, 0x9, 0x32d5368, 0x709fc6)
	/usr/local/go/src/testing/testing.go:917 +0x381
testing.runTests.func1(0xc000137500)
	/usr/local/go/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc000137500, 0xc0005c7e10)
	/usr/local/go/src/testing/testing.go:865 +0xc0
testing.runTests(0xc00024c2e0, 0x549dee0, 0x5, 0x5, 0x0)
	/usr/local/go/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc000513800, 0x0)
	/usr/local/go/src/testing/testing.go:1072 +0x162
github.com/cockroachdb/cockroach/pkg/sql/logictest.TestMain(0xc000513800)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/main_test.go:33 +0xb8
main.main()
	_testmain.go:48 +0x13e

goroutine 6 [syscall, 8 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 7 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1222 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:625 +0x121

goroutine 8 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:632 +0x126
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:626 +0x139

goroutine 3536319 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc012a7d180)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a

goroutine 3535591 [select]:

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/3b9a95bd7eb2cfa6d544fe7217852a85ec3b76f4

Parameters:

TAGS=
GOFLAGS=-parallel=2

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestLogic PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1422093&tab=buildLog


panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x20b3139]

goroutine 2010779 [running]:
github.com/cockroachdb/cockroach/pkg/sql/row.(*CFetcher).GetRangesInfo(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/cfetcher.go:1119
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*colBatchScan).DrainMeta(0xc020e529b0, 0x3907b20, 0xc014dac600, 0xc04df4fc30, 0x9b7f31, 0x2fefc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/colbatch_scan.go:76 +0x1f9
github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc.(*Outbox).sendMetadata(0xc009083f00, 0x3907b20, 0xc014dac600, 0x14fbe0891d60, 0xc00bcfbae0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc/outbox.go:249 +0xd2
github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc.(*Outbox).runWithStream(0xc009083f00, 0x3907b20, 0xc014dac600, 0x14fbe0891d60, 0xc00bcfbae0, 0xc01733b650)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc/outbox.go:289 +0x182
github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc.(*Outbox).Run(0xc009083f00, 0x3907b20, 0xc0123d10e0, 0x38a2080, 0xc03b89be60, 0xd3ee41ff00000001, 0x48245d8b0c45473a, 0xc07e4479aa, 0x2, 0xc01733b650)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/exec/colrpc/outbox.go:156 +0x555
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*vectorizedFlowCreator).setupRemoteOutputStream.func1(0x3907b20, 0xc0123d10e0, 0xc01733b650)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/column_exec_setup.go:1050 +0x97
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1(0xc021fc9c60, 0x3907b20, 0xc0123d10e0, 0xc01733b650, 0xc018983d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/column_exec_setup.go:1485 +0x44
created by github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/column_exec_setup.go:1484 +0x6f

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc000137600, 0x3190dd4, 0x9, 0x32d6118, 0x70a086)
	/usr/local/go/src/testing/testing.go:917 +0x381
testing.runTests.func1(0xc000137500)
	/usr/local/go/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc000137500, 0xc000651e10)
	/usr/local/go/src/testing/testing.go:865 +0xc0
testing.runTests(0xc00024c380, 0x549fee0, 0x5, 0x5, 0x0)
	/usr/local/go/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc000505800, 0x0)
	/usr/local/go/src/testing/testing.go:1072 +0x162
github.com/cockroachdb/cockroach/pkg/sql/logictest.TestMain(0xc000505800)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/logictest/main_test.go:33 +0xb8
main.main()
	_testmain.go:48 +0x13e

goroutine 6 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 7 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1222 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:625 +0x121

goroutine 8 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:632 +0x126
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:626 +0x139

goroutine 2000659 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x3907b20, 0xc067ee1260)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0847acb80, 0xc064e92090, 0xc0847acb70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker

This was referenced Aug 5, 2019
Closed
Merged
craig bot pushed a commit that referenced this issue Aug 7, 2019
@jordanlewis
Merge #39350
52e19db 
39350: row: fix behavior of uninitialized cfetchers r=jordanlewis a=jordanlewis

Previously, calling DrainMeta on a cfetcher that hadn't been fully
initialized was problematic. Add a guard for that.

Closes #36570.

Release note: None

Co-authored-by: Jordan Lewis <[email protected]>
@craig craig bot closed this as completed in #39350 Aug 7, 2019
@nvanbenschoten nvanbenschoten mentioned this issue Sep 18, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vivekmenezes vivekmenezes

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
19.1
Development

Successfully merging a pull request may close this issue.

row: fix behavior of uninitialized cfetchers jordanlewis/cockroach
3 participants
@jordanlewis @vivekmenezes @cockroach-teamcity