Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic in six node tpcc cluster with range merging turned off #34256

Closed
awoods187 opened this issue Jan 25, 2019 · 3 comments
Closed

Panic in six node tpcc cluster with range merging turned off #34256

awoods187 opened this issue Jan 25, 2019 · 3 comments
Labels
A-benchmarking C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.

Comments

@awoods187
Copy link
Contributor

Describe the problem

Node died while running tpc-c on a six node cluster with range merging turned off. In the shell I saw:

Error: error in payment: EOF
Error:  exit status 1

To Reproduce

  1. export CLUSTER=andy-base
  2. roachprod create $CLUSTER -n 7 --clouds=aws --aws-machine-type-ssd=c5d.4xlarge
  3. roachprod run $CLUSTER:1-6 -- 'sudo umount /mnt/data1; sudo mount -o discard,defaults,nobarrier /dev/nvme1n1 /mnt/data1/; mount | grep /mnt/data1'
  4. roachprod stage $CLUSTER:1-6 cockroach
  5. roachprod stage $CLUSTER:7 workload
  6. roachprod start $CLUSTER:1-6
  7. roachprod sql $CLUSTER:1
  8. SET CLUSTER SETTING kv.range_merge.queue_enabled = false
  9. roachprod adminurl --open $CLUSTER:1
  10. roachprod run $CLUSTER:1 -- "./cockroach workload fixtures import tpcc --warehouses=5000 --db=tpcc"
  11. roachprod run $CLUSTER:7 "./workload run tpcc --ramp=5m --warehouses=3400 --duration=15m --split --scatter {pgurl:1-3}"

Expected behavior
Completing tpc-c

Additional data / screenshots
image

Environment:
v2.2.0-alpha.20181217-836-g2952f08

From log to stderr

https://gist.github.com/awoods187/6c09caeb2bd844c990868bd612d3c316

* ERROR: [n3,client=172.31.45.222:54918,user=root] a SQL panic has occurred while executing "SELECT i_price, i_name, i_data FROM item WHERE i_id IN (8725, 24118, 39704, 41752, 57020, 65812, 66228) ORDER BY i_id": runtime error: index out of range
*
*
* ERROR: [n3,client=172.31.45.222:54918,user=root] a panic has occurred!
*
panic while executing 1 statements: SELECT _, _, _ FROM _ WHERE _ IN (_, _, __more5__) ORDER BY _; caused by runtime error: index out of range
@jordanlewis
Copy link
Member

This is the same as the other panics. It would be good to consolidate them.

@knz knz added S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. A-benchmarking C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Jan 26, 2019
@jordanlewis
Copy link
Member 

panic while executing 1 statements: SELECT _, _, _ FROM _ WHERE _ IN (_, _, __more5__) ORDER BY _; caused by runtime error: index out of range

goroutine 3269838 [running]:
runtime/debug.Stack(0x38b32a0, 0xc097b793c0, 0xc000000003)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/cockroachdb/cockroach/pkg/util/log.ReportPanic(0x38b32a0, 0xc097b793c0, 0xc0005fd300, 0x2f5c800, 0xc09d3afc50, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:212 +0xa6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc0229e7000, 0x38b32a0, 0xc097b793c0, 0x2d236e0, 0x5484fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:704 +0x2dd
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc0229e7000, 0x38b32a0, 0xc097b793c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:423 +0x61
panic(0x2d236e0, 0x5484fa0)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges.func1(0xc0242f7730, 0xc0242f7a20, 0xc0242f7960, 0xc0242f7a18, 0xc0242f7697, 0xc0242f7718, 0xc0242f769c)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:807 +0x50e
panic(0x2d236e0, 0x5484fa0)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/cockroachdb/cockroach/pkg/roachpb.(*ScanRequest).MarshalTo(0xc012b5a500, 0xc041f94db1, 0x15, 0x15, 0x17, 0x2, 0x13)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10150 +0x17d
github.com/cockroachdb/cockroach/pkg/roachpb.(*RequestUnion_Scan).MarshalTo(0xc0792dd790, 0xc041f94daf, 0x17, 0x17, 0xf2b11e, 0xc0792dd790, 0x19)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:13237 +0xdf
github.com/cockroachdb/cockroach/pkg/roachpb.(*RequestUnion).MarshalTo(0xc0242f6060, 0xc041f94daf, 0x17, 0x17, 0x19, 0xaf, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:13138 +0x73
github.com/cockroachdb/cockroach/pkg/roachpb.(*BatchRequest).MarshalTo(0xc08486d380, 0xc041f94d00, 0xc6, 0xc6, 0xc6, 0xc6, 0x30c54e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:14550 +0x246
github.com/cockroachdb/cockroach/pkg/roachpb.(*BatchRequest).Marshal(0xc08486d380, 0x30c54e0, 0xc08486d380, 0x7f050900f750, 0xc08486d380, 0xc0068ebd01)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:14525 +0x7f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/encoding/proto.codec.Marshal(0x30c54e0, 0xc08486d380, 0xc084164860, 0x3, 0xc000073470, 0xc000073400, 0x7f05090fefc8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/encoding/proto/proto.go:70 +0x19c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.encode(0x7f050424d000, 0x5910db8, 0x30c54e0, 0xc08486d380, 0xc0a959e480, 0x38de500, 0x38b40a0, 0xc0242f6288, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:487 +0x5e
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).sendMsg(0xc04278cb60, 0x30c54e0, 0xc08486d380, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:482 +0xca
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).SendMsg(0xc08486d480, 0x30c54e0, 0xc08486d380, 0xc000faa000, 0x313c547)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:403 +0x43
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x38b3360, 0xc0583c6600, 0x313c547, 0x21, 0x30c54e0, 0xc08486d380, 0x3019ee0, 0xc08aacf2d0, 0xc000faa000, 0xc0003b24e0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:75 +0xfe
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x38b3360, 0xc0583c6600, 0x313c547, 0x21, 0x30c54e0, 0xc08486d380, 0x3019ee0, 0xc08aacf2d0, 0xc000faa000, 0x3260380, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xb49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc000faa000, 0x38b3360, 0xc0583c6600, 0x313c547, 0x21, 0x30c54e0, 0xc08486d380, 0x3019ee0, 0xc08aacf2d0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x109
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc02b91a728, 0x38b3360, 0xc0583c6600, 0xc08486d380, 0x0, 0x0, 0x0, 0xc0583c6600, 0x1, 0x38b3360)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9299 +0xd2
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc0a959e420, 0x38b3360, 0xc0583c6600, 0x1, 0x3887320, 0xc02b91a728, 0x0, 0x0, 0x100000001, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x126
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc0a959e420, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x100000001, 0x1, 0x3f94, 0x0, 0xc0745eaa00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x130
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000129600, 0x38b3360, 0xc0583c6600, 0xc000129650, 0x3f94, 0xc05e848fa0, 0x3, 0x3, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1345 +0x2d3
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000129600, 0x38b3360, 0xc0583c6600, 0x3f94, 0xc05e848fa0, 0x3, 0x3, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:400 +0x244
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000129600, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0745eaa00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:478 +0x228
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000129600, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0745eaa00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1121 +0x322
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000129600, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0745eaa00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:944 +0x8b3
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000129600, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:692 +0x48b
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc0ab53b7d0, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:232 +0xe8
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetrics).SendLocked(0xc0ab53b798, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metrics.go:56 +0xa2
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc0ab53b700, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:160 +0x83
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc0ab53b700, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:101 +0xf9
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc0ab53b680, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:165 +0xf9
github.com/cockroachdb/cockroach/pkg/kv.(*txnIntentCollector).SendLocked(0xc0ab53b640, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_intent_collector.go:105 +0x474
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc0ab53b780, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_sequence_nums.go:66 +0x23b
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeat).SendLocked(0xc0ab53b588, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeat.go:248 +0x533
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc0ab53b400, 0x38b3360, 0xc0583c6600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc03fc9b500, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:650 +0x53b
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004e0780, 0x38b3360, 0xc0583c65d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:622 +0x119
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc0562eb8c0, 0x38b3360, 0xc0583c65d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:804 +0x13c
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch(0xc01fc89ba0, 0x38b3360, 0xc0583c65d0, 0xc00033f5a0, 0x172ad359)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:242 +0x626
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch(0xc01fc89ba0, 0x38b3360, 0xc0583c65d0, 0x0, 0x203016, 0xc00069db60, 0xc019c1b860, 0xc0242f9a80, 0x6e8b1f, 0xc050c3efc0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:326 +0x1dd
github.com/cockroachdb/cockroach/pkg/sql/row.(*kvFetcher).nextKV(0xc030910cd8, 0x38b3360, 0xc0583c65d0, 0x30f4540, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:71 +0x2ef
github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).NextKey(0xc030910ca0, 0x38b3360, 0xc0583c65d0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:489 +0x82
github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartScanFrom(0xc030910ca0, 0x38b3360, 0xc0583c65d0, 0x388dde0, 0xc01fc89ba0, 0x0, 0xe14801)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:479 +0x97
github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartScan(0xc030910ca0, 0x38b3360, 0xc0583c65d0, 0xc0562eb8c0, 0xc05c6de1a0, 0x7, 0x7, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:470 +0x1e7
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*tableReader).Start(0xc030910800, 0x38b3360, 0xc0583c6480, 0x59130a0, 0x2c1f4a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/tablereader.go:253 +0x271
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc030910800, 0x38b3360, 0xc0583c6480)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:801 +0x52
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc019c1b860, 0x38b3360, 0xc0583c6480, 0x325bb20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:657 +0x209
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000461180, 0xc049835e60, 0xc0562eb8c0, 0xc0242fa968, 0xc08fe94580, 0xc0229e7518, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:251 +0x8a0
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc000461180, 0x38b3360, 0xc019dd9c20, 0xc0229e7518, 0xc049835e60, 0xc0562eb8c0, 0x38b4e20, 0xc024610580, 0xc08fe94580)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:793 +0x200
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc0229e7000, 0x38b3360, 0xc019dd9c20, 0xc0229e7450, 0x3, 0x7f04e7fa6000, 0xc06db383f0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1012 +0x26e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0229e7000, 0x38b3360, 0xc019dd9c20, 0x38b8d20, 0xc047bc9540, 0xc0500080af, 0x84, 0x0, 0xc08486d180, 0x3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:856 +0x6bf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0229e7000, 0x38b3360, 0xc019dd9c20, 0x38b8d20, 0xc047bc9540, 0xc0500080af, 0x84, 0x0, 0xc08486d180, 0x3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:437 +0xe85
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0229e7000, 0x38b3360, 0xc019dd9c20, 0x38b8d20, 0xc047bc9540, 0xc0500080af, 0x84, 0x0, 0xc08486d180, 0x3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:93 +0x36f
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0229e7000, 0x38b32a0, 0xc097b793c0, 0xc0009d10b8, 0x5400, 0x15000, 0xc0009d1150, 0xc0234e6b50, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1231 +0x1439
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0005cdba0, 0x38b32a0, 0xc097b793c0, 0xc0229e7000, 0x5400, 0x15000, 0xc0009d1150, 0xc0234e6b50, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:425 +0xce
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func4(0xc0005cdba0, 0x38b32a0, 0xc097b793c0, 0xc0229e7000, 0x5400, 0x15000, 0xc0009d1150, 0xc0234e6b50, 0xc0234e6b60, 0xc01beaa7f4)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:319 +0x81
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:318 +0x1033

@jordanlewis jordanlewis mentioned this issue Jan 28, 2019
Closed
@jordanlewis
Copy link
Member

Duplicate of #34241.

@jordanlewis jordanlewis mentioned this issue Jan 28, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-benchmarking C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jordanlewis @knz @awoods187