Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/multiregionccl: TestColdStartLatency failed #106569

Closed
cockroach-teamcity opened this issue Jul 11, 2023 · 3 comments · Fixed by #107195
Closed

ccl/multiregionccl: TestColdStartLatency failed #106569

cockroach-teamcity opened this issue Jul 11, 2023 · 3 comments · Fixed by #107195
Assignees
@nvanbenschoten
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 11, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/multiregionccl.TestColdStartLatency failed with artifacts on master @ c69ae0c023ef390d36eaaf488b598458d31c9c13:

github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc0168c2c80?, {0x7917640?, 0xc00c664780}, {0xc023a96208, 0x8}, 0xc00e175320, 0x1000000000000?, 0xc01c685300, {{0x1770bbe43e99bd54, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:941 +0xb2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc0168c2c80, {0x7917640, 0xc00c664780}, 0xc00e175320, 0xc026c5f4a0, 0xc01c685300, {{0x1770bbe43e99bd54, 0x0, 0x0}, {0x1770bbe4211dffaf, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:124 +0x18f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc0168c2c80, {0x7917640, 0xc00c664780}, 0xc00e175320, 0xc026c5f4a0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:175 +0x7ca
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0168c2c80, {0x7917640, 0xc00c664780}, 0xc00e175320, 0x660c870)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:510 +0x383
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0168c2c80, {0x7917640?, 0xc00c664750?}, 0xc00e175320)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x756
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc008e8b500, {0x7917640?, 0xc00c664720?}, 0xc00e175320)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x66a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x0?, {0x7917640, 0xc00c664720}, 0xc00e175320)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xef
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc001ba3c00, {0x7917640?, 0xc00c6646c0?}, {0x6?}, 0xc00e175320)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1295 +0x490
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc001ba3c00, {0x7917640, 0xc00c664630}, 0xc00e175320)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1426 +0x2f5
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x7917640, 0xc00c664630}, {0x617e960?, 0xc00e175320})
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10099 +0x78
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1({0x7917640, 0xc00c664630}, {0x617e960, 0xc00e175320}, 0xc017ecbe60, 0xc02744ac60)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96 +0x254
google.golang.org/grpc.getChainUnaryHandler.func1({0x7917640, 0xc00c664630}, {0x617e960, 0xc00e175320})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x7917640, 0xc00c664630}, {0x617e960, 0xc00e175320}, 0xc017ecbe60?, 0xc004735640)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:167 +0x83
google.golang.org/grpc.getChainUnaryHandler.func1({0x7917640, 0xc00c664630}, {0x617e960, 0xc00e175320})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0023fe000?, {{0x45f007?}, {0x79556f0?, 0xc000c592c0?}}}, {0x7917640, 0xc00c664510}, {0x617e960, 0xc00e175320}, 0xc017ecbe60, 0xc004735600)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x2b4
google.golang.org/grpc.getChainUnaryHandler.func1({0x7917640, 0xc00c664510}, {0x617e960, 0xc00e175320})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x7917640?, 0xc00c664510?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x39
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc002a97f40, {0x7917640, 0xc00c664510}, {0x3c5?, 0xc01c687950?}, 0xc01c687948)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd1
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x7917640?, 0xc00c664510?}, {0x617e960?, 0xc00e175320?}, 0xc017ecbe60?, 0xc02744ac60?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:132 +0x95
google.golang.org/grpc.chainUnaryInterceptors.func1({0x7917640, 0xc00c664510}, {0x617e960, 0xc00e175320}, 0xc024e5da20?, 0x58cd1a0?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x8f
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x614c320?, 0xc001ba3c00}, {0x7917640, 0xc00c664510}, 0xc01010f420, 0xc002b1bca0)
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10101 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00416d860, {0x79693c0, 0xc0228fc4e0}, 0xc01462ca20, 0xc000d7d410, 0xad5d560, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc00416d860, {0x79693c0, 0xc0228fc4e0}, 0xc01462ca20, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x28a
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-29619
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jul 11, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Jul 11, 2023
@rafiss rafiss added T-serverless and removed T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jul 11, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestColdStartLatency failed with artifacts on master @ 2b6857a56c242c9f05e07c163fa69f44c1b7a97b:

github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc00d480c80?, {0x793c240?, 0xc0259e4d20}, {0xc030131608, 0x8}, 0xc00c82e000, 0x1000000000001?, 0xc031e63300, {{0x1771a7423e3c1c8a, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:960 +0xb2
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc00d480c80, {0x793c240, 0xc0259e4d20}, 0xc00c82e000, 0xc026d44820, 0xc031e63300, {{0x1771a7423e3c1c8a, 0x0, 0x0}, {0x1771a742208afe86, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:124 +0x18f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc00d480c80, {0x793c240, 0xc0259e4d20}, 0xc00c82e000, 0xc026d44820)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:179 +0x7ea
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00d480c80, {0x793c240, 0xc0259e4d20}, 0xc00c82e000, 0x6623a28)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:510 +0x383
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc00d480c80, {0x793c240?, 0xc0259e46f0?}, 0xc00c82e000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x756
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc00997aa80, {0x793c240?, 0xc0259e46c0?}, 0xc00c82e000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x66a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x0?, {0x793c240, 0xc0259e46c0}, 0xc00c82e000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xef
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0030f0c00, {0x793c240?, 0xc0259e4630?}, {0x6?}, 0xc00c82e000)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1296 +0x490
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0030f0c00, {0x793c240, 0xc0259e45a0}, 0xc00c82e000)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1427 +0x2f5
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x793c240, 0xc0259e45a0}, {0x6194de0?, 0xc00c82e000})
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10099 +0x78
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1({0x793c240, 0xc0259e45a0}, {0x6194de0, 0xc00c82e000}, 0xc030c524e0, 0xc02fcbb350)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96 +0x254
google.golang.org/grpc.getChainUnaryHandler.func1({0x793c240, 0xc0259e45a0}, {0x6194de0, 0xc00c82e000})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x793c240, 0xc0259e45a0}, {0x6194de0, 0xc00c82e000}, 0xc030c524e0?, 0xc031958640)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:167 +0x83
google.golang.org/grpc.getChainUnaryHandler.func1({0x793c240, 0xc0259e45a0}, {0x6194de0, 0xc00c82e000})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc003ffa000?, {{0x45f007?}, {0x797a2d0?, 0xc0010d3bc0?}}}, {0x793c240, 0xc0259e4480}, {0x6194de0, 0xc00c82e000}, 0xc030c524e0, 0xc031958600)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x2b4
google.golang.org/grpc.getChainUnaryHandler.func1({0x793c240, 0xc0259e4480}, {0x6194de0, 0xc00c82e000})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x793c240?, 0xc0259e4480?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x39
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0030ce640, {0x793c240, 0xc0259e4480}, {0x3c7?, 0xc031e65950?}, 0xc031e65948)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd1
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x793c240?, 0xc0259e4480?}, {0x6194de0?, 0xc00c82e000?}, 0xc030c524e0?, 0xc02fcbb350?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:132 +0x95
google.golang.org/grpc.chainUnaryInterceptors.func1({0x793c240, 0xc0259e4480}, {0x6194de0, 0xc00c82e000}, 0xc009ef1a20?, 0x58e1840?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x8f
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x6162a20?, 0xc0030f0c00}, {0x793c240, 0xc0259e4480}, 0xc016b48fc0, 0xc003f75580)
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10101 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc002bdb680, {0x798dfa0, 0xc014597d40}, 0xc0197d0c60, 0xc001175a40, 0xad91500, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc002bdb680, {0x798dfa0, 0xc014597d40}, 0xc0197d0c60, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x28a
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cucaroach
Copy link
Contributor

Full error:

=== RUN   TestColdStartLatency
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestColdStartLatency2561090214
    test_log_scope.go:81: use -show-logs to present logs inline
    cold_start_latency_test.go:178: 0 ALTER TENANT ALL SET CLUSTER SETTING sql.zone_configs.allow_for_secondary_tenant.enabled = true
    cold_start_latency_test.go:178: 1 ALTER TENANT ALL SET CLUSTER SETTING sql.multi_region.allow_abstractions_for_secondary_tenants.enabled = true
    cold_start_latency_test.go:178: 2 alter range meta configure zone using constraints = '{"+region=us-east1": 1, "+region=us-west1": 1, "+region=europe-west1": 1}';
    cold_start_latency_test.go:178: 0
        BEGIN;
        ALTER DATABASE system PRIMARY REGION "us-east1";
        ALTER DATABASE system ADD REGION "us-west1";
        ALTER DATABASE system ADD REGION "europe-west1";
        COMMIT;
*
* ERROR: a panic has occurred!
* panic: ClockTimestamp with Synthetic flag set
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:884
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:890
*   | github.com/cockroachdb/cockroach/pkg/util/hlc.ClockTimestamp.ToTimestamp
*   |   github.com/cockroachdb/cockroach/pkg/util/hlc/pkg/util/hlc/timestamp.go:425
*   | github.com/cockroachdb/cockroach/pkg/storage.MVCCValue.LocalTimestampNeeded
*   |   github.com/cockroachdb/cockroach/pkg/storage/mvcc_value.go:101
*   | github.com/cockroachdb/cockroach/pkg/storage.mvccResolveWriteIntent
*   |   github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:4808
*   | github.com/cockroachdb/cockroach/pkg/storage.MVCCResolveWriteIntentRange
*   |   github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:5202
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ResolveIntentRange
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_resolve_intent_range.go:58
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:476
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:295
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:708
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:676
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:450
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:900
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:988
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:124
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_write.go:179
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:510
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
*   |   github.com/cockroachdb/cockroach/pkg/server/node.go:1296
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
*   |   github.com/cockroachdb/cockroach/pkg/server/node.go:1427
*   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:637
*   | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
*   |   github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
*   | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:748
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:167
*   | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:748
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
*   | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:748
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
*   |   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
* Wraps: (2) panic: ClockTimestamp with Synthetic flag set
* Error types: (1) *withstack.withStack (2) *errutil.leafError

@blathers-crl blathers-crl bot added the T-kv KV Team label Jul 18, 2023
@andrewbaptist andrewbaptist added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-kv Anything in KV that doesn't belong in a more specific category. labels Jul 19, 2023
@nvanbenschoten nvanbenschoten self-assigned this Jul 19, 2023
@nvanbenschoten
Copy link
Member

This is fallout from #105523. That change eliminated the encoding and decoding of synthetic timestamps in MVCC keys. However, it failed to account for cases where the MVCCMetadata proto contains a synthetic timestamp that points to an MVCC key that does not. I'll send a fix.

@nvanbenschoten nvanbenschoten mentioned this issue Jul 19, 2023
Merged
craig bot pushed a commit that referenced this issue Jul 20, 2023
@nvanbenschoten
Merge #107195
c6f4c0e 
107195: storage: restore synthetic timestamp handling in MVCCValue.GetLocalTimestamp r=sumeerbhola a=nvanbenschoten

Fixes #106569.

In 3773994, we stopped decoding the synthetic timestamp bit from MVCC keys. When doing so, we removed logic from `MVCCValue.GetLocalTimestamp` that handles synthetic timestamps. In doing so, we missed the fact that the timestamp provided to `MVCCValue.GetLocalTimestamp` can also come from `MVCCMetadata.Timestamp`, which has not yet been stripped of synthetic timestamps. Eventually, we will get to that as part of #101938, but for now, we restore this handling.

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in ff6d581 Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
23.2
5 participants
@rafiss @nvanbenschoten @cockroach-teamcity @andrewbaptist @cucaroach