client-go batch client batchSendLoop panic #47691

crazycs520 · 2023-10-17T03:20:57Z

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Panic stack log:

[2023/10/13 14:00:33.948 +08:00] [ERROR] [client_batch.go:303] [batchSendLoop] [r={}] [stack="[github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop.func1](http://github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop.func1)
	/go/pkg/mod/[github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:305](http://github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:305)
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:884
runtime.goPanicIndex
	/usr/local/go/src/runtime/panic.go:113
[github.com/pingcap/kvproto/pkg/tikvpb.encodeVarintTikvpb](http://github.com/pingcap/kvproto/pkg/tikvpb.encodeVarintTikvpb)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:5438](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:5438)
[github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request_Coprocessor).MarshalToSizedBuffer](http://github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request_Coprocessor).MarshalToSizedBuffer)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:4325](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:4325)
[github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request_Coprocessor).MarshalTo](http://github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request_Coprocessor).MarshalTo)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:4313](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:4313)
[github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request).MarshalToSizedBuffer](http://github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest_Request).MarshalToSizedBuffer)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3850](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3850)
[github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).MarshalToSizedBuffer](http://github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).MarshalToSizedBuffer)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3808](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3808)
[github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).Marshal](http://github.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).Marshal)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3766](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:3766)
[google.golang.org/protobuf/internal/impl.legacyMarshal](http://google.golang.org/protobuf/internal/impl.legacyMarshal)
	/go/pkg/mod/[google.golang.org/[email protected]/internal/impl/legacy_message.go:402](http://google.golang.org/[email protected]/internal/impl/legacy_message.go:402)
[google.golang.org/protobuf/proto.MarshalOptions.marshal](http://google.golang.org/protobuf/proto.MarshalOptions.marshal)
	/go/pkg/mod/[google.golang.org/[email protected]/proto/encode.go:166](http://google.golang.org/[email protected]/proto/encode.go:166)
[google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend](http://google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend)
	/go/pkg/mod/[google.golang.org/[email protected]/proto/encode.go:125](http://google.golang.org/[email protected]/proto/encode.go:125)
[github.com/golang/protobuf/proto.marshalAppend](http://github.com/golang/protobuf/proto.marshalAppend)
	/go/pkg/mod/[github.com/golang/[email protected]/proto/wire.go:40](http://github.com/golang/[email protected]/proto/wire.go:40)
[github.com/golang/protobuf/proto.Marshal](http://github.com/golang/protobuf/proto.Marshal)
	/go/pkg/mod/[github.com/golang/[email protected]/proto/wire.go:23](http://github.com/golang/[email protected]/proto/wire.go:23)
[google.golang.org/grpc/encoding/proto.codec.Marshal](http://google.golang.org/grpc/encoding/proto.codec.Marshal)
	/go/pkg/mod/[google.golang.org/[email protected]/encoding/proto/proto.go:45](http://google.golang.org/[email protected]/encoding/proto/proto.go:45)
[google.golang.org/grpc.encode](http://google.golang.org/grpc.encode)
	/go/pkg/mod/[google.golang.org/[email protected]/rpc_util.go:595](http://google.golang.org/[email protected]/rpc_util.go:595)
[google.golang.org/grpc.prepareMsg](http://google.golang.org/grpc.prepareMsg)
	/go/pkg/mod/[google.golang.org/[email protected]/stream.go:1708](http://google.golang.org/[email protected]/stream.go:1708)
[google.golang.org/grpc.(*clientStream).SendMsg](http://google.golang.org/grpc.(*clientStream).SendMsg)
	/go/pkg/mod/[google.golang.org/[email protected]/stream.go:846](http://google.golang.org/[email protected]/stream.go:846)
[github.com/pingcap/kvproto/pkg/tikvpb.(*tikvBatchCommandsClient).Send](http://github.com/pingcap/kvproto/pkg/tikvpb.(*tikvBatchCommandsClient).Send)
	/go/pkg/mod/[github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:2068](http://github.com/pingcap/[email protected]/pkg/tikvpb/tikvpb.pb.go:2068)
[github.com/tikv/client-go/v2/internal/client.(*batchCommandsClient).send](http://github.com/tikv/client-go/v2/internal/client.(*batchCommandsClient).send)
	/go/pkg/mod/[github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:519](http://github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:519)
[github.com/tikv/client-go/v2/internal/client.(*batchConn).getClientAndSend](http://github.com/tikv/client-go/v2/internal/client.(*batchConn).getClientAndSend)
	/go/pkg/mod/[github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:381](http://github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:381)
[github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop](http://github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop)
	/go/pkg/mod/[github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:344](http://github.com/tikv/client-go/[email protected]/internal/client/client_batch.go:344)"]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

v6.5.5

The text was updated successfully, but these errors were encountered:

aytrack · 2023-10-17T05:23:09Z

/sig transaction
/severity major

…47716) (#47745) close #47691

ref #47691, close #48107

cfzjywxk · 2023-11-08T14:34:28Z

@crazycs520 Is it fixed on the master branch?

ref #47691

seiya-annie · 2024-06-27T09:17:45Z

/found customer

crazycs520 added the type/bug The issue is confirmed as a bug. label Oct 17, 2023

ti-chi-bot bot added sig/transaction SIG:Transaction severity/major may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Oct 17, 2023

crazycs520 mentioned this issue Oct 17, 2023

*: update tikv/client-go to fix batch-client send loop panic issue #47716

Closed

5 tasks

crazycs520 added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. labels Oct 17, 2023

ti-chi-bot bot removed may-affects-6.5 may-affects-7.1 labels Oct 17, 2023

crazycs520 added affects-7.4 affects-7.5 This bug affects the 7.5.x(LTS) versions. and removed may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-7.5 labels Oct 17, 2023

crazycs520 mentioned this issue Oct 18, 2023

*: update tikv/client-go to fix batch-client send loop panic issue (#47716) #47745

Merged

5 tasks

crazycs520 added the priority/release-blocker This issue blocks a release. Please solve it ASAP. label Oct 18, 2023

ti-chi-bot bot pushed a commit that referenced this issue Oct 18, 2023

*: update tikv/client-go to fix batch-client send loop panic issue (#…

536545d

…47716) (#47745) close #47691

qiancai mentioned this issue Oct 19, 2023

v7.1.2: add release notes pingcap/docs-cn#15256

Merged

18 tasks

cfzjywxk mentioned this issue Oct 31, 2023

*: update tikv/client-go to fix performance regression (#48109) #48139

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this issue Oct 31, 2023

*: update tikv/client-go to fix performance regression (#48109) (#48139)

93996da

ref #47691, close #48107

crazycs520 mentioned this issue Nov 22, 2023

*: update tikv/client-go #48768

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this issue Nov 22, 2023

*: update tikv/client-go (#48768)

1471cb7

ref #47691

crazycs520 closed this as completed Dec 12, 2023

ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client-go batch client batchSendLoop panic #47691

client-go batch client batchSendLoop panic #47691

crazycs520 commented Oct 17, 2023

aytrack commented Oct 17, 2023

cfzjywxk commented Nov 8, 2023

seiya-annie commented Jun 27, 2024

client-go batch client batchSendLoop panic #47691

client-go batch client batchSendLoop panic #47691

Comments

crazycs520 commented Oct 17, 2023

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

aytrack commented Oct 17, 2023

cfzjywxk commented Nov 8, 2023

seiya-annie commented Jun 27, 2024