Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crash when acquiring lock in 3.3.23 #12156

Closed
Roguelazer opened this issue Jul 20, 2020 · 7 comments
Closed

crash when acquiring lock in 3.3.23 #12156

Roguelazer opened this issue Jul 20, 2020 · 7 comments
Labels
stale

Comments

@Roguelazer
Copy link

Roguelazer commented Jul 20, 2020 •
edited
Loading

Attempting to acquire a named lock in a 3.3.23 cluster results in a panic of the leader:

panic: invalid Go type int for field github_com.coreos.etcd.etcdserver.etcdserverpb.loggablePutRequest.value_size
goroutine 661 [running]:
google.golang.org/protobuf/internal/impl.newSingularConverter(0x1205f80, 0xe90600, 0x1205e60, 0xc000085ce0, 0x11f0b01, 0xc002726c80)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/convert.go:143 +0xff6
google.golang.org/protobuf/internal/impl.NewConverter(0x1205f80, 0xe90600, 0x1205e60, 0xc000085ce0, 0x414127, 0xc00037d3b0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/convert.go:60 +0xd9
google.golang.org/protobuf/internal/impl.fieldInfoForScalar(0x1205e60, 0xc000085ce0, 0xe81cb9, 0x9, 0x0, 0x0, 0x1205f80, 0xe90600, 0xe81cc4, 0x2e, ...)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message_reflect_field.go:234 +0x110
google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc0003c4640, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xc001a6f350, 0xc001a6f3b0, 0xc001a6f440, 0xc001a6f4d0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message_reflect.go:67 +0x938
google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc0003c4640, 0x1205f80, 0xfb7880, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xc001a6f350, 0xc001a6f3b0, 0xc001a6f440, ...)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message_reflect.go:36 +0x63
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc0003c4640)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message.go:90 +0x181
google.golang.org/protobuf/internal/impl.(*MessageInfo).init(0xc0003c4640)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message.go:72 +0x3c
google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).Has(0xc000fac740, 0x1205e60, 0xc000085c00, 0xc000085c00)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message_reflect_gen.go:185 +0x34
github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc001a6f260, 0x11fcc80, 0xc000fac740, 0x0, 0x0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:278 +0x925
github.com/golang/protobuf/proto.(*TextMarshaler).marshal(0x19f4024, 0x11df660, 0xc001a6f110, 0x0, 0x0, 0xc000fac4f0, 0x0, 0xc001731780)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:86 +0x18b
github.com/golang/protobuf/proto.(*TextMarshaler).Text(...)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:44
github.com/golang/protobuf/proto.CompactTextString(...)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:106
github.com/coreos/etcd/etcdserver/etcdserverpb.(*loggablePutRequest).String(...)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/coreos/[email protected]+incompatible/etcdserver/etcdserverpb/raft_internal_stringer.go:182
github.com/coreos/etcd/etcdserver/etcdserverpb.(*requestOpStringer).String(0xc00000f1f8, 0xc00000f1f8, 0x0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/coreos/[email protected]+incompatible/etcdserver/etcdserverpb/raft_internal_stringer.go:124 +0x183
github.com/coreos/etcd/etcdserver/etcdserverpb.(*txnRequestStringer).String(0xc001731a28, 0x5e, 0x13)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/coreos/[email protected]+incompatible/etcdserver/etcdserverpb/raft_internal_stringer.go:98 +0x49b
github.com/coreos/etcd/etcdserver/api/v3rpc.logUnaryRequestStats(0x11e6920, 0xc0019a4f60, 0x0, 0xc001d747c0, 0xbfbda70cadaea6ac, 0x6b582d91a, 0x1a52440, 0xff35a0, 0xc000104690, 0xff3700, ...)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/coreos/[email protected]+incompatible/etcdserver/api/v3rpc/interceptor.go:140 +0x2ab
github.com/coreos/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1(0x11e6920, 0xc0019a4f60, 0xff35a0, 0xc000104690, 0xc001d747c0, 0xc0007250e0, 0xff3700, 0xc00037c640, 0x0, 0x0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/coreos/[email protected]+incompatible/etcdserver/api/v3rpc/interceptor.go:74 +0x1ab
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x11e6920, 0xc0019a4f60, 0xff35a0, 0xc000104690, 0xc001d747c0, 0xc001d747e0, 0xc001ad4bd0, 0x57920a, 0xfa3480, 0xc0019a4f60)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 +0x153
github.com/coreos/etcd/etcdserver/etcdserverpb._KV_Txn_Handler(0xf4de20, 0xc001ecdc00, 0x11e6920, 0xc0019a4f60, 0xc0010185a0, 0xc001f32720, 0x11e6920, 0xc0019a4f60, 0xc0006e7a00, 0x189)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/github.com/coreos/[email protected]+incompatible/etcdserver/etcdserverpb/rpc.pb.go:6380 +0x158
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0022846e0, 0x11f6820, 0xc00248a180, 0xc000109200, 0xc001f328a0, 0x1a43308, 0x0, 0x0, 0x0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/server.go:995 +0x466
google.golang.org/grpc.(*Server).handleStream(0xc0022846e0, 0x11f6820, 0xc00248a180, 0xc000109200, 0x0)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/server.go:1275 +0xda6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000498150, 0xc0022846e0, 0x11f6820, 0xc00248a180, 0xc000109200)
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0x9f
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/ANT.AMAZON.COM/leegyuho/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1

Simply running etcdctl --endpoints http://hostname:port lock /test ls is enough to trigger this.
@xiang90
Copy link
Contributor

xiang90 commented Jul 20, 2020

is this already fixed by #12106?

@ptabor
Copy link
Contributor

ptabor commented Jul 21, 2020

The fix is in 3.5 and 3.4.10. I don't think it was backported to 3.3.x branch.

Can you updated etcd to 3.4 or downgrade protobuf version used together with 3.3 ?

@Roguelazer
Copy link
Author

I cannot immediately upgrade to 3.4.

I am using your binary builds of 3.3; if you release a new 3.3 with a an older protobuf, then I'll gladly use it...

@cristichiru
Copy link

cristichiru commented Jul 22, 2020 •
edited
Loading

I confirm hitting the same bug. In our case, 3.3.x is used because of kubespray deployment of etcd.

For the time being, I have downgraded to 3.3.22 which seems to work fine.

@zhangguanzhang
Copy link
Contributor

same issue in v3.3.23 with commit id

Git SHA: 4873f5516
Go Version: go1.12.17

zhangguanzhang added a commit to zhangguanzhang/Kubernetes-ansible that referenced this issue Jul 24, 2020
@zhangguanzhang
change etcd back to v3.3.22,because etcd-io/etcd#12156
097f721
zhangguanzhang added a commit to zhangguanzhang/Kubernetes-ansible that referenced this issue Jul 24, 2020
@zhangguanzhang
change etcd back to v3.3.22,because etcd-io/etcd#12156
7586883
zhangguanzhang added a commit to zhangguanzhang/Kubernetes-ansible that referenced this issue Jul 24, 2020
@zhangguanzhang
change etcd back to v3.3.22,because etcd-io/etcd#12156
3105f61
zhangguanzhang added a commit to zhangguanzhang/Kubernetes-ansible that referenced this issue Jul 24, 2020
@zhangguanzhang
change etcd back to v3.3.22,because etcd-io/etcd#12156
e2e07c5
@wenjiaswe
Copy link
Contributor

3.3.23 has security fix for CVE-2020-15106 and CVE-2020-15112, could we backport #12000 (#12106) to 3.3?

cc @YoyinZyc

@wenjiaswe wenjiaswe mentioned this issue Aug 13, 2020
Merged
@stale
Copy link

stale bot commented Nov 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Nov 11, 2020
@stale stale bot closed this as completed Dec 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Milestone
No milestone
Development

No branches or pull requests
6 participants
@Roguelazer @xiang90 @ptabor @zhangguanzhang @cristichiru @wenjiaswe