Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes unit tests fail with Panic messages stating "invalid type" #92070

Closed
Rajalakshmi-Girish opened this issue Jun 12, 2020 · 18 comments · Fixed by #93118
Closed

kubernetes unit tests fail with Panic messages stating "invalid type" #92070

Rajalakshmi-Girish opened this issue Jun 12, 2020 · 18 comments · Fixed by #93118
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Milestone

Comments

@Rajalakshmi-Girish
Copy link
Contributor

Rajalakshmi-Girish commented Jun 12, 2020

logs_ci-unit-kubernetes_50_build-log.txt

What happened:
Below are the 27 unit tests that panic and FAIL while I run "make test"

Also the analysis says that the failure started occurring after the below PR merge:
#283d9f7
This commit updates github.com/golang/protobuf from v1.3.3 to v1.4.2.
The invalid Type failures that occur are from the newer implementation of protobuf that is added at "kubernetes/vendor/google.golang.org/protobuf" which did not exist prior to this commit.

Failure Message: panic: field etcdserverpb.Request.ID has invalid type: got uint64, want pointer

  1. pkg/master TestStorageVersionHashes
  2. pkg/registry/apps/daemonset/storage TestDelete
  3. pkg/registry/apps/replicaset/storage TestList
  4. pkg/registry/core/configmap/storage TestCreate
  5. pkg/registry/core/event/storage TestDelete
  6. pkg/registry/core/persistentvolume/storage TestCreate
  7. pkg/registry/core/persistentvolumeclaim/storage TestGet
  8. pkg/registry/core/pod/storage TestEviction
  9. pkg/registry/core/service/storage
  10. pkg/registry/scheduling/priorityclass/storage TestUpdate
  11. pkg/registry/storage/csinode/storage TestCreate
  12. pkg/registry/storage/volumeattachment/storage TestWatch
  13. vendor/k8s.io/apiserver/pkg/registry/generic/registry TestDryRunCreateDoesntCreate
  14. vendor/k8s.io/apiserver/pkg/storage/etcd3 TestCreateWithTTL
  15. vendor/k8s.io/apiserver/pkg/storage/tests TestWatchBookmarksWithCorrectResourceVersion
    Failure message: panic: invalid Go type int for field k8s_io.kubernetes.vendor.go_etcd_io.etcd.etcdserver.etcdserverpb.loggablePutRequest.value_size
  16. pkg/registry/apps/deployment/storage TestUpdate
  17. pkg/registry/apps/statefulset/storage TestStatusUpdate
  18. pkg/registry/batch/cronjob/storage TestCreate
  19. pkg/registry/core/namespace/storage TestCreate
  20. pkg/registry/core/node/storage TestList (
  21. pkg/registry/core/replicationcontroller/storage TestWatch
  22. pkg/registry/core/secret/storage TestCreate
  23. pkg/registry/networking/networkpolicy/storage TestCreate
  24. pkg/registry/settings/podpreset/storage TestUpdate
  25. pkg/registry/storage/csidriver/storage TestList
  26. pkg/registry/storage/storageclass/storage TestUpdate
  27. vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource

What you expected to happen:
The unit tests must pass without error message.

How to reproduce it (as minimally and precisely as possible):

  1. Setup Go lang 1.13.5
  2. Clone kubernetes source code : git clone [email protected]:kubernetes/kubernetes.git
  3. Ensure you are on master branch : git checkout master
  4. cd kubernetes/hack/tools
  5. GO111MODULE=on go install gotest.tools/gotestsum
  6. make generated_files
  7. make test WHAT=
    eg: make test WHAT=k8s.io/kubernetes/pkg/registry/apps/deployment/storage/

Anything else we need to know?:
This failure consistently occurs when unit test is run through an automated Jenkins pipeline. But the failure is seen intermittently when run "make test" manually. Sometimes, easily reproducible when we do "make clean", "make clean_generated" followed by "make test"
Please see the consistent failure at https://k8s-testgrid.appspot.com/sig-node-ppc64le#unit-tests

Environment:
Architechture: linux/ppc64le

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release): Ubuntu 18.04.4
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
    @mkumatag
@Rajalakshmi-Girish Rajalakshmi-Girish added the kind/bug Categorizes issue or PR as related to a bug. label Jun 12, 2020
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jun 12, 2020
@Rajalakshmi-Girish Rajalakshmi-Girish changed the title kubernetes unit tests fail with Panic messages stating "invalid type" [WIP] kubernetes unit tests fail with Panic messages stating "invalid type" Jun 12, 2020
@Rajalakshmi-Girish Rajalakshmi-Girish changed the title [WIP] kubernetes unit tests fail with Panic messages stating "invalid type" kubernetes unit tests fail with Panic messages stating "invalid type" Jun 12, 2020
@neolit123
Copy link
Member

Also the analysis says that the failure started occurring after the below PR merge:
#283d9f7
This commit updates github.com/golang/protobuf from v1.3.3 to v1.4.2.

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 14, 2020
@Rajalakshmi-Girish
Copy link
Contributor Author

Rajalakshmi-Girish commented Jun 15, 2020

Sample stacktrace:

=== FAIL: pkg/registry/apps/deployment/storage TestList (6.31s)
I0608 16:47:26.338842   14665 client.go:360] parsed scheme: "endpoint"
I0608 16:47:26.338887   14665 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{unix://localhost:10576811193999092490  <nil> 0 <nil>}]
I0608 16:47:26.339843   14665 client.go:360] parsed scheme: "endpoint"
I0608 16:47:26.339872   14665 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{unix://localhost:10576811193999092490  <nil> 0 <nil>}]
I0608 16:47:26.346212   14665 client.go:360] parsed scheme: "endpoint"
I0608 16:47:26.346280   14665 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{unix://localhost:10576811193999092490  <nil> 0 <nil>}]
panic: invalid Go type int for field k8s_io.kubernetes.vendor.go_etcd_io.etcd.etcdserver.etcdserverpb.loggablePutRequest.value_size

goroutine 808 [running]:
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.newSingularConverter(0x11e4f800, 0x116a87a0, 0x11e4f6e0, 0xc0000ed1e0, 0x11e17101, 0xc0010707c0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/convert.go:143** +0xe10
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.NewConverter(0x11e4f800, 0x116a87a0, 0x11e4f6e0, 0xc0000ed1e0, 0x11e171a0, 0xc000ffad80)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/convert.go:60 +0xd4
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.fieldInfoForScalar(0x11e4f6e0, 0xc0000ed1e0, 0x1166623b, 0x9, 0x0, 0x0, 0x11e4f800, 0x116a87a0, 0x11666246, 0x2e, ...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect_field.go:234 +0xe0
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc00098cf00, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xc000ffac60, 0xc000ffac90, 0xc000ffacc0, 0xc000ffacf0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:67 +0x8f8
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc00098cf00, 0x11e4f800, 0x118e2e20, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xc000ffac60, 0xc000ffac90, 0xc000ffacc0, ...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:36 +0x68
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc00098cf00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message.go:90 +0x1ac
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).init(0xc00098cf00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message.go:72 +0x60
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).Has(0xc0010c5d30, 0x11e4f6e0, 0xc0000ed100, 0xc0000ed100)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect_gen.go:185 +0x30
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc000ffac30, 0x11e3f400, 0xc0010c5d30, 0x0, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:278 +0x794
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.(*TextMarshaler).marshal(0x12f2b6b4, 0x11decce0, 0xc000ffaba0, 0xc0010c5b20, 0x0, 0x0, 0xc0010c5b20, 0xc000046838)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:86 +0x170
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.(*TextMarshaler).Text(...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:44
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.CompactTextString(...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:106
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*loggablePutRequest).String(...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:182
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*requestOpStringer).String(0xc000bae758, 0x0, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:124 +0x194
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*txnRequestStringer).String(0xc000bae898, 0xc00079dc40, 0x18)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:98 +0x1ec
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*InternalRaftStringer).String(0xc00074c180, 0x16e5eda4e, 0x12fadb00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:67 +0xfc
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.warnOfExpensiveGenericRequest(0xc0010f3bc0, 0xbfafc6d7a8c93cf8, 0x16e5eda4e, 0x12fadb00, 0x11db0ae0, 0xc00074c180, 0x0, 0x0, 0xc0000f7f28, 0x7, ...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/util.go:149 +0x3d4
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.warnOfExpensiveRequest(0xc0010f3bc0, 0xbfafc6d7a8c93cf8, 0x16e5eda4e, 0x12fadb00, 0x11db0ae0, 0xc00074c180, 0x11decaa0, 0xc0006b9e00, 0x0, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/util.go:111 +0xa0
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply.func1(0xc0002f9960, 0xc000f14460, 0xc0006b9b60, 0xbfafc6d7a8c93cf8, 0x16e5eda4e, 0x12fadb00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/apply.go:118 +0xe8
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply(0xc0002f9960, 0xc000f14460, 0xc0006b9b60)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/apply.go:176 +0x240
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply(0xc000a4db30, 0xc000f14460, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/apply_auth.go:60 +0x118
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc000132000, 0xc000baf440)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:2213 +0x1f4
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).apply(0xc000132000, 0xc00079e4c8, 0x1, 0x1, 0xc000208fa0, 0x2, 0xd, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:2127 +0x514
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries(0xc000132000, 0xc000208fa0, 0xc0005f9d00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:1379 +0xb4
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll(0xc000132000, 0xc000208fa0, 0xc0005f9d00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:1103 +0x5c
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8(0x11e01fa0, 0xc0007abd00)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:1048 +0x3c
k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule.(*fifo).run(0xc00111fa40)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x11c
created by k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x144
FAIL	k8s.io/kubernetes/pkg/registry/apps/deployment/storage	6.310s

@Rajalakshmi-Girish
Copy link
Contributor Author

@RainbowMango @liggitt

@RainbowMango
Copy link
Member

ETCD has fixed this issue by etcd-io/etcd#12000.

There still no semantic version for ETCD, I suggest updating the ETCD dependency with a pseudo version, can I? @liggitt

@RainbowMango
Copy link
Member

/assign

@RainbowMango
Copy link
Member

The required version in k/k go.mod is weird:

go.etcd.io/etcd v0.5.0-alpha.5.0.20200520232829-54ba9589114f

Do you know why we use this v0.5.0-alpha.5.0.x instead of v3.4.x ?involved @dims as well.

The etcd commit ID which fixed this issue is ed04d0d98aad8e41c0176a27a792e8ab114f53e8, how to get the version for go.mod?

@liggitt
Copy link
Member

liggitt commented Jun 17, 2020

etcd was just updated in #92075, does that sha not include the fix?

We use a sha in our go.mod because we cannot refer to etcd using a v3.4.x semver version, because it includes a go.mod file and did not rename its module to add a /v3 suffix.

@RainbowMango
Copy link
Member

RainbowMango commented Jun 17, 2020

etcd was just updated in #92075, does that sha not include the fix?

Unfortunately it does not include the fix. The sha in #92075 is v0.5.0-alpha.5.0.20200520232829-54ba9589114f which commit time is 20200520232829, but this fix's commit time is 20200612231056.

Do you know how to create a v0.5.0-alpha.5.0.x-x kind version used by go.mod?

because it includes a go.mod file and did not rename its module to add a /v3 suffix.

etcd already add /v3 suffix in its module. Should we update the import path in k/k accordingly and change it to a semantic version?
https://github.com/etcd-io/etcd/blob/d8c8f903eee10b8391abaef7758c38b2cd393c55/go.mod#L1

module go.etcd.io/etcd/v3

@liggitt
Copy link
Member

liggitt commented Jun 17, 2020

Is etcd fixed in the 3.4.x branch?

@RainbowMango
Copy link
Member

@liggitt
Copy link
Member

liggitt commented Jun 17, 2020

Can that fix be picked to 3.4.x compatibly?

@RainbowMango
Copy link
Member

RainbowMango commented Jun 17, 2020

Is etcd fixed in the 3.4.x branch?

Sorry, I can't get your point, why you ask? And can you share a guideline about what we should do?

Can that fix be picked to 3.4.x compatibly?

Let me give it a try.

@Rajalakshmi-Girish
Copy link
Contributor Author

ETCD has fixed this issue by etcd-io/etcd#12000.

There still no semantic version for ETCD, I suggest updating the ETCD dependency with a pseudo version, can I? @liggitt

looks like it doesnot fix the below error:

=== FAIL: pkg/registry/apps/statefulset/storage TestWatch (5.63s)
panic: field etcdserverpb.Request.ID has invalid type: got uint64, want pointer

goroutine 841 [running]:
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.fieldInfoForScalar(0x11e00fc0, 0xc000557000, 0x115b9553, 0x2, 0x0, 0x0, 0x11e010e0, 0x11670720, 0x115b9557, 0x29, ...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect_field.go:228 +0x7b4
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc0002bf540, 0xffffffffffffffff, 0xffffffffffffffff, 0x98, 0xffffffffffffffff, 0xc000e4a900, 0xc000e4a960, 0xc000e4a990, 0xc000e4a9c0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:67 +0x8f8
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc0002bf540, 0x11e010e0, 0x11982c80, 0xffffffffffffffff, 0xffffffffffffffff, 0x98, 0xffffffffffffffff, 0xc000e4a900, 0xc000e4a960, 0xc000e4a990, ...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:36 +0x68
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc0002bf540)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message.go:90 +0x1ac
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*MessageInfo).init(0xc0002bf540)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message.go:72 +0x60
k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).Has(0xc000eb3c00, 0x11e00fc0, 0xc000557000, 0xc000557000)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/impl/message_reflect_gen.go:185 +0x30
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc000e4a810, 0x11df0e40, 0xc000eb3c00, 0x0, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:278 +0x794
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.(*TextMarshaler).marshal(0x12ecb5f4, 0x11d9e620, 0xc000e78160, 0x14e7b3852, 0x0, 0xbfafc6d96507766a, 0x14567ec8a, 0x12f4d9e0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:86 +0x170
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.(*TextMarshaler).Text(...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:44
k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto.CompactTextString(...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/protobuf/proto/text_encode.go:106
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*Request).String(...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/etcdserver.pb.go:158
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*RequestV2).String(0xc0002973f0, 0x14567ec8a, 0x12f4d9e0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/v2_server.go:164 +0xc0
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.warnOfExpensiveGenericRequest(0xc0008a53e0, 0xbfafc6d96507766a, 0x14567ec8a, 0x12f4d9e0, 0x11d62ac0, 0xc0002973f0, 0x0, 0x0, 0x0, 0x0, ...)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/util.go:149 +0x3d4
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.warnOfExpensiveRequest(0xc0008a53e0, 0xbfafc6d96507766a, 0x14567ec8a, 0x12f4d9e0, 0x11d62ac0, 0xc0002973f0, 0x0, 0x0, 0x0, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/util.go:111 +0xa0
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyV2Request(0xc000134c00, 0xc0002973f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/apply_v2.go:123 +0x254
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc000134c00, 0xc001019440)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:2188 +0xa60
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).apply(0xc000134c00, 0xc0006c2120, 0x2, 0x2, 0xc000634d20, 0x1, 0x1, 0x0)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:2127 +0x514
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries(0xc000134c00, 0xc000634d20, 0xc000fc8300)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:1379 +0xb4
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll(0xc000134c00, 0xc000634d20, 0xc000fc8300)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:1103 +0x5c
k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8(0x11db3f20, 0xc000128d40)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/etcdserver/server.go:1048 +0x3c
k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule.(*fifo).run(0xc0010ae660)
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0x11c
created by k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler
	/jenkins/workspace/k8s-master-unit-tests/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x144
FAIL	k8s.io/kubernetes/pkg/registry/apps/statefulset/storage	5.632s

@gavincabbage
Copy link

gavincabbage commented Jul 14, 2020

@Rajalakshmi-Girish Did you ever find a solution to the second bug above? I'm seeing the same after applying the original fix from etcd-io/etcd#12000.

@Rajalakshmi-Girish
Copy link
Contributor Author

@Rajalakshmi-Girish Did you ever find a solution to the second bug above? I'm seeing the same after applying the original fix from etcd-io/etcd#12000.

No I haven't. @liggitt Is there a solution?

@liggitt
Copy link
Member

liggitt commented Jul 15, 2020

looks like etcd-io/etcd#12106 will pick that to etcd 3.4, then we'd need a 3.4.10 tag and a bump in k8s.io/kubernetes

@liggitt liggitt added this to the v1.19 milestone Jul 15, 2020
@liggitt liggitt added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jul 15, 2020
@bart0sh
Copy link
Contributor

bart0sh commented Jul 16, 2020

@Rajalakshmi-Girish

looks like it doesnot fix the below error

Please report it to etcd similar to how the first bug was reported.

@bart0sh
Copy link
Contributor

bart0sh commented Jul 16, 2020

the fix is in the release-3.4 branch. However, 3.4.10 has not been released yet.

go.mod PR has been created as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants