reflect: k8s unit tests on ppc64le sees SEGV fault with tip #48238

Karthik-K-N · 2021-09-08T06:16:17Z

What version of Go are you using (`go version`)?

$ go version
go version devel go1.18-6226020c2f Tue Sep 7 03:56:13 2021 +0000 linux/ppc64le

Does this issue reproduce with the latest release?

No

What operating system and processor architecture are you using (`go env`)?

Linux ppc64le

go env Output

$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/go/bin/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/go/bin:/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/go/bin:/root/bin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="devel go1.18-f5bdbf311c Wed Sep 8 05:32:14 2021 +0000"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/kubernetes/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3272977011=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Installed go and kubernetes using below steps and ran the test for kubernetes

Install upstream dev version of golang:
    Have a stable version of go already installed.
    git clone https://github.com/golang/go.git
    cd go/src
    ./make.bash
    cd ../..
    rm -rf /usr/local/go
    cp -r ./go /usr/local
    export PATH=/usr/local/go/bin:$PATH
    export PATH=$GOPATH/bin:$PATH
Clone k8s/k8s repo and run below steps: (git clone https://github.com/kubernetes/kubernetes)
    export KUBE_TIMEOUT='--timeout=600s' export KUBE_COVER="n" export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export LOG_LEVEL=4
    cd kubernetes
    pushd ./hack/tools
    GO111MODULE=on go install gotest.tools/gotestsum
    popd
    make test WHAT=./cmd/kubeadm/app/cmd/

What did you expect to see?

Test passing successfully

What did you see instead

Test panics with the below stack trace, Even observed similar panic on x86 as well.

[root@karthik-workspace kubernetes]# make test WHAT=./cmd/kubeadm/app/cmd/
+++ [0907 08:51:10] Running tests without code coverage and with -race
[certs] Generating "ca" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node] and IPs [10.96.0.1 1.2.3.4]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node] and IPs [1.2.3.4 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node] and IPs [1.2.3.4 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[kubeconfig] Writing "admin.conf" kubeconfig file
--- FAIL: TestRunRenewCommands (0.17s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1 pc=0x10a6d9b0]

goroutine 67 [running]:
testing.tRunner.func1.2({0x12c11d20, 0x142f1640})
	/usr/local/go/src/testing/testing.go:1209 +0x36c
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1212 +0x334
panic({0x12c11d20, 0x142f1640})
	/usr/local/go/src/runtime/panic.go:820 +0x2ac
reflect.mapiterinit(0x12bf2600, 0x0, 0x1)
	/usr/local/go/src/runtime/map.go:1340 +0x3c
k8s.io/kubernetes/vendor/github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/modern-go/reflect2/unsafe_map.go:112
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0xc00034b9b0, 0xc0001e45a8)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:333 +0xb0
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*placeholderEncoder).IsEmpty(0xc00028c8d0, 0xc0001e45a8)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:336 +0x60
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0xc000363aa0, 0xc0001e4510)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:118 +0x9c
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structEncoder).Encode(0xc000363bf0, 0xc0001e4510, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:148 +0x9f4
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*placeholderEncoder).Encode(0xc00028d540, 0xc0001e4510, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:332 +0x68
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000363e60, 0xc0001e4500, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0xa8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structEncoder).Encode(0xc000363ec0, 0xc0001e4500, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0xc68
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*sliceEncoder).Encode(0xc000360720, 0xc0003501b8, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_slice.go:38 +0x7a8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc000364c30, 0xc000350160, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:110 +0xa8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*structEncoder).Encode(0xc000364d20, 0xc000350160, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_struct_encoder.go:158 +0xc68
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc0003552f0, 0xc0001a6970, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect_optional.go:70 +0xe0
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc000355310, 0xc000350160, 0xc0004fbe00)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:219 +0xc8
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*Stream).WriteVal(0xc0004fbe00, {0x12cd1e20, 0xc000350160})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/reflect.go:98 +0x20c
k8s.io/kubernetes/vendor/github.com/json-iterator/go.(*frozenConfig).Marshal(0xc0002779a0, {0x12cd1e20, 0xc000350160})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/json-iterator/go/config.go:299 +0xbc
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).doEncode(0xc00018f450, {0x1327dab0, 0xc000350160}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:306 +0xb8
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Encode(0xc00018f450, {0x1327dab0, 0xc000350160}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:301 +0x19c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).doEncode(0xc0003265a0, {0x1327da60, 0xc0004fbd40}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:244 +0xb14
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Encode(0xc0003265a0, {0x1327da60, 0xc0004fbd40}, {0x1324ff60, 0xc0002b7530})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:184 +0x19c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.Encode({0x7fffc9c82118, 0xc0003265a0}, {0x1327da60, 0xc0004fbd40})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:50 +0xd8
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.Write(...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:466
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.WriteToFile({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0xc0002b7470, 0xc0002b74d0, 0xc0002b74a0, {0xc000517260, ...}, ...}, ...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/loader.go:419 +0x11c
k8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig.WriteToDisk(...)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig/kubeconfig.go:98
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createKubeConfigFileIfNotExists({0xc00059a571, 0xf}, {0x12e78e96, 0xa}, 0xc0004fbce0)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:270 +0x22c
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createKubeConfigFiles({0xc00059a571, 0xf}, 0xc00063e000, {0xc000161468, 0x1, 0x1})
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:131 +0x1b0
k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.CreateKubeConfigFile({0x12e78e96, 0xa}, {0xc00059a571, 0xf}, 0xc00063e000)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:104 +0x1a8
k8s.io/kubernetes/cmd/kubeadm/app/cmd.TestRunRenewCommands(0xc000583380)
	/root/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/certs_test.go:137 +0x190c
testing.tRunner(0xc000583380, 0x12f68a58)
	/usr/local/go/src/testing/testing.go:1259 +0x1b8
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1306 +0x6b8
FAIL	k8s.io/kubernetes/cmd/kubeadm/app/cmd	0.330s
FAIL
make: *** [Makefile:184: test] Error 1

When I traced the above scenario found a recent change 1b2d794 and the test runs before this commit is successfully passing.

The text was updated successfully, but these errors were encountered:

cuonglm · 2021-09-08T06:37:30Z

I think modern-go/reflect2 must be changed to adopt the new change to mapiterinit: https://github.com/modern-go/reflect2/blob/333559e1834b0c1840d283d79220bf121d704022/unsafe_link.go#L32

It uses //go:linkname, thus is not Go 1 compatibility guarantee.

cc @josharian

mkumatag · 2021-09-08T06:49:21Z

cc @taowen

ALTree · 2021-09-08T12:03:18Z

Putting in the 1.18 milestone just to keep this on the radar for a decision, but if the issue is caused by go:linkname usage in a dependency -as Cuong pointed out-, it should be fixed there. As noted, that's not covered by the Go 1 backward compatibility promise.

ianlancetaylor · 2021-09-09T18:18:44Z

Upstream bug is modern-go/reflect2#24. I'm going to close this issue because there isn't anything to change in the Go repository. (This is an example of why I personally encourage people to avoid using reflect2. It's impossible for us to support that kind of package. Where there are deficiencies in reflect, let's fix them, rather than introducing an unsupportable third party package. Just my opinion, of course.)

taowen · 2021-09-10T15:41:24Z

anyone can help to maintain https://github.com/modern-go/reflect2 ? I no longer have the insight to keep up with the language evolution.

taowen · 2021-09-11T02:21:30Z

fixed at
https://github.com/modern-go/reflect2/releases/tag/v1.0.2
https://github.com/json-iterator/go/releases/tag/v1.1.12

When reading and then subsequently writing a BMH yaml within a bootstrap ignition file we encountered a panic. This seems to be a rather well known issue with these two module versions and upgrading fixes the problem. This was fixed in master by cc8f353 when updating some seemingly unrelated packages, but it seems safer to just updated the required ones rather than also bringing back the prometheus client update. References: golang/go#48238 kubernetes/kubernetes#104947 https://bugzilla.redhat.com/show_bug.cgi?id=2122747

cuonglm added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Sep 8, 2021

cuonglm mentioned this issue Sep 8, 2021

Update mapiterinit to reflect changes in go1.18 reflect modern-go/reflect2#24

Closed

ALTree changed the title ~~k8s unit tests on ppc64le sees SEGV fault with latest golang~~ reflect: k8s unit tests on ppc64le sees SEGV fault with tip Sep 8, 2021

ALTree added this to the Go1.18 milestone Sep 8, 2021

ianlancetaylor closed this as completed Sep 9, 2021

mkumatag mentioned this issue Sep 13, 2021

Bump github.com/json-iterator/go to v1.1.12 kubernetes/kubernetes#104947

Closed

4 tasks

liggitt mentioned this issue Sep 14, 2021

remove use of json-iterator / reflect2 kubernetes-sigs/structured-merge-diff#202

Open

liggitt mentioned this issue Oct 22, 2021

Remove use of unsafe json-iterator / reflect2 libraries etcd-io/etcd#13434

Closed

tamalsaha mentioned this issue Mar 17, 2022

it will panic fault when Marshal map (go version 1.18) json-iterator/go#608

Open

carbonin mentioned this issue Sep 9, 2022

Bug 2122747: Update json-iterator and reflect2 dependencies to fix panic openshift/assisted-service#4378

Merged

21 tasks

golang locked and limited conversation to collaborators Sep 11, 2022

gopherbot added the FrozenDueToAge label Sep 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reflect: k8s unit tests on ppc64le sees SEGV fault with tip #48238

reflect: k8s unit tests on ppc64le sees SEGV fault with tip #48238

Karthik-K-N commented Sep 8, 2021 •

edited

Loading

cuonglm commented Sep 8, 2021 •

edited

Loading

mkumatag commented Sep 8, 2021

ALTree commented Sep 8, 2021 •

edited

Loading

ianlancetaylor commented Sep 9, 2021

taowen commented Sep 10, 2021

taowen commented Sep 11, 2021

reflect: k8s unit tests on ppc64le sees SEGV fault with tip #48238

reflect: k8s unit tests on ppc64le sees SEGV fault with tip #48238

Comments

Karthik-K-N commented Sep 8, 2021 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead

cuonglm commented Sep 8, 2021 • edited Loading

mkumatag commented Sep 8, 2021

ALTree commented Sep 8, 2021 • edited Loading

ianlancetaylor commented Sep 9, 2021

taowen commented Sep 10, 2021

taowen commented Sep 11, 2021

Karthik-K-N commented Sep 8, 2021 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

cuonglm commented Sep 8, 2021 •

edited

Loading

ALTree commented Sep 8, 2021 •

edited

Loading