Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add recover() check on each iteration to guard against goja panics #1601

Closed
na-- opened this issue Aug 21, 2020 · 0 comments
Closed

Add recover() check on each iteration to guard against goja panics #1601

na-- opened this issue Aug 21, 2020 · 0 comments

Comments

@na--
Copy link
Member

na-- commented Aug 21, 2020

We've seen this panic when a k6 instance was stopped (essentially equivalent to curl -X PATCH --data '{"data":{"type":"status","id":"default","attributes":{"stopped":true}}}' 'http://localhost:6565/v1/status'):

exit status 2

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x8ecaa0]

goroutine 106 [running]:
github.com/loadimpact/k6/vendor/github.com/dop251/goja.AssertFunction.func1.1(0xc028769b08)
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/runtime.go:1489 +0x98
panic(0xd7b6a0, 0x19917f0)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/loadimpact/k6/vendor/github.com/dop251/goja.(*vm).try.func1(0xc006eabee0, 0x0, 0xc0287699e0, 0x0, 0x0, 0x0, 0xc028769a58)
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/vm.go:377 +0x3da
panic(0xd7b6a0, 0x19917f0)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/loadimpact/k6/vendor/github.com/dop251/goja.(*vm).run(0xc006eabee0)
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/vm.go:289 +0x60
github.com/loadimpact/k6/vendor/github.com/dop251/goja.(*funcObject).Call(0xc00b8d5e00, 0x10ec460, 0x19d6600, 0xc0253f23b0, 0x1, 0x1, 0x203009, 0xc028769990)
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/func.go:154 +0x2cc
github.com/loadimpact/k6/vendor/github.com/dop251/goja.AssertFunction.func1.2()
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/runtime.go:1494 +0x96
github.com/loadimpact/k6/vendor/github.com/dop251/goja.(*vm).try(0xc006eabee0, 0xc028769a70, 0x0)
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/vm.go:383 +0x132
github.com/loadimpact/k6/vendor/github.com/dop251/goja.AssertFunction.func1(0x10ec460, 0x19d6600, 0xc0253f23b0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/home/alpine/go/src/github.com/loadimpact/k6/vendor/github.com/dop251/goja/runtime.go:1493 +0xfc
github.com/loadimpact/k6/js.(*VU).runFn(0xc0044a4990, 0x10df620, 0xc023dbee40, 0x1, 0xc001e0e3c0, 0xc0253f23b0, 0x1, 0x1, 0x10ec460, 0x19d6600, ...)
	/home/alpine/go/src/github.com/loadimpact/k6/js/runner.go:554 +0x14e
github.com/loadimpact/k6/js.(*ActiveVU).RunOnce(0xc0258d14e0, 0x0, 0x0)
	/home/alpine/go/src/github.com/loadimpact/k6/js/runner.go:517 +0x220
github.com/loadimpact/k6/lib/executor.getIterationRunner.func1(0x10df560, 0xc01cb51ac0, 0x10cc400, 0xc0258d14e0, 0x1)
	/home/alpine/go/src/github.com/loadimpact/k6/lib/executor/helpers.go:86 +0x5e
github.com/loadimpact/k6/lib/executor.(*vuHandle).runLoopsIfPossible(0xc027609100, 0xc005d65ee0)
	/home/alpine/go/src/github.com/loadimpact/k6/lib/executor/vu_handle.go:220 +0x53a
created by github.com/loadimpact/k6/lib/executor.RampingVUs.Run
	/home/alpine/go/src/github.com/loadimpact/k6/lib/executor/ramping_vus.go:632 +0xc36

It's nigh impossible to reproduce, and while panics should never happen in k6, the bigger problem here is that everything dies because a single VU had some weird issue. So, after #877 (comment) or something like it is done, we should add recover() around each executed JS iteration. If it panicked, we should emit a interrupted_iterations{cause:error} =1 metric, or maybe even interrupted_iterations{cause:panic}... We should also log the panic via our logger, with a PanicLevel, but we shouldn't actually panic and abort the whole k6 process!

@na-- na-- added this to the v0.28.0 milestone Aug 21, 2020
@na-- na-- modified the milestones: v0.28.0, v0.29.0 Sep 8, 2020
mstoykov added a commit that referenced this issue Oct 30, 2020
salem84 pushed a commit to salem84/k6 that referenced this issue Nov 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants