Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: DB Console causing cockroach process to crash #71728

Closed
Azhng opened this issue Oct 19, 2021 · 0 comments · Fixed by #71739
Closed

server: DB Console causing cockroach process to crash #71728

Azhng opened this issue Oct 19, 2021 · 0 comments · Fixed by #71739
Assignees
@dhartunian
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@Azhng
Copy link
Contributor

Azhng commented Oct 19, 2021

Describe the problem

Opening DB Console crashes Cockroach process on master

To Reproduce

./cockroach start-single-node --insecure --logtostderr --host=127.0.0.1

Open DB Console at localhost:8080

Expected behavior

DB Console loads

Additional data / screenshots

Stack trace

I211019 20:24:54.611489 1154 server/authentication.go:612 ⋮ [-] 86  found a matching cookie that failed decoding: session cookie could not be unmarshaled: ‹proto: wrong wireType = 2 for field ID›
I211019 20:24:54.618315 1154 1@util/log/log.go:34 ⋮ [-] 87  the server is terminating due to a fatal error (see the DEV channel for details)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88  unexpected panic: runtime error: invalid memory address or nil pointer dereference
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !goroutine 1154 [running]:
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xf03fb00, 0x0, 0xc001f16d28, 0x405534c)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb9
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc0012f8aa0, 0xc00108e270, 0x24, 0x1, 0x0, 0x0, 0x0, 0x16af885671e7d860, 0x400000000, 0x0, ...)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:305 +0xc47
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(0xc8c1c60, 0xc00007a0b8, 0x1, 0x4, 0x8e150cf, 0x14, 0xc001f174b8, 0x1, 0x1)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/util/log/channels.go:60 +0x198
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_channels_generated.go:834
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/util/log.FatalOnPanic()
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:34 +0xbd
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !panic(0x86694c0, 0xefab390)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/usr/local/Cellar/go/1.16.5/libexec/src/runtime/panic.go:965 +0x1b9
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/server.(*authenticationServer).verifySession(0xc000546140, 0xc8c1c28, 0xc00243e980, 0x0, 0x8ebb4aa, 0x30, 0xc001dcb9a8, 0x1, 0x1)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/server/authentication.go:374 +0x78
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/server.(*authenticationMux).getSession(0xc0011b4be0, 0xc88b570, 0xc00116be00, 0xc00167e200, 0x0, 0x0, 0x0, 0x0, 0x0)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/server/authentication.go:621 +0x1d2
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/server.(*authenticationMux).ServeHTTP(0xc0011b4be0, 0xc88b570, 0xc00116be00, 0xc00167e200)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/server/authentication.go:548 +0x67
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !net/http.(*ServeMux).ServeHTTP(0xc001449e58, 0xc88b570, 0xc00116be00, 0xc00167e200)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/usr/local/Cellar/go/1.16.5/libexec/src/net/http/server.go:2448 +0x1ad
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !github.com/cockroachdb/cockroach/pkg/server.(*Server).ServeHTTP(0xc001449800, 0xc89bc80, 0xc00175a540, 0xc00167e200)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/Users/azhng/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2674 +0x1dc
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !net/http.serverHandler.ServeHTTP(0xc0014fe2a0, 0xc89bc80, 0xc00175a540, 0xc00167e200)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/usr/local/Cellar/go/1.16.5/libexec/src/net/http/server.go:2887 +0xa3
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !net/http.(*conn).serve(0xc00298a8c0, 0xc8c1cd0, 0xc00243e8c0)
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/usr/local/Cellar/go/1.16.5/libexec/src/net/http/server.go:1952 +0x8cd
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !created by net/http.(*Server).Serve
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !	/usr/local/Cellar/go/1.16.5/libexec/src/net/http/server.go:3013 +0x39b
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !For more context, check log files in: /Users/azhng/tmp/mso_crdb/logs
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !****************************************************************************
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !This node experienced a fatal error (printed above), and as a result the
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !process is terminating.
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !will try to determine the root cause, recommend next steps, and we can
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !improve CockroachDB based on your report.
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !Please submit a crash report by following the instructions here:
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !If you would rather not post publicly, please contact us directly at:
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !    [email protected]
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !
F211019 20:24:54.618364 1154 util/log/log.go:34 ⋮ [-] 88 !The Cockroach Labs team appreciates your feedback.
@Azhng Azhng added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-server-and-security DB Server & Security labels Oct 19, 2021
@dhartunian dhartunian self-assigned this Oct 19, 2021
@dhartunian dhartunian mentioned this issue Oct 19, 2021
Merged
@exalate-issue-sync exalate-issue-sync bot removed the T-server-and-security DB Server & Security label Oct 19, 2021
craig bot pushed a commit that referenced this issue Oct 20, 2021
@erikgrinaker @dhartunian
Merge #68370 #71739
300e4b0 
68370: kvserver: avoid exceeding `TargetBytes` r=nvanbenschoten,yuzefovich a=erikgrinaker

**kvserver: avoid exceeding `TargetBytes`**

Previously, `Header.TargetBytes` would always overshoot the target, i.e.
a scan would terminate only after the result set hit or exceeded
`TargetBytes`.

This patch changes the behavior to never overshoot, except for when the
first result exceeds `TargetBytes` (to make sure clients can always make
progress). However, this is not backwards-compatible with the 21.2
`DistSender` logic, which relies on responses overshooting `TargetBytes`
when enforcing limits for split batches. It therefore adds a
corresponding `TargetBytesAvoidExcess` version gate that only enables
this for 22.1, and `MVCCScanOptions.TargetBytesAvoidExcess` to vary the
scan behavior based on it.

Release note: None

**kvserver: add `TargetBytesAllowEmpty` option**

The `Header.TargetBytes` request limit will be overshot if the first
result exceeds it, to ensure the client can make progress. However, an
upcoming [parallel get/scan streaming library][1] needs to be able to
disable this behavior to avoid exceeding memory budgets when dispatching
parallel requests.

This patch adds a field `TargetBytesAllowEmpty` which will allow an
empty response if the first result exceeds `TargetBytes`. This only has
an effect for `Get` and `Scan` requests, and only on 22.1 clusters since
21.2 `DistSender` logic relies on the limit always being overshot. It
is not supported for `Export` requests where it will return an error.

[1]: https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20210617_index_lookups_memory_limits.md

Resolves #68050.

Release note: None

**kvserver: add `ResponseHeader.ResumeNextBytes`**

This patch adds the field `ResponseHeader.ResumeNextBytes` which may be
populated with the size of the next result in the `ResumeSpan` if
`ResumeReason` is `RESUME_BYTE_LIMIT` (i.e. if `TargetBytes` was
exceeded). This will be needed by an upcoming [parallel get/scan
streaming library][1] to better manage its memory budget.

The field is best-effort, and may be omitted in some cases where the
result size exactly equals `TargetBytes` and we would have to do
additional work to obtain `ResumeNextBytes` (e.g. at the end of a range,
where we would have to send an additional RPC to the next range).

[1]: https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20210617_index_lookups_memory_limits.md

Release note: None

**kvclient: rename replyResults to replyKeys**

Release note: None

71739: server: fix panic on invalid session cookie. r=azhng a=dhartunian

A recent change (#70792) added some code to deal with duplicate cookies
in the HTTP header on DB Console requests to account for situations
where someone may have multiple `session` cookies in addition to the
CRDB one. That change failed to properly check for a dangling `err`
value outside of a loop before proceeding.

The tests have been modified to exercise the code that resulted in a
panic. The bad cookie header is now also appended to unauthenticated
requests in the test to manufacture a situation where only invalid
cookies are in the header.

Resolves #71728

Release note: None

Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: David Hartunian <[email protected]>
@craig craig bot closed this as completed in 65fa563 Oct 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhartunian dhartunian

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

server: fix panic on invalid session cookie. dhartunian/cockroach
2 participants
@dhartunian @Azhng