Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query node hitting storage node causes storage node to panic #878

Closed
fortman opened this issue Mar 1, 2019 · 1 comment
Closed

Query node hitting storage node causes storage node to panic #878

fortman opened this issue Mar 1, 2019 · 1 comment

Comments

@fortman
Copy link

fortman commented Mar 1, 2019

What happened
Adding store nodes to query node configuration causes panics when queries go to store node. The store nodes never return data as they always crash (every time)

What you expected to happen
To return the correct data and not crash

How to reproduce it (as minimally and precisely as possible):
It happens every time a query goes to the storage node. If the time range is low enough and only goes to the sidecars, the query is fine. When the time range is extended into aggregate data time, it goes to store nodes and crashes the store node

Full logs to relevant components
From store node docker log

level=info ts=2019-03-01T14:56:01.834337679Z caller=main.go:308 msg="Listening for metrics" address=0.0.0.0:10902
panic: runtime error: slice bounds out of range
goroutine 13116 [running]:
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).loadSeries(0xc42a4792c0, 0x11e87c0, 0xc43bbe0c80, 0xc4289bf8e0, 0x1, 0x2, 0x237e5460, 0x23805460, 0x0, 0x0)
/go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1399 +0x54f
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).PreloadSeries.func2(0x0, 0x0)
/go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1372 +0xa2
github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1(0xc43bbfcfc0, 0xc43bb9c820, 0xc43bb3ffa0)
/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run
/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:37 +0xa8

Thanos, Prometheus and Golang version used
Thanos version: see docker image info below (0.3.0)
Prometheus version: 2.6.0
Docker image: improbable/thanos:v0.3.0
"Cmd": [
"store",
"--objstore.config-file",
"/etc/thanos/aws-bucket.yml",
"--data-dir=/mnt/data/thanos-store"
],

Anything else we need to know
Store blocks coming from s3.

@mreichardt95
Copy link
Contributor

Store on v0.3.0 has a bug which causes the Store to panic when querying over more than 24h( see #829 ). Please use at least v0.3.1 or better current master build, which has another fix for the store node (see #833 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants