You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What happened
Adding store nodes to query node configuration causes panics when queries go to store node. The store nodes never return data as they always crash (every time)
What you expected to happen
To return the correct data and not crash
How to reproduce it (as minimally and precisely as possible):
It happens every time a query goes to the storage node. If the time range is low enough and only goes to the sidecars, the query is fine. When the time range is extended into aggregate data time, it goes to store nodes and crashes the store node
Full logs to relevant components
From store node docker log
level=info ts=2019-03-01T14:56:01.834337679Z caller=main.go:308 msg="Listening for metrics" address=0.0.0.0:10902
panic: runtime error: slice bounds out of range
goroutine 13116 [running]:
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).loadSeries(0xc42a4792c0, 0x11e87c0, 0xc43bbe0c80, 0xc4289bf8e0, 0x1, 0x2, 0x237e5460, 0x23805460, 0x0, 0x0)
/go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1399 +0x54f
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).PreloadSeries.func2(0x0, 0x0)
/go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1372 +0xa2
github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1(0xc43bbfcfc0, 0xc43bb9c820, 0xc43bb3ffa0)
/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run
/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:37 +0xa8
Thanos, Prometheus and Golang version used
Thanos version: see docker image info below (0.3.0)
Prometheus version: 2.6.0
Docker image: improbable/thanos:v0.3.0
"Cmd": [
"store",
"--objstore.config-file",
"/etc/thanos/aws-bucket.yml",
"--data-dir=/mnt/data/thanos-store"
],
Anything else we need to know
Store blocks coming from s3.
The text was updated successfully, but these errors were encountered:
Store on v0.3.0 has a bug which causes the Store to panic when querying over more than 24h( see #829 ). Please use at least v0.3.1 or better current master build, which has another fix for the store node (see #833 )
What happened
Adding store nodes to query node configuration causes panics when queries go to store node. The store nodes never return data as they always crash (every time)
What you expected to happen
To return the correct data and not crash
How to reproduce it (as minimally and precisely as possible):
It happens every time a query goes to the storage node. If the time range is low enough and only goes to the sidecars, the query is fine. When the time range is extended into aggregate data time, it goes to store nodes and crashes the store node
Full logs to relevant components
From store node docker log
level=info ts=2019-03-01T14:56:01.834337679Z caller=main.go:308 msg="Listening for metrics" address=0.0.0.0:10902
panic: runtime error: slice bounds out of range
goroutine 13116 [running]:
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).loadSeries(0xc42a4792c0, 0x11e87c0, 0xc43bbe0c80, 0xc4289bf8e0, 0x1, 0x2, 0x237e5460, 0x23805460, 0x0, 0x0)
/go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1399 +0x54f
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).PreloadSeries.func2(0x0, 0x0)
/go/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1372 +0xa2
github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1(0xc43bbfcfc0, 0xc43bb9c820, 0xc43bb3ffa0)
/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run
/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:37 +0xa8
Thanos, Prometheus and Golang version used
Thanos version: see docker image info below (0.3.0)
Prometheus version: 2.6.0
Docker image: improbable/thanos:v0.3.0
"Cmd": [
"store",
"--objstore.config-file",
"/etc/thanos/aws-bucket.yml",
"--data-dir=/mnt/data/thanos-store"
],
Anything else we need to know
Store blocks coming from s3.
The text was updated successfully, but these errors were encountered: