store: crashes with new memcached config

[kamilhristov]

Thanos version used: v0.10.0-rc.0

Object Storage Provider: AWS S3

What happened: Thanos Store crashes when trying to query any data.

Arguments:

thanos store --index-cache.config-file=/app/cache-config.yaml --chunk-pool-size=6GB --objstore.config-file=/app/objstore-config.yaml

Config:

type: MEMCACHED
config:
  addresses: ["*********.cache.amazonaws.com:11211"]

Logs:

level=info ts=2020-01-09T16:58:44.91368086Z caller=factory.go:43 msg="loading bucket configuration"
level=info ts=2020-01-09T16:58:45.096521711Z caller=factory.go:30 msg="loading index cache configuration"
level=info ts=2020-01-09T16:58:45.114047201Z caller=memcached.go:54 msg="created memcached index cache"
level=info ts=2020-01-09T16:58:45.114407301Z caller=options.go:20 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-01-09T16:58:45.114663961Z caller=store.go:288 msg="starting store node"
level=info ts=2020-01-09T16:58:45.114779469Z caller=store.go:243 msg="initializing bucket store"
level=info ts=2020-01-09T16:58:45.115289288Z caller=prober.go:127 msg="changing probe status" status=healthy
level=info ts=2020-01-09T16:58:45.115307464Z caller=http.go:53 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-01-09T16:58:45.312703838Z caller=fetcher.go:361 component=block.MetaFetcher msg="successfully fetched block metadata" duration=197.897143ms cached=40 returned=40 partial=0
level=info ts=2020-01-09T16:59:37.571359888Z caller=store.go:249 msg="bucket store ready" init_duration=52.456553581s
level=info ts=2020-01-09T16:59:37.571749739Z caller=grpc.go:98 service=gRPC/server component=store msg="listening for StoreAPI gRPC" address=0.0.0.0:10901
level=info ts=2020-01-09T16:59:37.571618464Z caller=prober.go:107 msg="changing probe status" status=ready
level=info ts=2020-01-09T16:59:37.692079452Z caller=fetcher.go:361 component=block.MetaFetcher msg="successfully fetched block metadata" duration=120.681881ms cached=40 returned=40 partial=0
panic: runtime error: index out of range [0] with length 0
/go/src/github.com/thanos-io/thanos/pkg/cacheutil/memcached_server_selector.go:79 +0x29a
github.com/thanos-io/thanos/pkg/cacheutil.(*MemcachedJumpHashSelector).PickServer(0xc000a69c80, 0xc0f84e77c0, 0x48, 0x43148c, 0xc000942c88, 0x45be30, 0xc00058a600)
goroutine 1567 [running]:
/go/pkg/mod/github.com/bradfitz/[email protected]/memcache/memcache.go:470 +0x3da
github.com/bradfitz/gomemcache/memcache.(*Client).GetMulti(0xc000a69cb0, 0xc000b04a20, 0x2, 0x2, 0x0, 0x0, 0x0)
github.com/thanos-io/thanos/pkg/cacheutil.(*memcachedClient).getMultiSingle(0xc00012a180, 0x2195280, 0xc0f84909c0, 0xc000b04a20, 0x2, 0x2, 0x0, 0x0, 0x0)
github.com/thanos-io/thanos/pkg/cacheutil.(*memcachedClient).getMultiBatched(0xc00012a180, 0x2195280, 0xc0f84909c0, 0xc000b04a20, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/thanos-io/thanos/pkg/cacheutil/memcached_client.go:375 +0x19d
github.com/thanos-io/thanos/pkg/cacheutil.(*memcachedClient).GetMulti(0xc00012a180, 0x2195280, 0xc0f84909c0, 0xc000b04a20, 0x2, 0x2, 0x0)
/go/src/github.com/thanos-io/thanos/pkg/cacheutil/memcached_client.go:301 +0x460
/go/src/github.com/thanos-io/thanos/pkg/cacheutil/memcached_client.go:275 +0x84
github.com/thanos-io/thanos/pkg/store/cache.(*MemcachedIndexCache).FetchMultiPostings(0xc000ac8060, 0x2195280, 0xc0f84909c0, 0xa30431c4d18a6f01, 0x9b232c5bbf6af09f, 0xc0f8490b00, 0x2, 0x2, 0x38, 0x1bb1940, ...)
/go/src/github.com/thanos-io/thanos/pkg/store/cache/memcached.go:88 +0x3ef
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings(0xc0f7cb9500, 0xc0001defb0, 0x2, 0x2, 0x2, 0xc0001defb0)
/go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1425 +0x233
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings(0xc0f7cb9500, 0xc0001def40, 0x2, 0x2, 0x21acc20, 0xc000b04100, 0xc000acbfc0, 0x1e5e068, 0x0)
/go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1301 +0x19d
/go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:613 +0x6a
github.com/thanos-io/thanos/pkg/store.blockSeries(0x2195280, 0xc0f84909c0, 0xa30431c4d18a6f01, 0x9b232c5bbf6af09f, 0xc000a4bf20, 0xc0f7cb9500, 0xc0f7cb9560, 0xc0001def40, 0x2, 0x2, ...)
github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func1(0xc000be3768, 0xc000c4c4e0)
/go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:848 +0xea
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x64
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0f81de660, 0xc0005178f0)
created by golang.org/x/sync/errgroup.(*Group).Go
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

[bwplotka]

Thanks for the report (:

We are on it.

cc @pracucci

[pracucci]

Thanks for the report. I take care of the fix.

[pracucci]

@kamilhristov Sorry for the issue. If you have a chance to compile Thanos, I would ask you to build it including #1975 which should fix it.