Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic during reconnect to mulfunctional Redis Cluster #55

Closed
FZambia opened this issue Jun 21, 2022 · 17 comments · Fixed by #170
Closed

Panic during reconnect to mulfunctional Redis Cluster #55

FZambia opened this issue Jun 21, 2022 · 17 comments · Fixed by #170

Comments

@FZambia
Copy link
Contributor

FZambia commented Jun 21, 2022

Got the panic like this after reconnect to a Redis Cluster:

panic: protocol bug, message handled out of order

goroutine 1148 [running]:
github.com/rueian/rueidis.(*pipe)._backgroundRead(0xc0014ae210)
  /Users/fz/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:301 +0x725
github.com/rueian/rueidis.(*pipe)._background(0xc0014ae210)
  /Users/fz/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:135 +0x185
created by github.com/rueian/rueidis.(*pipe).background.func1
  /Users/fz/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:113 +0x5a
exit status 2

In my case I had a working Redis cluster, then stopped the cluster (I am using docker-compose from https://github.com/Grokzen/docker-redis-cluster), ran it again. Newly created cluster was not properly functional due to cluster setup error, so every operation on every node at this point returned:

127.0.0.1:7005> set x 1
(error) CLUSTERDOWN Hash slot not served

I'll try to provide a minimal reproducer example soon. This may be a bit tricky to reproduce though.

@FZambia FZambia changed the title Panic during reconnect to Redis Cluster Panic during reconnect to mulfunctional Redis Cluster Jun 21, 2022
@rueian
Copy link
Collaborator

rueian commented Jun 21, 2022

Oh, I might know the cause of this. This could also take me a couple of days.

@rueian
Copy link
Collaborator

rueian commented Jun 23, 2022

Hi @FZambia,

PR #56 should fix this. However, I need another couple of days to do more tests on this.

@FZambia
Copy link
Contributor Author

FZambia commented Jun 24, 2022

I was trying to reproduce the panic, but can not now. Maybe I changed the code somehow to avoid it to happen. Were you able to reproduce it btw?

One more thing though I came across when trying to find a minimal reproducer is that program like this:

func run() {
	client, err := rueidis.NewClient(rueidis.ClientOption{
		InitAddress: []string{"127.0.0.1:7000", "127.0.0.1:7001"},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	for {
		log.Println("start pub/sub")
		conn, _ := client.Dedicate()
		wait := conn.SetPubSubHooks(rueidis.PubSubHooks{
			OnMessage: func(m rueidis.PubSubMessage) {

			},
		})
		log.Println("pub/sub is ready")
		err := <-wait
		log.Println("pub/sub error", err)
		conn.Close()
		time.Sleep(1 * time.Millisecond)
	}
}

Does not detect cluster shutdowns - i.e. with single Redis instance PUB/SUB interrupted on Redis close and goes to reconnect loop, but when using Redis Cluster and shutting it down – PUB/SUB connection close not detected.

@rueian
Copy link
Collaborator

rueian commented Jun 24, 2022

Hi @FZambia,

For the dedicated cluster client, the underlying connection is actually not established until you send a command. Because which node to connect to is decided by the key slot of the command.

That is why the above program has no reaction to cluster shutdown, since the underlying connection is not yet established after all.

@FZambia
Copy link
Contributor Author

FZambia commented Jun 24, 2022

I see, makes sense!

@FZambia
Copy link
Contributor Author

FZambia commented Jun 24, 2022

Think I was able to reproduce a panic by moving Cluster slot to another node with active PUB/SUB connection (using SSUBSCRIBE).

It's reproduced with 2eba08d

@FZambia
Copy link
Contributor Author

FZambia commented Jun 24, 2022

So steps to reproduce.

  1. Run Redis Cluster
  2. Run program like this:
package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/rueian/rueidis"
)

func main() {
	run()
}

func run() {
	client, err := rueidis.NewClient(rueidis.ClientOption{
		InitAddress: []string{"127.0.0.1:7000", "127.0.0.1:7001"},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	for {
		log.Println("start pub/sub")
		conn, _ := client.Dedicate()
		wait := conn.SetPubSubHooks(rueidis.PubSubHooks{
			OnMessage: func(m rueidis.PubSubMessage) {

			},
			OnSubscription: func(s rueidis.PubSubSubscription) {
				fmt.Println(s.Kind, s.Channel)
			},
		})
		log.Println("pub/sub is ready")
		conn.Do(context.Background(), conn.B().Ssubscribe().Channel("test").Build())
		err := <-wait
		log.Println("pub/sub error", err)
		conn.Close()
		time.Sleep(1 * time.Millisecond)
	}
}
  1. Migrate slot from one cluster node to another:

In my case slot 6918 originally belonged to 7001.

redis-cli -p 7001
127.0.0.1:7001> cluster nodes
01b094549dcda3ffc26a09ad24b9b47b817986f9 127.0.0.1:7005@17005 slave 5c15c1758732ad5dd6208ae65a24b694a215f385 0 1656068049529 3 connected
5c15c1758732ad5dd6208ae65a24b694a215f385 127.0.0.1:7002@17002 master - 0 1656068049000 3 connected 10923-16383
6c95c902602fdfb2251ec7631b6b0d590bcd4111 127.0.0.1:7004@17004 slave 8d075329d148e201e4ecb48d8317dece5e304745 0 1656068049940 8 connected
cce502301ce3bb3421b5a7b611df84f51d9c220d 127.0.0.1:7000@17000 master - 0 1656068048912 9 connected 0-5460 5649 6918
d322a8b4aba02949c34104886a169523662bfb75 127.0.0.1:7003@17003 slave cce502301ce3bb3421b5a7b611df84f51d9c220d 0 1656068049530 9 connected
8d075329d148e201e4ecb48d8317dece5e304745 127.0.0.1:7001@17001 myself,master - 0 1656068048000 8 connected 5461-5648 5650-6917 6919-10922 [5649->-cce502301ce3bb3421b5a7b611df84f51d9c220d] [6918->-cce502301ce3bb3421b5a7b611df84f51d9c220d]

On Node 7000:

127.0.0.1:7000> CLUSTER SETSLOT 6918 IMPORTING 8d075329d148e201e4ecb48d8317dece5e304745
OK

On Node 7001:

127.0.0.1:7001> CLUSTER SETSLOT 6918 MIGRATING cce502301ce3bb3421b5a7b611df84f51d9c220d

On Node 7000:

127.0.0.1:7000> CLUSTER SETSLOT 6918 NODE cce502301ce3bb3421b5a7b611df84f51d9c220d
OK

At this moment panic happens:

2022/06/24 13:49:50 pub/sub is ready
ssubscribe test
sunsubscribe test
panic: protocol bug, message handled out of order

goroutine 22 [running]:
github.com/rueian/rueidis.(*pipe)._backgroundRead(0xc0000f0210)
	/Users/fz/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:295 +0x865
github.com/rueian/rueidis.(*pipe)._background(0xc0000f0210)
	/Users/fz/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:137 +0x172
created by github.com/rueian/rueidis.(*pipe).background.func1
	/Users/fz/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:115 +0x5a
exit status 2

@rueian
Copy link
Collaborator

rueian commented Jun 24, 2022

Oh, thank you very much.

Redis sends a sunsubscribe message right after the CLUSTER SETSLOT 6918 NODE cce502301ce3bb3421b5a7b611df84f51d9c220d. I need to find a way to deal with it.

@rueian
Copy link
Collaborator

rueian commented Jun 25, 2022

Hi @FZambia,

The above issues are fixed in the latest commit of PR #56:

  • Failures of subscribe/unsubscribe are not handled previously and cause panic with "protocol bug". This is the CLUSTERDOWN case you mentioned.
  • sunsubscribe messages send by redis proactively are not handled previously. This is the CLUSTER SETSLOT case you reproduced.

@FZambia
Copy link
Contributor Author

FZambia commented Jun 25, 2022

Just checked – no panic, thanks! 🎉

Though one last thing – after migrating a slot I am successfully getting sunsubscribe and trying to restart PUB/SUB for a slot (since it's migrated – I need to re-subscribe from scratch using new Dedicated connection). But reconnect/resubscribe loop constantly receives an error for SSUBSCRIBE op:

MOVED 5649 127.0.0.1:7000

Seems like information about slots is not updated in rueidis cache? Or slot refresh not triggered? If I restart my app – then all subscriptions are established correctly, picking correct node for slot subscribe.

@rueian
Copy link
Collaborator

rueian commented Jun 25, 2022

Hi @FZambia,

Yes, you are right. I had missed triggering slot refresh in the dedicated client. I have reworked that in the latest commit.

Now it should work with the following program:

package main

import (
	"context"
	"fmt"

	"github.com/rueian/rueidis"
)

func main() {
	client, err := rueidis.NewClient(rueidis.ClientOption{
		InitAddress: []string{"127.0.0.1:7000", "127.0.0.1:7001"},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	for {
		conn, _ := client.Dedicate()
		sig := make(chan struct{})
		conn.SetPubSubHooks(rueidis.PubSubHooks{
			OnMessage: func(m rueidis.PubSubMessage) {
				fmt.Println(m)
			},
			OnSubscription: func(s rueidis.PubSubSubscription) {
				fmt.Println(s)
				if s.Kind == "sunsubscribe" {
					close(sig)
				}
			},
		})
		if err := conn.Do(context.Background(), conn.B().Ssubscribe().Channel("test").Build()).Error(); err == nil {
			<-sig
		}
		conn.Close()
	}
}

Note that the MOVED error will be returned from the conn.Do() and the dedicated client will not switch to the new redis node automatically. You need to go through conn, _ := client.Dedicate() again to switch to the correct redis node.

I have tried to implement handling MOVED error transparently in dedicated client but failed. It is mainly because it is quite hard to make the channel returned from conn.SetPubSubHooks() decoupled from the underlying connection.

@FZambia
Copy link
Contributor Author

FZambia commented Jun 25, 2022

the dedicated client will not switch to the new redis node automatically. You need to go through conn, _ := client.Dedicate() again to switch to the correct Redis node.

Yep, I already do this with Dedicated connection, I think re-establishing is pretty reasonable in this case, should not happen often.

Actually I'd prefer Redis to close PUB/SUB connection automatically upon slot migration, possibly with an error containing where slot was moved to instead of sending sunsubscribe (so that we could immediately update slot map on the client side). Given at most once nature of PUB/SUB connection I am a bit worried that sunsubscribe can be lost in some cases. And also connection will receive sunsubscribe for all the channels in slot - sth I don't really need. Closing from Redis side seems much more attractive. Maybe I'll try to address this in Redis repo a bit later.

I just tried the latest commit – and everything works as expected. Such a great progress, many thanks – will continue experimenting, the feeling is that it's very close :) Also want to test Sentinel scenario - seems the last part of the migration.

@rueian
Copy link
Collaborator

rueian commented Jun 26, 2022

Hi @FZambia,

I just tried the latest commit – and everything works as expected. Such a great progress, many thanks – will continue experimenting, the feeling is that it's very close :) Also want to test Sentinel scenario - seems the last part of the migration.

Thank you for your assistance, v0.0.59 is released. Please let me know if you find any issues. I am happy to help.

Actually I'd prefer Redis to close PUB/SUB connection automatically upon slot migration, possibly with an error containing where slot was moved to instead of sending sunsubscribe

Agreed. sunsubscribe message is the only special case currently that does not fit well into the request<->response communication model: It may or may not be an out of band message. This makes the client hard to handle.

It seems that they had discussed just closing the connection from the Redis side once the slot was moved: redis/redis#8621 (comment), but it just not happened finally. And I didn't find further discussion on this.

Given at most once nature of PUB/SUB connection I am a bit worried that sunsubscribe can be lost in some cases.

https://github.com/redis/redis/blob/a64b29485d4f2359b9d698c0e21e890a212ad1bb/src/cluster.c#L5848

Well, you don't have to worry about that currently. There is no case that the sunsubscribe can be unsent unless the client is disconnected.

@verejoel
Copy link

I ran into this problem again using the client in Thanos. The panic was preceded by many exceeded context deadlines.

level=warn ts=2023-01-27T20:40:57.861170828Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861231427Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861180428Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861237527Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861248327Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861216228Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861274627Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861305927Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861302627Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
level=warn ts=2023-01-27T20:40:57.861303927Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
panic: protocol bug, message handled out of order

goroutine 964911 [running]:
github.com/rueian/rueidis.(*pipe)._backgroundRead(0xc003877560)
	/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:414 +0x8ff
github.com/rueian/rueidis.(*pipe)._background(0xc003877560)
	/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:226 +0x7e
created by github.com/rueian/rueidis.(*pipe).background.func1
	/go/pkg/mod/github.com/rueian/[email protected]/pipe.go:210 +0x5a

@rueian
Copy link
Collaborator

rueian commented Jan 28, 2023

Hi @verejoel, thanks for the report. I will try to figure it out. Do you have any other clue that could help reproduce the problem?

@verejoel
Copy link

verejoel commented Jan 28, 2023

We're using Azure Cache for Redis, Standard tier 26 GB (C5 instance). I just checked and the Redis version is 4, not the latest 6. On Monday I will ask our team to upgrade to 6 to see if this might resolve the issue.

Thanos version is the latest release (v0.30.1) and I'm using their default connection settings for the redis client with TLS enabled, as documented here.

Thanks for taking a look! I had a look in Loki and found another panic that seems to be related.

2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.52291604Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.57278981Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578190874Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578346573Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578570772Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578626971Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578660771Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578655071Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578708671Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578744471Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.578618271Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.744474774Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830296006Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830349105Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830363905Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830377305Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830464105Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830506004Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830528604Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830566804Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830727703Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830741603Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830762903Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830849502Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.831096501Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.831340199Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830338106Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.831718396Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837531058Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837556858Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837571358Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837591458Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837630857Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837649557Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837682857Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.837655357Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.838458952Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.838478152Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.838507451Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.838522051Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.838591351Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.838510051Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.843391319Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.843417919Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.843429619Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.843442719Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.843492218Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.846906596Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982704797Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQ9DEW21TEBAK8N2QY8MNXKY:HDYlTVWvn__W8_83QBt5_sgOHrsyRYvA77OGAzthyv8 value_size=12
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982722397Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJY68GB7XKPTM8796536A2:WmXrpbxKJz4okGoD4bA9ZMfUfX--pdCJKTljfSqRFsQ value_size=67
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982754897Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVR721JY97JZDBHY9DT98X:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=11
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982731697Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVW3XPJZHZKDFZSF1FF262:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=11
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982769597Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJY68GB7XKPTM8796536A2:WmXrpbxKJz4okGoD4bA9ZMfUfX--pdCJKTljfSqRFsQ value_size=67
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982771697Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJRSQ823AEKEE2K208ASBY:DG328Is9WwBZ-YuKgRdYb_pCfd16aAIDODLsw-rYAfk value_size=11855
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982774597Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQSVWAE64PQJT9SF09DEXSTP:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=11
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982779397Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJWG7C14KKTF97XDCCAHJK:WmXrpbxKJz4okGoD4bA9ZMfUfX--pdCJKTljfSqRFsQ value_size=76
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.982796697Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVP3XQX13VHM9WZZ7YHFWY:8dvPq4vuJN7GFbgbkPG2h66sMUJ3Uv46deqC4Znx8Ng value_size=461
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.064867554Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.52360465Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.523702549Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.523780748Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.58242355Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.58244335Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.58245715Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.582492749Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.582733547Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.582759347Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.582820246Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.582873246Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.582884946Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.583001745Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.583098244Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.828428359Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.828453259Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.828542558Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.828711156Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.828723856Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829109353Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829184452Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829369051Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829722448Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829829847Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829980646Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830014945Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830336043Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830379842Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830510341Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.83061504Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.83064324Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.83068014Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830713339Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830832038Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.830962237Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.872116087Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983305542Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVSARV4R9H5JNBCZWM77J3:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=14
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983308342Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVW3XPJZHZKDFZSF1FF262:8dvPq4vuJN7GFbgbkPG2h66sMUJ3Uv46deqC4Znx8Ng value_size=479
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983433541Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVXR3YM8KCHYDHD975MR6B:YiZVXIpzEi1c0E5PxmuGZXdaP9Zv_6TMlqUblytboS0 value_size=548
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983444541Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJ1Y9J9H6YG7SQX1NHAH6J:WmXrpbxKJz4okGoD4bA9ZMfUfX--pdCJKTljfSqRFsQ value_size=15
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983439641Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJY68GB7XKPTM8796536A2:HDYlTVWvn__W8_83QBt5_sgOHrsyRYvA77OGAzthyv8 value_size=24
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983450441Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJQ11VRBGJ1J98M06PX8K7:HDYlTVWvn__W8_83QBt5_sgOHrsyRYvA77OGAzthyv8 value_size=22
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983459141Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJTKM81WKBRWZSPJ3KMFBF:HDYlTVWvn__W8_83QBt5_sgOHrsyRYvA77OGAzthyv8 value_size=25
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983467741Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVVB4JPG5831YZ5XVRD13M:YiZVXIpzEi1c0E5PxmuGZXdaP9Zv_6TMlqUblytboS0 value_size=458
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983442941Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJTKM81WKBRWZSPJ3KMFBF:HDYlTVWvn__W8_83QBt5_sgOHrsyRYvA77OGAzthyv8 value_size=25
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.98356144Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVYG2TZR90CDT119E8B0PS:YiZVXIpzEi1c0E5PxmuGZXdaP9Zv_6TMlqUblytboS0 value_size=488
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.983503641Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVSARV4R9H5JNBCZWM77J3:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=14
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.98352654Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJRSQ823AEKEE2K208ASBY:HDYlTVWvn__W8_83QBt5_sgOHrsyRYvA77OGAzthyv8 value_size=34
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.98353294Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVWZ1J4ANRP40ZM9H0Q5NR:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=15
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.98355264Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQRVWZ1J4ANRP40ZM9H0Q5NR:BhL0gelLKlDlyqM_V3VzzagSiVML8b_UTM0vkiXpGwc value_size=15
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.000562596Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.126834622Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.287735255Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.306764593Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.306770393Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829903691Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.829923791Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.844463267Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.845730156Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.845829055Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.845842455Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.845972654Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.84759694Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858400247Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858426047Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858438947Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858459747Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858636445Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858678345Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858690845Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858730744Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.859498738Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.859617437Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.858659445Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.859669136Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.859728236Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.859731736Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.859783135Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.860549029Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.86155822Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.979992605Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.986782647Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQEJ3SPTC4C30AK7H3W215CA:YiZVXIpzEi1c0E5PxmuGZXdaP9Zv_6TMlqUblytboS0 value_size=526
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.987155144Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQKQFQJJPM50RGV56V0BGHJY:YiZVXIpzEi1c0E5PxmuGZXdaP9Zv_6TMlqUblytboS0 value_size=454
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.987237543Z caller=redis_client.go:245 msg="failed to set item into redis" err="context deadline exceeded" key=P:01GQKQGQX3JA0FY4SZSJQXXYWZ:YiZVXIpzEi1c0E5PxmuGZXdaP9Zv_6TMlqUblytboS0 value_size=524
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:34.996629163Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.045141347Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.311395166Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.315019634Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.318981301Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.448426691Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.454553239Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.505713Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.545066663Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.626929362Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:35+01:00	level=warn ts=2023-01-27T20:13:35.627227559Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.166542381Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.279290135Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.279813231Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.328977206Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.54588917Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.625188945Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.852212443Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.984603867Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.038749509Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.100866498Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.362479319Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.455121832Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.587400508Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.625387485Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.664648651Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.736446641Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.782260551Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.894269799Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.943837678Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.006835742Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.070296303Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.150313523Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.158463153Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213482886Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213512586Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213527385Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213493186Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213557185Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213538985Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.213604985Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.240913353Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.312525444Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.762149003Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.783738318Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.81268247Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:35.948499706Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.041955205Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	level=warn ts=2023-01-27T20:13:36.150895372Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:36+01:00	panic: runtime error: index out of range [1] with length 1
2023-01-27T21:13:36+01:00	
2023-01-27T21:13:36+01:00	goroutine 1752852 [running]:
2023-01-27T21:13:36+01:00	github.com/rueian/rueidis.clientMGetCache({0x2c07a80?, 0xc000ce0390?}, {0x2bf4250?, 0xc006d30000?}, 0xc003511920?, {0xc006042ea0?, 0x5e00?, 0xb2d0?}, {0xc000c08040, 0x1, ...})
2023-01-27T21:13:36+01:00		/go/pkg/mod/github.com/rueian/[email protected]/helper.go:42 +0x250
2023-01-27T21:13:36+01:00	github.com/rueian/rueidis.MGetCache({0x2c07a80, 0xc000ce0390?}, {0x2bf4250, 0xc006d30000}, 0xc000ab1740?, {0xc000c08040, 0x1, 0x1})
2023-01-27T21:13:36+01:00		/go/pkg/mod/github.com/rueian/[email protected]/helper.go:21 +0x198
2023-01-27T21:13:36+01:00	github.com/thanos-io/thanos/pkg/cacheutil.(*RedisClient).GetMulti(0xc00052e2c0, {0x2bf4288, 0xc003511920}, {0xc000c08040, 0x1, 0x1})
2023-01-27T21:13:36+01:00		/app/pkg/cacheutil/redis_client.go:287 +0x167
2023-01-27T21:13:36+01:00	github.com/thanos-io/thanos/pkg/store/cache.(*RemoteIndexCache).FetchMultiPostings(0xc000ab1650, {0x2bf4288, 0xc003511920}, {0x1, 0x85, 0xf3, 0xd5, 0x6a, 0x5c, 0x43, ...}, ...)
2023-01-27T21:13:36+01:00		/app/pkg/store/cache/memcached.go:90 +0x1a2
2023-01-27T21:13:36+01:00	github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings(0xc00053e300, {0x2bf4288, 0xc003511920}, {0xc0033bfa20?, 0x1, 0x1}, {0x2bdb280, 0xc003511c50})
2023-01-27T21:13:36+01:00		/app/pkg/store/bucket.go:2243 +0x23c
2023-01-27T21:13:36+01:00	github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings(0xc00053e300, {0x2bf4288, 0xc003511920}, {0xc003ca8088, 0x1, 0xc001d543c0?}, {0x2bdb280, 0xc003511c50})
2023-01-27T21:13:36+01:00		/app/pkg/store/bucket.go:2094 +0x77b
2023-01-27T21:13:36+01:00	github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings(0xc00184bb80, {0xc003ca8088?, 0x1, 0x1}, {0x2bdb2c0, 0xc003511ce0})
2023-01-27T21:13:36+01:00		/app/pkg/store/bucket.go:899 +0x7a
2023-01-27T21:13:36+01:00	github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func2()
2023-01-27T21:13:36+01:00		/app/pkg/store/bucket.go:1233 +0x379
2023-01-27T21:13:36+01:00	golang.org/x/sync/errgroup.(*Group).Go.func1()
2023-01-27T21:13:36+01:00		/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
2023-01-27T21:13:36+01:00	created by golang.org/x/sync/errgroup.(*Group).Go
2023-01-27T21:13:36+01:00		/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.240856771Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.24108577Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.413977825Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.424182058Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443766728Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443802828Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443796928Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443776828Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443820428Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443856828Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443842728Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443897927Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443838628Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443852528Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443851728Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737295786Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737312186Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737330486Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737350086Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737357085Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737351185Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737369285Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737379285Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737383985Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737382085Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.737412485Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755475066Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755494165Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755527365Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755519665Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755533065Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755537465Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755632865Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755655664Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755667564Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755741264Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.755814463Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.846251465Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443922627Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443965927Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443994226Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443936727Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444000926Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444022726Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444041026Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444062326Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444065626Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444058226Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444079826Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444047826Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444111425Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.443972527Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444125525Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444123925Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444006626Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444144525Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444164025Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	level=warn ts=2023-01-27T20:13:36.444154425Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0
2023-01-27T21:13:37+01:00	panic: runtime error: index out of range [1] with length 1
2023-01-27T21:13:37+01:00	
2023-01-27T21:13:37+01:00	goroutine 1533308 [running]:
2023-01-27T21:13:37+01:00	github.com/rueian/rueidis.clientMGetCache({0x2c07a80?, 0xc000c82090?}, {0x2bf4250?, 0xc004d58ae0?}, 0xc01149cf00?, {0xc004fc05b8?, 0x5e00?, 0xb2d0?}, {0xc0061827f0, 0x1, ...})
2023-01-27T21:13:37+01:00		/go/pkg/mod/github.com/rueian/[email protected]/helper.go:42 +0x250
2023-01-27T21:13:37+01:00	github.com/rueian/rueidis.MGetCache({0x2c07a80, 0xc000c82090?}, {0x2bf4250, 0xc004d58ae0}, 0xc000d04900?, {0xc0061827f0, 0x1, 0x1})
2023-01-27T21:13:37+01:00		/go/pkg/mod/github.com/rueian/[email protected]/helper.go:21 +0x198
2023-01-27T21:13:37+01:00	github.com/thanos-io/thanos/pkg/cacheutil.(*RedisClient).GetMulti(0xc0008e2420, {0x2bf4288, 0xc01149cf00}, {0xc0061827f0, 0x1, 0x1})
2023-01-27T21:13:37+01:00		/app/pkg/cacheutil/redis_client.go:287 +0x167
2023-01-27T21:13:37+01:00	github.com/thanos-io/thanos/pkg/store/cache.(*RemoteIndexCache).FetchMultiPostings(0xc000d04810, {0x2bf4288, 0xc01149cf00}, {0x1, 0x85, 0xf4, 0x7f, 0xff, 0xc5, 0xcb, ...}, ...)
2023-01-27T21:13:37+01:00		/app/pkg/store/cache/memcached.go:90 +0x1a2
2023-01-27T21:13:37+01:00	github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings(0xc00735e750, {0x2bf4288, 0xc01149cf00}, {0xc0034a8720?, 0x1, 0x1}, {0x2bdb280, 0xc01149d230})
2023-01-27T21:13:37+01:00		/app/pkg/store/bucket.go:2243 +0x23c
2023-01-27T21:13:37+01:00	github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings(0xc00735e750, {0x2bf4288, 0xc01149cf00}, {0xc00028a4e0, 0x1, 0xc0013575c0?}, {0x2bdb280, 0xc01149d230})
2023-01-27T21:13:37+01:00		/app/pkg/store/bucket.go:2094 +0x77b
2023-01-27T21:13:37+01:00	github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings(0xc007276420, {0xc00028a4e0?, 0x1, 0x1}, {0x2bdb2c0, 0xc01149d2c0})
2023-01-27T21:13:37+01:00		/app/pkg/store/bucket.go:899 +0x7a
2023-01-27T21:13:37+01:00	github.com/thanos-io/thanos/pkg/store.(*BucketStore).Series.func2()
2023-01-27T21:13:37+01:00		/app/pkg/store/bucket.go:1233 +0x379
2023-01-27T21:13:37+01:00	golang.org/x/sync/errgroup.(*Group).Go.func1()
2023-01-27T21:13:37+01:00		/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
2023-01-27T21:13:37+01:00	created by golang.org/x/sync/errgroup.(*Group).Go
2023-01-27T21:13:37+01:00		/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5
2023-01-27T21:13:38+01:00	level=warn ts=2023-01-27T20:13:37.281607584Z caller=redis_client.go:289 msg="failed to mget items from redis" err="context deadline exceeded" items=0

@rueian
Copy link
Collaborator

rueian commented Feb 1, 2023

Hi @verejoel,

The bug should be fixed by #170. I will later open a PR to Thanos for updating rueidis as well.

This bug won't be triggered if you have client-side caching enabled with Redis >= 6. So, if you can't wait for a new Thanos release, upgrading your Redis should also work around the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants