-
Notifications
You must be signed in to change notification settings - Fork 700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
redis-shake 4.0集群之间的同步存在数据丢失的情况,且消耗的性能也比2版本大 #787
Comments
|
对比.pdf Keyspacedb0:keys=27020349,expires=0,avg_ttl=0 |
@tentosleep 一种缓解方法是,如果源端有 3 个分片,那么启动 3 个 redis-shake。其中 reader 分别配置为三个源端,writer 配置为目的端集群。可以解决同步慢的问题,内存膨胀问题难解决,现在应该不会很严重,你可以给些数据看看,比如源端内存使用量,shake内存使用量,是否用大 hash、set、list 等结构。 |
@suxb201 好的多谢,内存慢的问题我试一下尝试开三个进程,但这样和老版本想比更为繁琐; Biggest string found '"188683"' has 1219 bytes |
问题描述(Issue Description)
1.使用Redis-shake4.0在数据量较大的场景下存在数据丢失的情况,总内存15G,每个实例两千万左右的key,同步完成后一对主从的key丢失
2.Redisshake4.0同步消耗的主机内存远大于用2.0同步消耗的内存,是正常情况吗
Please provide a brief description of the issue you encountered.
环境信息(Environment)
日志信息(Logs)
如果有错误日志或其他相关日志,请在这里提供。
If there are any error logs or other relevant logs, please provide them here.
执行日志
大数据量情况
{"level":"info","time":"2024-03-29T00:05:57+08:00","message":"not set status port"}
{"level":"info","time":"2024-03-29T00:05:57+08:00","message":"start syncing..."}
{"level":"info","time":"2024-03-29T00:06:02+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:06:07+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:12+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:17+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:06:22+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:27+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:32+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:06:37+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:42+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:47+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:06:52+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:06:57+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:02+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:07:07+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:12+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:17+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:07:22+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:27+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:32+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:07:37+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:42+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:47+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:07:52+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:07:57+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:08:02+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:08:07+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-29T00:08:12+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, receiving rdb"}
{"level":"info","time":"2024-03-29T00:08:17+08:00","message":"read_count=[320289], read_ops=[64766.12], write_count=[320288], write_ops=[64767.12], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:08:22+08:00","message":"read_count=[651989], read_ops=[68279.07], write_count=[651988], write_ops=[68279.07], src-2, receiving rdb"}
{"level":"info","time":"2024-03-29T00:08:27+08:00","message":"read_count=[953445], read_ops=[53125.92], write_count=[953445], write_ops=[53125.92], src-0, syncing rdb, size=[226 MiB/6.6 GiB]"}
{"level":"info","time":"2024-03-29T00:08:32+08:00","message":"read_count=[1206712], read_ops=[50456.36], write_count=[1206711], write_ops=[50455.36], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:08:37+08:00","message":"read_count=[1459741], read_ops=[50093.92], write_count=[1459740], write_ops=[50092.92], src-2, syncing rdb, size=[71 MiB/6.6 GiB]"}
{"level":"info","time":"2024-03-29T00:08:42+08:00","message":"read_count=[1715609], read_ops=[50889.46], write_count=[1715608], write_ops=[50889.46], src-0, syncing rdb, size=[321 MiB/6.6 GiB]"}
{"level":"info","time":"2024-03-29T00:08:47+08:00","message":"read_count=[1978753], read_ops=[52896.65], write_count=[1978752], write_ops=[52895.65], src-1, hand shaking"}
{"level":"info","time":"2024-03-29T00:08:52+08:00","message":"read_count=[2236633], read_ops=[52020.92], write_count=[2236632], write_ops=[52020.92], src-2, syncing rdb, size=[168 MiB/6.6 GiB]"}
{"level":"info","time":"2024-03-29T00:08:57+08:00","message":"read_count=[2489089], read_ops=[49442.99], write_count=[2489088], write_ops=[49441.99], src-0, syncing rdb, size=[418 MiB/6.6 GiB]"}
{"level":"info","time":"2024-03-29T00:09:02+08:00","message":"read_count=[2743574], read_ops=[49838.39], write_count=[2743573], write_ops=[49838.39], src-1, hand shaking"}
小数据量情况
{"level":"info","time":"2024-03-31T15:03:52+08:00","message":"not set status port"}
{"level":"info","time":"2024-03-31T15:03:52+08:00","message":"start syncing..."}
{"level":"info","time":"2024-03-31T15:03:57+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, waiting bgsave"}
{"level":"info","time":"2024-03-31T15:04:02+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave"}
{"level":"info","time":"2024-03-31T15:04:07+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, waiting bgsave"}
{"level":"info","time":"2024-03-31T15:04:12+08:00","message":"read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, waiting bgsave"}
{"level":"info","time":"2024-03-31T15:04:17+08:00","message":"read_count=[33454], read_ops=[0.00], write_count=[33454], write_ops=[0.00], src-2, receiving rdb"}
{"level":"info","time":"2024-03-31T15:04:22+08:00","message":"read_count=[258016], read_ops=[45752.25], write_count=[258015], write_ops=[45753.25], src-0, syncing rdb, size=[123 MiB/1.3 GiB]"}
{"level":"info","time":"2024-03-31T15:04:27+08:00","message":"read_count=[486263], read_ops=[44478.29], write_count=[486262], write_ops=[44478.29], src-1, syncing rdb, size=[165 MiB/1.3 GiB]"}
{"level":"info","time":"2024-03-31T15:04:32+08:00","message":"read_count=[731859], read_ops=[49249.41], write_count=[731858], write_ops=[49248.41], src-2, syncing rdb, size=[309 MiB/1.3 GiB]"}
{"level":"info","time":"2024-03-31T15:04:37+08:00","message":"read_count=[978768], read_ops=[49185.96], write_count=[978768], write_ops=[49185.96], src-0, syncing rdb, size=[462 MiB/1.3 GiB]"}
{"level":"info","time":"2024-03-31T15:04:42+08:00","message":"read_count=[1225308], read_ops=[50100.23], write_count=[1225307], write_ops=[50099.23], src-1, syncing rdb, size=[165 MiB/1.3 GiB]"}
@@@
其他信息(Additional Information)
请提供任何其他相关的信息,如配置文件、错误信息或截图等。
配置文件
Redisshake4配置文件.txt
Please pro
微信图片_20240331141833.pdf
vide any additional information, such as configuration files, error messages, or screenshots.
The text was updated successfully, but these errors were encountered: