Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SENTINEL GET-MASTER-ADDR-BY-NAME newsrecdoc19 returned failed. #135

Closed
yyhclimacool opened this issue Dec 27, 2020 · 6 comments
Closed

SENTINEL GET-MASTER-ADDR-BY-NAME newsrecdoc19 returned failed. #135

yyhclimacool opened this issue Dec 27, 2020 · 6 comments

Comments

@yyhclimacool
Copy link

yyhclimacool commented Dec 27, 2020

I'm using redis-plus-plus compiling from latest master branch code in SENTINEL mode.
I got the exception of "Failed to create connection with sentinel".
and it's because "StopIterError" exception or "Error" exception.
so I did some logging.
and deeply it's because "_get_master_addr_by_name(sentinel, master_name)" throwed an exception.
and inside this function, it's because "reply::parse<Optional<std::pair<std::string, std::string>>>(*reply)" throwed an exception, and reply.elements != 2,so exception is "NOT key-value PAIR reply".

the redis server is OK, I checked by run redis-cli.

I'm running program in docker env, base image is from "tensorflow/tensorflow:1.15.0-gpu-py3", I don't think docker env has anything to with these exceptions.

OS : Ubuntu 18.04.3 LTS
hiredis : release v1.0.0
gcc : gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

the weired thing is, I have this sentinel in a seperate single threaded test program, and it get redis key successfully.

any idea on this problem ?

@sewenew
Copy link
Owner

sewenew commented Dec 28, 2020

@yyhclimacool That's strange. SENTINEL GET-MASTER-ADDR-BY-NAME master-name should return the master node's ip and port as an array reply, and reply.elements should equal to 2. If the master-name does not exist, sentinel should return a NIL reply. That's why the code tries to parse the reply as Optional<std::pair<std::string, std::string>>. Check this for detail.

However, your debugging shows that it's not a valid reply. Can you use redis-cli to connect to the sentinel node, and send SENTINEL GET-MASTER-ADDR-BY-NAME master-name command to check the return value?

the weired thing is, I have this sentinel in a seperate single threaded test program, and it get redis key successfully.

Do you use redis-plus-plus to run this test program?

Regards

@yyhclimacool
Copy link
Author

Thank you for your reply.

the weired thing is, I have this sentinel in a seperate single threaded test program, and it get redis key successfully.
Do you use redis-plus-plus to run this test program?
yes, I used redis-plus-plus in this program. and it runs fine.

I also checked with redis-cli, and it runs fine.

I also used tcpdump to capture network traffic, I both captured redis-cli command network traffic and my_program network traffic, they got the same pattern, like this:
image

but in my_program, redis-plus-plus tells me that reply.elements of SENTINEL GET-MASTER-ADDR-BY-NAME master-name is 0, but 2. How weired is that.

I also tried compile hiredis-v1.0.0 and redis-plus-plus with -O0/-O1/-O2/-fPIC options and mix them, still get the same exception NOT key-value PAIR reply.

thanks again, regards.

@sewenew
Copy link
Owner

sewenew commented Dec 29, 2020

So you only got the problem in a multiple threaded environment? It's weird!

Can you show me your code that can reproduce the problem? Even better, if you can also give me a docker image that can reproduce it. So that I can do some debugging.

Regards

@mfatemipour
Copy link

I have the same issue ...

@yyhclimacool
Copy link
Author

Sorry, I almost forgot this issue.
My problem solved, the reason is I have a mixed use of redis-plus-plus and hiredis-vip, and gcc(ld percisely) somehow mixed these symbols.
solution : I removed all code about hiredis-vip and only use redis-plus-plus, it turns out fine.

@sewenew
Copy link
Owner

sewenew commented Jan 19, 2021

@mfatemipour Can you check if you installed multiple versions of hiredis? Or as @yyhclimacool mentioned, your environment installed hiredis-vip? There's another issue, when multiple versions of hiredis installed, you might get some weird error.

If you only installed one hiredis, please give me your version info (or branch info) on Redis, hiredis and redis-plus-plus. I'll try to reproduce it.

Regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants