Reconnect stops working if server is down #211
Comments
|
Hm, if I remove the following from Once Redis server is back up and running, everything works as expected. EDIT: Another solution that seems to work is calling |
|
Unfortunately I can't reproduce this issue, I get normal reconnection attempts |
|
How are you shutting down the redis server? Since you are disabling health-checks if the server does not send EOF downstream the connection will never notice that redis is down and might behave as if it were connected. |
|
I stop it simply by running However, I observe the same behavior if I run the sample program and the redis server is never started, i.e. no redis server is running. Effectively the program is trying to connect to a inactive/unused port. Could you please also try that scenario on your end?
|
|
Hi,
Ok, that is also what I am doing.
Nothing changed, the reconnect loop runs non-stop until I restart the server.
Debian testing + gcc 14.2.0
That would be really helpful. Thanks. |
|
Created a batteries-included Dockerfile (please see attachment Dockerfile_redis_reproduction.txt) that makes a fresh executable that replicates this behavior.
I built it as you normally would, e.g. tagged with redis_reproduction |
|
Many thanks. I will try it when I get some free time, that might take a week in my current schedule. |
cfg.health_check_interval = {}; //Disable PING
cfg.reconnect_wait_interval = std::chrono::seconds(3); //Increase retry timebtw, what is the point of disabling health-checks but enabling reconnects? Both are coupled to each other, with disabled health-checks we can only detect clean disconnects initiated by the server. It will react to a server that freezes for example. |
|
Thanks, hope it is helpful and I didn't do something silly to cause this behavior. No real point to disabling the health checks to be honest, just wanted to disable the implicit health-check to keep the logger a little bit quitter. Once I observed that it does not matter, i.e. the behavior remained the same, I left it disabled to have as few moving parts as possible. |
|
Short feedback: I tried to reproduce this many times but still could not do it. I couldn't build the docker file too. |
|
Thank you for trying to reproduce. I don't know how our environments differ that you do not observe this... At a bit of a loss, especially if you can't get the supplied Dockerfile to build. I just retried a full rebuild of the docker file I attached. It took around half an hour, but it built on the first try for me :\ |
|
I just managed to reproduce this problem on your container and will investigate it further. |
|
Having a similar issue, my environment is Ubuntu, Linux 6.8.0-45-generic x86_64, boost 1.85 with boost cobalt when my app code is started with out the Redis Server running. all I tried @zlausd's solution and it did help resolve this.
source code for my calls to boost::redis: program log; health_check_id here is unnamed: |
|
If you are using a local redis running on a container: I was able to reproduce like this: #include <boost/asio.hpp>
#include <boost/redis.hpp>
#include <boost/redis/connection.hpp>
#include <boost/redis/src.hpp>
#include <iostream>
#include <semaphore>
void Test(boost::redis::connection& connection,
boost::asio::io_context& io_context,
std::counting_semaphore<1>& semaphore,
int counter) {
auto request = std::make_shared<boost::redis::request>();
request->push("SET", "key", "value");
auto response = std::make_shared<boost::redis::response<std::string>>();
io_context.post([&connection, request, response, &semaphore, counter]() {
std::cout << "post " << counter << std::endl;
connection.async_exec(
*request, *response,
[request, response, &semaphore, counter]
(const boost::system::error_code& error, std::size_t) {
if (error) {
std::cout << "Failed to execute: " << error.to_string() << std::endl;
}
std::cout << "Done for " << counter << std::endl;
semaphore.release();
});
});
}
int main() {
boost::asio::io_context io_context{1};
std::thread thread;
// Set up.
boost::redis::connection connection(io_context);
boost::redis::config config;
connection.async_run(config, {boost::redis::logger::level::info},
[](const boost::system::error_code& error) {
if (error) {
std::cout << "Redis server setup error: " << error.to_string() << std::endl;
}
});
thread = std::thread([&io_context] {
std::cout << "RedisGlobalCache running" << std::endl;
io_context.run();
std::cout << "RedisGlobalCache finished" << std::endl;
});
std::vector<std::unique_ptr<std::counting_semaphore<1>>> semaphores;
for (int it = 0; it < 3; ++it) {
semaphores.push_back(std::make_unique<std::counting_semaphore<1>>(0));
Test(connection, io_context, *semaphores[it], it);
}
// Stop.
system("docker stop server_local_redis");
std::cout << "Stopping local redis server" << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(10));
// Add more work.
std::counting_semaphore<1> last_semaphore(0);
Test(connection, io_context, last_semaphore, 10);
// Start.
system("docker start server_local_redis");
std::cout << "Starting local redis server" << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(1));
for (auto& semaphore : semaphores) {
semaphore->acquire();
}
last_semaphore.acquire();
io_context.post([&connection] { connection.cancel(); });
thread.join();
std::cout << std::endl << std::endl;
return 0;
}Compiled with:
Output was: It hangs at the end as no reconnections are attempted again and there are still pending operations. Adding |
Using the (I downloaded the include/boost and just copied it to my boost include folder) |
The connector class is not a template anymore, can you please check the connector.hpp file was replaced? See
|
My bad! I copied it to the wrong location. It compiled now, and it works with my example! |
|
Related question: I have set |
I am not exactly sure what the code is doing. cancel_if_not_connected is checked only once at the time async_exec is called. that means, if a disconnect occurs after that it won't cause the request to be canceled. If you have more questions about that I suggest opening another issue. |
Thanks for responding, I think I got the gist of it. To clarify: if the connection to the server goes down, all enqueued async_exec will get cancelled if |
My environment is Boost 1.85, GCC 14.2.1 (C++23) and Redis 7.2.5.
Observing a small quirk and not sure if its an issue with my setup or something else.
If I leave the Redis server shutdown and initiate a redis::connection with async_run, then it will only attempt to reconnect two times.
After the second ' Connection refused', nothing else on the connection instance works (i.e. async_exec no longer runs or cancels).
Expected behavior when Redis is down: Perpetual reconnect loop and requests should be failing since
cancel_if_not_connectedis explicitly set to true.Boost/Redis logs with added timestamps
Reproduction source code
The text was updated successfully, but these errors were encountered: