High speed messages sending to Redis with confirmation.

[ghetze]

Summary: What are you wanting to achieve?


Hi.
I'm currently making some tests to see the performance when sending lots of records no a redis db.
The idea is to send 30 000 000 records to redis as fast as possible and then to wait for all the results to see how many of them crashed or not.
I'm using polly because we need also a retry mechanism.
For the sending i use stackexchange.redis Task StringSetAsync (......) method.
Is my code ok ? or there is a fastest way to accomplish the highest sending speed.

What code or approach do you have so far?

private void TestPerformane()
        {
            int _retryCount = 10;
            int _retryIntervalSeconds = 20;
            try
            {
                int totalSuccessDeliveries = 0;
                int totalDeliveries = 0;
                var _sendMessagesToRedisWatcher = new Stopwatch();
                _sendMessagesToRedisWatcher.Start();
                int recordsNo = 0;
                for (int i = 1; i <= recordsNo; i++)
                {
                    try
                    {
                        var rslt =
                            Policy
                            .Handle<RedisConnectionException>()
                            .WaitAndRetryAsync(_retryCount, inc => TimeSpan.FromSeconds(_retryIntervalSeconds), (result, timeSpan, retryCount, context) =>
                            {
                                logger.Warn($"Request failed.Waiting {timeSpan} sec. Retry attempt {retryCount}");
                                LogEntries.Add(new LogEntry { Message = $"Request failed.Waiting {timeSpan} sec. Retry attempt {retryCount}", DateTime = DateTime.Now });
                            })
                            .ExecuteAsync
                            (
                                () => DB.StringSetAsync(i.ToString(), i.ToString())
                            );

                        rslt.ConfigureAwait(false);
                        rslt.ContinueWith(task =>
                        {
                            if (!task.IsFaulted)
                            {
                                Interlocked.Increment(ref totalSuccessDeliveries);
                            }

                            if (task.Exception != null)
                                logger.Error(task.Exception.InnerException.Message);

                            Interlocked.Increment(ref totalDeliveries);
                        });
                    }
                    catch (Exception ex)
                    {
                        logger.Error($"Failed after {_retryCount} retries" + ex.Message);
                    }
                }

                //wait until all records were sent to redis
                while (totalDeliveries < recordsNo)
                {
                    Thread.Sleep(1000);
                }
            }
            catch (Exception ex)
            {
                logger.Error(ex.Message);
            }

( EDITED by @reisenberger for code layout only )

[reisenberger]

You can declare the policy just once, outside the loop and re-use it within the loop (this will save a large number of allocations and garbage collections):

// outside loop
var policy = Policy
    .Handle<RedisConnectionException>()
    .WaitAndRetryAsync(/* etc */);

// inside loop
var rslt = policy.ExecuteAsync(...)

You can remove the .ConfigureAwait(false). Since you are never await-ing that task, it adds no value.

You can consider batching with Redis as an alternative to pipelining. You would have to benchmark whether it made any difference for your use case.

Googling reveals a number of existing discussions around your use case:

• https://stackoverflow.com/questions/27796054/pipelining-vs-batching-in-stackexchange-redis
• https://stackoverflow.com/questions/44585100/batch-set-data-from-dictionary-into-redis
• Does it support mass insertion? StackExchange/StackExchange.Redis#432

(and there are without doubt more).

EDIT: link to related question on StackExchange.Redis

[ghetze]

Thank you for the response.