Pipelining is very slow

[baverman]

It seems each operation leads to separate sendto syscall. Correct me if I wrong but the whole idea behind pipelining is to pack multiple operations into one network send.

[paul-nameless]

Test file

import sys
import redis
import aioredis


N = 100


async def test0(loop):
    r = await aioredis.create_redis_pool('redis://localhost:6379', loop=loop)
    multi = r.pipeline()
    for i in range(N):
        multi.sismember('key', 'value')
    result = await multi.execute()
    print('Result0', result)


def test1():
    r = redis.Redis()
    multi = r.pipeline(False)
    for i in range(N):
        multi.sismember('key', 'value')
    result = multi.execute()
    print('Result1', result)


def main():
    if sys.argv[-1] == '1':
        print('Test1')
        test1()
    elif sys.argv[-1] == '0':
        print('Test0')
        loop = asyncio.get_event_loop()
        loop.run_until_complete(test0(loop))


if __name__ == '__main__':
    main()

Before patching:

 16.49    0.003046          30       101           sendto
# strace -c python ../test.py 1 2>&1 | grep sendto
  0.17    0.000027          14         2           sendto

After patching:

  0.67    0.000181          91         2           sendto
# strace -c python ../test.py 0 2>&1 | grep sendto
  0.10    0.000013           7         2           sendto

The problem
According to redis documentation: https://redis.io/topics/pipelining
when you use pipeline you send command in one step, so we can reduce the latency cost due to the round trip time.
As you can see in example above before patching it had 101 sendto calls and after patching just 2.

Fix
#467

[gjcarneiro]

It seems each operation leads to separate sendto syscall.

I am not seeing this in my environment. What I see is that the entire pipeline is a single writev system call with lots of little data segments, one per redis operation in the pipeline. This seems reasonably efficient. At least not as bad as multiple sendto calls.

aioredis 1.1.0, Python 3.7, uvloop 0.11.2 here.

Ah, I see, disabling uvloop, vanilla asyncio, I see the multiple sendto calls in strace.

[popravich]

Please check this in master.