poor performance on pipeline with buffers

[GGeorgi]

Hi everyone.
I'm trying to insert into redis about 15k buffers but getting poor performance.
But if I try to insert same amount of data using JSON.stringify it works fast.

const pipe = redis.pipeline();
users.map(user => {
   pipe.set(`{users}:${user.id}`, UserModel.encode(user));
});
pipe.exec();
console.log("done");

on pipe.exec(); code gets freeze about for 10 seconds.

Update:
UserModel is protobuf. But this does not matters: Buffer.from(JSON.stringify(user)) is also same.

[knoxcard]

@luin - any thoughts?

[tuananh]

I traced it down and it has something to do with the recently introduced MixedBuffer class that replace the old flex-buffer it seems.

I tried the version 4.9.1 (prior to removing flex-buffer), the performance is much better.

@GGeorgi @knoxcard can you try 4.9.1 to see if it fixes your problem?

[knoxcard]

@GGeorgi - which ioredis version are you on?

Also, are you using Yarn or NPM?

[GGeorgi]

@knoxcard @tuananh
ioredis: 4.14.1
npm: 6.4.1

[knoxcard]

@tuananh - @GGeorgi is using the latest version.

[GGeorgi]

@tuananh performance seems ok at 4.9.1 version

[luin]

Hi @GGeorgi, Could you provide a reproducible example so I can test on my machine? I used the following code and the performance is good (v4.14.1):

const Redis = require('ioredis')
const faker = require('faker')

const redis = new Redis()

// Create a buffer from an object
function encode(obj) {
  return Buffer.from(JSON.stringify(obj))
}

// Generate some fake users
const users = new Array(150000).fill(0).map(() => (
  {
    id: Math.random() * 100000 | 0,
    name: faker.name.findName(),
    email: faker.internet.email(),
    avatar: faker.internet.avatar(),
    ip: faker.internet.ip(),
    phone: faker.phone.phoneNumber(),
    company: faker.company.companyName(),
    createdAt: faker.date.past()
  }
))

const pipe = redis.pipeline();
users.map(user => {
  pipe.set(`{users}:${user.id}`, encode(user));
});

console.log('Start #exec()')
console.time('exec')
pipe.exec();
console.timeEnd('exec')

Prints:

Start #exec()
exec: 38.897ms

[tuananh]

@luin because you did not await p.exec() there

in v4.9.1 even if you await p.exec(), it's still a lot faster.

[knoxcard2]

#991

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.