Bulk API doesnt break up actions

[angelo0000]

I am trying to upgrade to the latest MongoJS as well as Mongo 3.0. When I run code that I have had in production for 2yrs I get an error on doing bulk updates.

{ name: 'MongoError',
message: 'exceeded maximum write batch size of 1000',
ok: 0,
code: 16,
errmsg: 'exceeded maximum write batch size of 1000' }

Shouldn't mongojs be managing breaking that up or the underlying driver at least?

[angelo0000]

Example Code I have:

bulk.find({ memberId: memberId }).updateOne({ $set: updateData });

memberId is a number and updateData is just an json Object of field/values. I may have 5-10k of these in a single batch that gets executed. In my previous version of MongoJs this works just fine.

[saintedlama]

Had the same problems with exceeded batch size and circumvented this by implementing a module splitting my data into chunks

var async = require('async');
var util  =require('util');

const CHUNK_SIZE = 1000;

module.exports = function(collection, data, next) {
  if (!data) {
    return next();
  }

  if (!util.isArray(data)) {
    return collection.insert(data, next);
  }

  var chunkCount = Math.ceil(data.length / CHUNK_SIZE);
  var chunks = [];

  for (var i=0;i<chunkCount;i++) {
    chunks.push(data.slice(i*CHUNK_SIZE, i*CHUNK_SIZE + CHUNK_SIZE));
  }

  async.forEachSeries(chunks, function(chunk, next) {
    collection.insert(chunk, next);
  }, next);
};

We're currently working on a new major mongojs release based on mongodb-native (instead of mongodb-core) driver to get these problems fixed. For more detail see #256

[angelo0000]

Maybe it is best I just don't update to a 1.x version. I am runing .18.1 now and it uses the native driver which seems to not do the 1000 bulk issue

[angelo0000]

Does 0.18.1 support mongo 3?

[saintedlama]

Should be fixed in version 2.0 (released just some seconds ago)

[saintedlama]

Just tested this out with mongojs 2.0.0 - works like a charm. The underlying driver breaks up bulks 👍

[angelo0000]

2.0 breaks for me too. It now throws different issues in other areas: Maximum call stack exceeded.

/express/node_modules/mongojs/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:247
throw err;
^
RangeError: Maximum call stack size exceeded

[saintedlama]

Did you test with my wrapper or without. The wrapper will not work in 2.0 - could insert an array of 10000 items without bulk exceeded errors or the like

[angelo0000]

I am using your MongoJS wrapper - did not change any code just updated the module to mongojs 2.0. I never got to the point of doing the insertions.. I got the call stack exceeded before that.

[saintedlama]

Related to #270

[saintedlama]

Fixed in newest mongodb-core release. You'll need to npm install mongojs to get the newest version of mongodb-core driver 🎉

[lionvs]

Got the same error in the newest version of mongojs.

MongoJS version: "2.3.0"

Error:

{ [MongoError: exceeded maximum write batch size of 1000]
  name: 'MongoError',
  message: 'exceeded maximum write batch size of 1000',
  ok: 0,
  code: 16,
  errmsg: 'exceeded maximum write batch size of 1000' }

Code example:

var bulk = collection.collection('test').initializeUnorderedBulkOp();
for(var i=0; i<1033; i++){
    bulk.insert({a: 'a'});
}
Promise.fromNode(function(callback){
    bulk.execute(callback)
})
.then(function(data){
    console.log(data);
})
.catch(function(err){
    console.log(err);
});

[saintedlama]

@lionvs there is still a bug in bulk API. Thanks for your code - helped me reproduce the problem!

[aPoCoMiLogin]

Any update on it ?