MongoDB Insert slow down after drop big collection.

[SunguckLee]

I am testing just insert performance(There's only insert) of MongoDB + RocksDB.
Sometimes, after dropping big collection (500million document collection generated by sysbench mongodb test tool), Insert performance is dropped drastically and periodically.

This is insert/sec throughput from mongostat.

insert	query	update	delete	getmore	command
33130	*0	*0	*0	0	37|0	6
26741	*0	*0	*0	0	30|0	6
33356	*0	*0	*0	0	37|0	6
32758	*0	*0	*0	0	36|0	6
34051	*0	*0	*0	0	37|0	6
33034	*0	*0	*0	0	38|0	6
33856	*0	*0	*0	0	37|0	6
34155	*0	*0	*0	0	39|0	6
32910	*0	*0	*0	0	37|0	6
12993	*0	*0	*0	0	15|0	6
126	*0	*0	*0	0	1|0	6
128	*0	*0	*0	0	4|0	6
151	*0	*0	*0	0	5|0	6
94	*0	*0	*0	0	2|0	6
590	*0	*0	*0	0	11|0	6
367	*0	*0	*0	0	5|0	6
970	*0	*0	*0	0	6|0	6
369	*0	*0	*0	0	3|0	6
225	*0	*0	*0	0	2|0	6
380	*0	*0	*0	0	11|0	6
467	*0	*0	*0	0	10|0	6
513	*0	*0	*0	0	4|0	6
247	*0	*0	*0	0	4|0	6
364	*0	*0	*0	0	3|0	6

Sometimes this perf-drop is keeping 5 minutes. And I have found these at slow down.

1. No slowdown or stall by L0 compaction delay.
2. Slow down is happened in several seconds after manual compaction is called
   (2018-01-07T23:20:51.450+0900 D STORAGE [RocksCompactionThread] Starting compaction of range: 00000007 .. 000000075A51F65100008E0E (rangeDropped is 0)).

Now big collection drop, this huge perf-slowdown is not happened. So I think this is caused by (huge collection drop + manual compaction by MongoRocks).

I have used level-compaction and this is automatic (I think), then..

1. Why do we need manual compaction from MongoRocks (https://github.com/mongodb-partners/mongo-rocks/blob/master/src/rocks_compaction_scheduler.cpp#L248-L265) ?
2. And is this really caused by big collection drop ? and is this expected slow down ?
3. Is there any way to avoid this manual compaction ?

This is stacktrace at the moment of slowdown.
mongorocks-slowdown-stack.txt