mempool growth issue

[objmagic]

This issue is from Heron 0.14.11 (in which we are still using HeronTupleSet, unlike master branch).

A topology in Twitter environment has high memory usage. We found out its memory pool is growing without bound. #2073 assumes that each message in mempool is around 8MB. However, we observed one single message becomes extremely large. And it is strange that there is only one message of this type in the memory pool.

N5heron5proto6system13HeronTupleSetE - nums: 1 - bytes used: 657693336

We then printed out the size of each incoming HeronTupleSet in bytes. Here is the increase. It seems we are using this single one message repeated and although we clear it every time we acquire it, this protobuf message object keeps growing.

cc @srkukarni @huijunw

[srkukarni]

Better to have a discussion over hangouts so that we can see what exactly is happening

[objmagic]

a related mailing list thread https://groups.google.com/forum/#!searchin/protobuf/reuse$20memory%7Csort:relevance/protobuf/TSzsFu3DPmQ/9BjFXpkdAc0J