Limit size of HeronTupleSet. #2253

objmagic · 2017-08-28T19:00:07Z

Fix #2234. We limit the size of HeronTupleSet. If it is larger than the maximum size, we release it back to allocator instead of memory pool.

Tested on local machine.

srkukarni · 2017-08-28T19:02:04Z

heron/stmgr/src/cpp/manager/stmgr-server.cpp

@@ -431,7 +433,15 @@ void StMgrServer::HandleTupleSetMessage(Connection* _conn,
        ->incr_by(_message->control().fails_size());
  }
  stmgr_->HandleInstanceData(iter->second, instance_info_[iter->second]->local_spout_, _message);
-  __global_protobuf_pool_release__(_message);
+  auto message_size = _message->ByteSize();


ByteSizeLong?

I could not find ByteSizeLong API in generated code. Let me try again...

seems this is introduced in 3.1.0 and then deprecated in 3.4.0

also, ByteSize calculates size of serialized message, which seems will be slightly larger than actual size in memory.

so if the message is 60MB but only little part of it is used to hold the incoming message (because incoming message is small), we will not delete it because I believe serialized size is much smaller than 60MB. i'm not sure about

objmagic · 2017-08-28T21:11:54Z

Did some experiment:

  LOG(INFO) << _message->ByteSize();
  LOG(INFO) << _message->SpaceUsed();
  _message->Clear();
  LOG(INFO) << _message->ByteSize();
  LOG(INFO) << _message->SpaceUsed();

gives us

I0828 14:10:23.306396 2502742976 stmgr-server.cpp:437] 97
I0828 14:10:23.306404 2502742976 stmgr-server.cpp:438] 832
I0828 14:10:23.306416 2502742976 stmgr-server.cpp:440] 0
I0828 14:10:23.306422 2502742976 stmgr-server.cpp:441] 832

srkukarni · 2017-08-28T21:14:50Z

Thats interesting obs. So ByteSize is probably more related to wire format size and SpaceUsed is related to actual in memory repr. In that case, shouldn't you use SpaceUsed? But isn;t that very slow?

objmagic · 2017-08-28T21:29:32Z

@srkukarni It seems that we have no choice but to use SpaceUsed here. We can do some benchmark here to measure its performance.

huijunw · 2017-08-28T21:56:14Z

If the performance is a concern, I would suggest the old method: put it in mempool and run a garage collection against mempool to remove large tuples every 1 min.

objmagic · 2017-08-28T22:24:29Z

Some benchmarking for SpaceUsed. Size is in bytes. It looks not bad @srkukarni

I0828 15:22:59.759600 2502742976 stmgr-server.cpp:439] size: 1105717
I0828 15:22:59.759625 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:22:59.967779 2502742976 stmgr-server.cpp:439] size: 1250892
I0828 15:22:59.967838 2502742976 stmgr-server.cpp:440] time: 10 microseconds
I0828 15:23:00.128224 2502742976 stmgr-server.cpp:439] size: 1250892
I0828 15:23:00.128247 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:23:00.290628 2502742976 stmgr-server.cpp:439] size: 1250892
I0828 15:23:00.290647 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:00.485448 2502742976 stmgr-server.cpp:439] size: 2150892
I0828 15:23:00.485477 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:00.642523 2502742976 stmgr-server.cpp:439] size: 2150892
I0828 15:23:00.642571 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:23:00.818881 2502742976 stmgr-server.cpp:439] size: 2150892
I0828 15:23:00.818902 2502742976 stmgr-server.cpp:440] time: 5 microseconds
I0828 15:23:00.970676 2502742976 stmgr-server.cpp:439] size: 3100892
I0828 15:23:00.970695 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:01.183621 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.183640 2502742976 stmgr-server.cpp:440] time: 3 microseconds
I0828 15:23:01.350108 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.350129 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:01.512719 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.512740 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:01.721442 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.721462 2502742976 stmgr-server.cpp:440] time: 3 microseconds
I0828 15:23:01.896715 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:01.896742 2502742976 stmgr-server.cpp:440] time: 7 microseconds
I0828 15:23:02.010160 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.010188 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:02.057590 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.057623 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:02.272822 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.272848 2502742976 stmgr-server.cpp:440] time: 6 microseconds
I0828 15:23:02.437445 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.437464 2502742976 stmgr-server.cpp:440] time: 4 microseconds
I0828 15:23:02.597956 2502742976 stmgr-server.cpp:439] size: 4000892
I0828 15:23:02.598012 2502742976 stmgr-server.cpp:440] time: 8 microseconds

srkukarni · 2017-08-28T22:31:21Z

Could you also share the thruput figures? Particularly in exclamation or other topologies that are used for Heron benchmarking?

objmagic · 2017-08-29T05:39:53Z

word count, parallelism=20

Stmgr CPU user time doubled, Data Tuples from Instances dropped ~25%.

We need to fix #1908 first to see more metrics.

Limit size of HeronTupleSet.

9105695

objmagic requested a review from srkukarni August 28, 2017 19:00

srkukarni reviewed Aug 28, 2017

View reviewed changes

Really assign.

0ce41d5

Use SpaceUsed.

459f7f5

objmagic mentioned this pull request Aug 29, 2017

add back METRIC_BYTES_FROM_STMGRS #2035

Closed

objmagic added 2 commits August 29, 2017 12:05

Call SpaceUsed less often.

4558b1d

Check every 1024 times.

c7ec49c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit size of HeronTupleSet. #2253

Limit size of HeronTupleSet. #2253

objmagic commented Aug 28, 2017

srkukarni Aug 28, 2017

objmagic Aug 28, 2017

objmagic Aug 28, 2017

objmagic Aug 28, 2017

objmagic Aug 28, 2017 •

edited

Loading

objmagic commented Aug 28, 2017

srkukarni commented Aug 28, 2017

objmagic commented Aug 28, 2017

huijunw commented Aug 28, 2017

objmagic commented Aug 28, 2017 •

edited

Loading

srkukarni commented Aug 28, 2017

objmagic commented Aug 29, 2017

Limit size of HeronTupleSet. #2253

Are you sure you want to change the base?

Limit size of HeronTupleSet. #2253

Conversation

objmagic commented Aug 28, 2017

srkukarni Aug 28, 2017

Choose a reason for hiding this comment

objmagic Aug 28, 2017

Choose a reason for hiding this comment

objmagic Aug 28, 2017

Choose a reason for hiding this comment

objmagic Aug 28, 2017

Choose a reason for hiding this comment

objmagic Aug 28, 2017 • edited Loading

Choose a reason for hiding this comment

objmagic commented Aug 28, 2017

srkukarni commented Aug 28, 2017

objmagic commented Aug 28, 2017

huijunw commented Aug 28, 2017

objmagic commented Aug 28, 2017 • edited Loading

srkukarni commented Aug 28, 2017

objmagic commented Aug 29, 2017

objmagic Aug 28, 2017 •

edited

Loading

objmagic commented Aug 28, 2017 •

edited

Loading