Audit memory allocation

[mrBliss]

After letting a node sync the first 540k blocks:

761,473,422,968 bytes allocated in the heap                                                                                                                                 
  63,238,559,440 bytes copied during GC                                                                                                                                        
     128,763,248 bytes maximum residency (292 sample(s))                                                                                                                       
       3,529,360 bytes maximum slop                                                    
             122 MB total memory in use (0 MB lost due to fragmentation)               
                                     Tot time (elapsed)  Avg pause  Max pause          
  Gen  0     724781 colls,     0 par   730.289s  731.350s     0.0010s    0.0256s       
  Gen  1       292 colls,     0 par   11.830s  11.962s     0.0410s    0.0940s          
  TASKS: 7 (1 bound, 6 peak workers (6 total), using -N1)                              
  SPARKS: 0(0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)                       
  INIT    time    0.000s  (  0.000s elapsed)                                    
  MUT     time  773.008s  (709.649s elapsed)                                           
  GC      time  742.119s  (743.313s elapsed)                                           
  EXIT    time    0.001s  (  0.009s elapsed)             
  Total   time  1515.128s  (1452.970s elapsed)                                         
  Alloc rate    985,078,173 bytes per MUT second                                       
  Productivity  51.0% of total user, 48.8% of total elapsed

The productivity is low, but not dramatically low. We should do some allocation profiling to see whether we're not allocating too much at some places.

[edsko]

Related: #1420

[mrBliss]

I recently tried again and got the following after syncing the first 1,758,300 blocks:

3,602,638,107,880 bytes allocated in the heap
  70,881,748,848 bytes copied during GC
     504,668,016 bytes maximum residency (181 sample(s))
       8,258,336 bytes maximum slop
             481 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     113573 colls, 113573 par   243.896s  153.133s     0.0013s    0.0365s
  Gen  1       181 colls,   180 par   31.907s  17.807s     0.0984s    0.2727s

  Parallel GC work balance: 64.05% (serial 0%, perfect 100%)

  TASKS: 16 (1 bound, 15 peak workers (15 total), using -N2)

  SPARKS: 0(0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.001s  (  0.001s elapsed)
  MUT     time  3281.462s  (2470.461s elapsed)
  GC      time  275.802s  (170.940s elapsed)
  EXIT    time    0.001s  (  0.009s elapsed)
  Total   time  3557.266s  (2641.411s elapsed)

  Alloc rate    1,097,876,082 bytes per MUT second

  Productivity  92.2% of total user, 93.5% of total elapsed

This is looking much better already.

[edsko]

Ok, I think this is looking very reasonable. Let's close this and reopen new tickets if we have specific concerns.