Memory usage is too high

[pboothe]

No description provided.

[stephen-soltesz]

Memory usage is still too high, causing node failure at less than 10 simultaneous ndt5 download clients.

Using pprof heap profiling, the culprit is the very generous allocation in newTCP() for the buffered channel pchan with 833333 capacity.

Due to ndt5 protocol, every "test" generates two tcp connections that remain allocated for at least 60s, the load test at ~10 streams, will generate around 100 connections / minute (assuming ~12sec / test).

Once there are ~150 streams allocated, that's ~2GB just for pointers.

Writing directly do disk also eliminates the in-memory buffering.

[stephen-soltesz]

Part of https://github.com/m-lab/dev-tracker/issues/442

[stephen-soltesz]

(I'm working on patches for this)

[pboothe]

Let's not wait for the UUID. Either it arrives by the time we stop capturing, or we can assume it will never arrive.

[stephen-soltesz]

While theoretically it may be possible to create a fixed-memory profile for packet-headers (rather than one that grows proportionately with number of connections) the memory usage of the latest version (v0.5.4) is not too high.

I believe this issue is resolved.