Improved memory usage in gCNV.

[samuelklee]

-Changed sampling of denoised copy ratios to address memory spike and updated output formats and filenames. Partially addresses #5754.
-Updated theano version to 1.0.4 and changed numpy install source to conda defaults to enable MKL.
-Updated theano flags to use MKL and OpenMP elemwise.

Closes #5764.

[samuelklee]

Prior to this, gCNV shard size and VM type have never been optimized. For WES cohort mode, we arbitrarily ran with:

37 shards of 200 samples by 5000 intervals on n1-standard-8s (8 CPU, 30GB memory, $0.08 / hr) each taking ~2 hours = ~3 cents / sample

This PR gets rid of a memory spike in the sampling of denoised copy ratios, fixes a memory leak by updating theano, and also adds some theano flags that typically yield a factor of ~2 speedup (notably, the OpenMP elemwise flag, although we also get a slight boost from using numpy MKL). This allows us to run, e.g.:

2 shards of 50 samples by 100000 intervals on n1-standard-8s (8 CPU, 30GB memory, $0.08 / hr) each taking ~5 hours = ~1.6 cents / sample
4 shards of 50 samples by 50000 intervals on n1-highmem-4s (4 CPU, 26GB memory, $0.05 / hr) each taking ~3.25 hours = ~1.3 cents / sample
45 shards of 50 samples by 5000 intervals on n1-standard-1s (1CPU, 3.75GB memory, $0.01 / hr) each taking ~0.5 hours = ~0.5 cents / sample

For these runs, we used a slightly larger interval list and 1/4 the number of samples than in the first example, but because everything scales linearly, it's probably fair to compare the per-sample-and-interval costs. So we get a factor of ~8 savings if we keep the shard size the same.

The cost was already satisfactory, but fixing the leak allows us to more easily run scatters that are not so wide, which may be crucial for running the megaWDL. Adding the OpenMP flag also lets CPU scalability work as intended.

We can do a more systematic optimization for cost if desired, and we should also revalidate to make sure performance doesn't vary too much with shard size (from spot checking, it looks like marginal and/or single-bin calls may flicker on and off).

Note that we have still not optimized inference for WES, although I believe @vruano has done some optimizations for WGS. @mwalker174 @vruano for WGS with 2kb bins, I would expect the cost of the gCNV step to be ~10 cents in cohort mode before inference optimizations, assuming we address #5716 to minimize disk costs.

@asmirnov239 can you review? And maybe you can address dCR output in PostprocessGermlineCNVCalls and expose the number of samples in a separate PR? We can make some further changes to the dCR format there if we need.

[samuelklee]

@lucidtronix @cmnbroad @jamesemery also note that I went ahead and switched over to numpy MKL, I'm assuming you have no objections!

[codecov-io]

Codecov Report

Merging #5781 into master will decrease coverage by 0.001%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##              master     #5781       +/-   ##
===============================================
- Coverage     86.999%   86.998%   -0.001%     
  Complexity     32110     32110               
===============================================
  Files           1974      1974               
  Lines         147249    147249               
  Branches       16218     16218               
===============================================
- Hits          128105    128104        -1     
  Misses         13236     13236               
- Partials        5908      5909        +1

Impacted Files	Coverage Δ	Complexity Δ
...nder/utils/runtime/StreamingProcessController.java	67.299% <0%> (-0.474%)	33% <0%> (ø)

[asmirnov239]

@samuelklee Back to you!

[asmirnov239]

Can you try using approximation.sample_node method instead?

[samuelklee]

Reminder to self to recheck for memory spike as we discussed.

[samuelklee]

No spike and the results are exactly the same when using sample_node. Sorry for not realizing that the other method cribbed from this earlier.

Interestingly, I also checked whether compilation affects memory usage---it doesn't. Compilation can take a non-trivial amount of time for short-running shards, so we might experiment with distributing a precompiled model for fixed shard sizes in production. Not sure if this works in practice or what the effect of getting VMs on different architectures might be, though.

[samuelklee]

Thanks @asmirnov239, back to you!

[asmirnov239]

Looks good @samuelklee! Thanks for fixing it