Faster filter

[averagehat]

Closes #204
This changes ngs_filter to:

• Use generators rather than list
• Do the filtering only once rather than filtering by index and then by Ns (which requires iterating twice)
• Change all calls to print to log.info or log.debug
• The call to multiprocessing.Pool should utilize threads from config
• If dropNs=false and indexqualitymin=0, then just symlinks the data

In theory this should be significantly faster. But it still looks at every sequence and index sequence, and biopython will load and convert quality, etc. Further optimizations would be possible by skipping that process

[necrolyte2]

So I don't get how it does the diff, but this branch is 3 commits ahead and 61 commits behind master.
This scares me a bit becuase github is not showing us that inside the PR.

If you switch to the branch on the code page on GitHub it shows that.

[averagehat]

Yeah, weird. I'll look into this tomorrow.

[averagehat]

Apparently my faster code makes the tests run too slow? lol https://travis-ci.org/VDBWRAIR/ngs_mapper/builds/118045722#L2242

edit: Indeed, my "faster" code was written in a delusional state, apparently. It's excessively slow, and not lazy at all because of the use of reduce.

[averagehat]

It's possible this use of itertools.chain is slow.

edit: it's very slow.

[averagehat]

Assuming the second build passes, this is ready for review. If not I'll fix it up monday.

[necrolyte2]

• Change all calls to print to log.info or log.debug
• The call to multiprocessing.Pool should utilize threads from config
• How hard would it be to detect if no filter arguments were specified to just symlink and return?

[averagehat]

Weird unrelated test failure:

https://travis-ci.org/VDBWRAIR/ngs_mapper/builds/119042228#L2233-L2242

Note that slow_tests pass anyway

[averagehat]

Wow that failure was literally a fluke: https://travis-ci.org/VDBWRAIR/ngs_mapper/builds/119050624#L2233-L2235

[necrolyte2]

This looks like it would work fine, but I'm wondering why you took this approach instead of just pool = multiprocessing.Pool(threads)

[averagehat]

I didn't know about that option