what is ngs_filter doing?

[necrolyte2]

I know what it can do, but the stage takes a long time to run even though nothing is selected for it to do.

Not a huge deal, but from an outsider perspective it is confusing

[averagehat]

I changed ngs_filter to simply copy the file over if the settings are set such that nothing would be done. Is this what you're looking for?

[necrolyte2]

I think we should investigate that stage some more. I saw very high CPU and memory usage while that stage was running which is what concerned me. It also added about 5-10 minutes to the analysis.

I'm wondering if we should consider somehow combining filter and trim out put otherwise you end up with input fastq + filter fastq + trim fastq + bam

Essentially 4x data size

[averagehat]

The code uses lists rather than generators for simplicity.

ngs_mapper/ngs_mapper/nfilter.py

Lines 141 to 158 in 4104557

	reads = list(fq_open(readpath))
	except AssertionError, E:
	print "skipping biopython assertion error"
	print readpath
	#sys.stderr.write(str(E))
	if index and idxQualMin:
	idxreads = fq_open(index)
	intermediate = [r for r, idx in izip(reads, idxreads) if idx_filter(r, idx, idxQualMin) ]
	else:
	intermediate = reads
	if dropNs:
	hasNs = lambda rec: 'N' in str(rec.seq).upper()
	filtered = list(ifilterfalse(hasNs, intermediate))
	else:
	filtered = intermediate
	total, badIndex, hadN = len(reads), len(reads) - len(intermediate), \
	len(intermediate) - len(filtered)
	return filtered, total, badIndex, hadN