Error with gzipped FASTQ generated by bcl2fastq

[vaofford]

Command:

pycroquet single-guide -g library.tsv -q oe_g2_g6_1_S1_R1_001.fastq.gz -s Sample_1 -o Sample1.pycroquet.tsv

Compressed FASTQ:

pycroquet_fastq_for_keiran/oe_g2_g6_1_S1_R1_001.fastq.gz

Error:

INFO: Number of duplicate guides: 0
INFO: Total unique guides: 120
INFO: uncompressed data (assume fastq)
Traceback (most recent call last):
  File "/opt/wsi-t78/venv/bin/pycroquet", line 33, in <module>
    sys.exit(load_entry_point('pycroquet==1.2.0', 'console_scripts', 'pycroquet')())
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/cli.py", line 216, in wrapper
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/cli.py", line 129, in wrapper
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/cli.py", line 236, in wrapper
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/cli.py", line 265, in single_guide
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/singleguide.py", line 63, in run
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/main.py", line 143, in process_reads
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/readparser.py", line 137, in parse_reads
  File "/opt/wsi-t78/venv/lib/python3.9/site-packages/pycroquet-1.2.0-py3.9.egg/pycroquet/readparser.py", line 269, in parse_fastq
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Worked fine once FASTQ was uncompressed. This was unprocessed output from bcl2fastq version 2.20.0 (/software/CASM/modules/modulefiles/bcl2fastq/2.20.0). Was run as an interactive job on farm5.

[keiranmraine]

It didn't recognise the file as compressed:

INFO: uncompressed data (assume fastq)

Can you run file pycroquet_fastq_for_keiran/oe_g2_g6_1_S1_R1_001.fastq.gz?

(I couldn't access the file you provided the full path for, but also removed the path as it exposed your internal username and data structures on the HPC).

[vaofford]

That gives me:

pycroquet_fastq_for_keiran/oe_g2_g6_1_S1_R1_001.fastq.gz: gzip compressed data, extra field

[keiranmraine]

Okay, so I've emulated what 1.2.0 does on the command line and I can seewhy it doesn't work:

$ module load pycroquet/1.2.0
$ pycroquet-shell 
Singularity> python3
Python 3.9.7 (default, Sep  9 2021, 23:20:13) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import magic
>>> bool("gzip compressed data" in magic.from_file("oe_g2_g6_1_S1_R1_001.fastq.gz"))
False
>>> magic.from_file("oe_g2_g6_1_S1_R1_001.fastq.gz")
'Blocked GNU Zip Format (BGZF; gzip compatible), block length 15484'

While on older python/magic library I get:

$ python3
Python 3.7.4 (default, Aug 13 2019, 14:20:12) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import magic
>>> bool("gzip compressed data" in magic.from_file("oe_g2_g6_1_S1_R1_001.fastq.gz"))
True

Just needs an extra check to allow for gzip compatible.