Multiprocessing error with .cool file

[IsMM15]

Hello,
I am trying to run the tool with the following command:

python diffdomains.py dvsd multiple /data/T1_10000.cool /data/T2_10000.cool /data/TADS_DM.bed --reso 10000

But after appearing to run fine for a while it throws the following error:

...
chr9:75210000:75430000
chr9:106940000:107100000
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "diffdomains.py", line 66, in
comp2domins_by_twtest_parallel(0)
File "diffdomains.py", line 59, in comp2domins_by_twtest_parallel
fhic0=opts[''], fhic1=opts[''],min_nbin=int(opts['--min_nbin']),f=opts['--f'])
File "/diffdomain-py3/utils.py", line 338, in comp2domins_by_twtest
mat0 = contact_matrix_from_hic(chrn, start, end, reso, fhic0, hicnorm)
File "/diffDomain/diffdomain-py3/utils.py", line 181, in contact_matrix_from_hic
c = cooler.Cooler(f'{hic_norm}')
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 85, in init
self._refresh()
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 89, in _refresh
with open_hdf5(self.store, **self.open_kws) as h5:
File "/home/micromamba/envs/diffDomain3/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/util.py", line 525, in open_hdf5
fh = h5py.File(fp, mode, *args, **kwargs)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in init
swmr=swmr)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/micromamba/envs/diffDomain3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "diffdomains.py", line 59, in comp2domins_by_twtest_parallel
fhic0=opts[''], fhic1=opts[''],min_nbin=int(opts['--min_nbin']),f=opts['--f'])
File "/diffDomain/diffdomain-py3/utils.py", line 338, in comp2domins_by_twtest
mat0 = contact_matrix_from_hic(chrn, start, end, reso, fhic0, hicnorm)
File "/diffDomain/diffdomain-py3/utils.py", line 181, in contact_matrix_from_hic
c = cooler.Cooler(f'{hic_norm}')
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 85, in init
self._refresh()
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 89, in _refresh
with open_hdf5(self.store, **self.open_kws) as h5:
File "/home/micromamba/envs/diffDomain3/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/util.py", line 525, in open_hdf5
fh = h5py.File(fp, mode, *args, **kwargs)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in init
swmr=swmr)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "diffdomains.py", line 76, in
result.append(i.get())
File "/home/micromamba/envs/diffDomain3/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
OSError: Unable to open file (file signature not found)

Any help will be highly appreciated!
Thanks :)

[Tian-Dechao]

We have not encountered similar issue before. It might be due to file corruption, see a link here.

I would recommend double check the cool files. By default, diffDomain does parallel computation with 10 thread, first can try using only one thread by setting --ncore at 1.
python diffdomains.py dvsd multiple /data/T1_10000.cool /data/T2_10000.cool /data/TADS_DM.bed --reso 10000 --ncore 1

Alternatively, try run basic analysis with the cool files say call TADs with other tools.