Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing Problem #68

Closed
yingeddi2008 opened this issue Aug 21, 2017 · 13 comments
Closed

Testing Problem #68

yingeddi2008 opened this issue Aug 21, 2017 · 13 comments

Comments

@yingeddi2008
Copy link

Dear Midas programmers,

In order to set up MIDAS, I created a python virtual environment on our server, since our server has to use python2.6 for its core programs. I set up the python environmental variables as instructed and running the testing script. However, I encounter the following error message, and do not have any clue to solve it. Could you please help me with this?

(ve) [hulin@HSDMongo1 test]$ ./test_midas.py -vf
test_class (__main__._01_CheckEnv) ... ok
test_class (__main__._02_ImportDependencies) ... ok
test_class (__main__._03_CheckVersions) ... ok
test_class (__main__._04_HelpText) ... ok
test_class (__main__._05_RunSpecies) ... ok
test_class (__main__._06_RunGenes) ... FAIL

======================================================================
FAIL: test_class (__main__._06_RunGenes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_midas.py", line 96, in test_class
    self.assertTrue(code==0, msg=err)
AssertionError: Traceback (most recent call last):
  File "/home/hulin/data/tools/midas/MIDAS/scripts/run_midas.py", line 742, in <module>
    run_program(program, args)
  File "/home/hulin/data/tools/midas/MIDAS/scripts/run_midas.py", line 79, in run_program
    genes.run_pipeline(args)
  File "/home/hulin/data/tools/midas/MIDAS/midas/run/genes.py", line 285, in run_pipeline
    pangenome_coverage(args, species, genes)
  File "/home/hulin/data/tools/midas/MIDAS/midas/run/genes.py", line 148, in pangenome_coverage
    count_mapped_bp(args, species, genes)
  File "/home/hulin/data/tools/midas/MIDAS/midas/run/genes.py", line 174, in count_mapped_bp
    bamfile = pysam.AlignmentFile(bam_path, "rb")
  File "pysam/libcalignmentfile.pyx", line 401, in pysam.libcalignmentfile.AlignmentFile.__cinit__ (pysam/libcalignmentfile.c:5835)
  File "pysam/libcalignmentfile.pyx", line 611, in pysam.libcalignmentfile.AlignmentFile._open (pysam/libcalignmentfile.c:8072)
ValueError: file has no sequences defined (mode='rb') - is it SAM/BAM format? Consider opening with check_sq=False


----------------------------------------------------------------------
Ran 6 tests in 44.447s

FAILED (failures=1)

Thank you very much.

Eddi

@snayfach
Copy link
Owner

Hi Eddi,

Are you running the latest version of MIDAS? Have you tried testing with python 2.7? I just downloaded and tested the latest version on my system running 2.7 and the test completed without errors. If you can confirm the error is related to python 2.6 I can fix the code.

Best,
Stephen

@yingeddi2008
Copy link
Author

yingeddi2008 commented Aug 21, 2017 via email

@snayfach
Copy link
Owner

Hmmm, I'm not sure what's wrong.

Could you please try executing the commands listed in the tutorial? After running the run_midas.py genes, please send me the full error message and log file.

Thanks,
Stephen

@yingeddi2008
Copy link
Author

yingeddi2008 commented Aug 22, 2017

Hi I have the error message here

(ve) [hulin@HSDMongo1 midas]$ run_midas.py genes midas_output/sample_1 -1 example/sample_1.fq.gz &
[1] 61136
(ve) [hulin@HSDMongo1 midas]$
MIDAS: Metagenomic Intra-species Diversity Analysis System
version 1.3.0; github.com/snayfach/MIDAS
Copyright (C) 2015-2016 Stephen Nayfach
Freely distributed under the GNU General Public License (GPLv3)

===========Parameters===========
Command: /home/hulin/data/tools/midas/MIDAS/scripts/run_midas.py genes midas_output/sample_1 -1 example/sample_1.fq.gz
Script: run_midas.py genes
Database: /home/hulin/data/tools/midas/midas_db_v1.2
Output directory: midas_output/sample_1
Remove temporary files: False
Pipeline options:
  build bowtie2 database of pangenomes
  align reads to bowtie2 pangenome database
  quantify coverage of pangenomes genes
Database options:
  include all species with >=3.0X genome coverage
Read alignment options:
  input reads (unpaired): example/sample_1.fq.gz
  alignment speed/sensitivity: very-sensitive
  number of reads to use from input: use all
  number of threads for database search: 1
Gene coverage options:
  minimum alignment percent identity: 94.0
  minimum alignment coverage of reads: 0.75
  minimum read quality score: 20
  minimum mapping quality score: 0
  trim 0 base-pairs from 3'/right end of read
================================

Reading reference data
  0.04 minutes
  0.07 Gb maximum memory

Building pangenome database
  total species: 1
  total genes: 22080
  total base-pairs: 19868510

(ve) [hulin@HSDMongo1 midas]$   0.52 minutes
  0.18 Gb maximum memory

Aligning reads to pangenomes
  finished aligning
  checking bamfile integrity
  0.0 minutes
  0.18 Gb maximum memory

Computing coverage of pangenomes
Traceback (most recent call last):
  File "/home/hulin/data/tools/midas/MIDAS/scripts/run_midas.py", line 742, in <module>
    run_program(program, args)
  File "/home/hulin/data/tools/midas/MIDAS/scripts/run_midas.py", line 79, in run_program
    genes.run_pipeline(args)
  File "/home/hulin/data/tools/midas/MIDAS/midas/run/genes.py", line 285, in run_pipeline
    pangenome_coverage(args, species, genes)
  File "/home/hulin/data/tools/midas/MIDAS/midas/run/genes.py", line 148, in pangenome_coverage
    count_mapped_bp(args, species, genes)
  File "/home/hulin/data/tools/midas/MIDAS/midas/run/genes.py", line 174, in count_mapped_bp
    bamfile = pysam.AlignmentFile(bam_path, "rb")
  File "pysam/libcalignmentfile.pyx", line 401, in pysam.libcalignmentfile.AlignmentFile.__cinit__ (pysam/libcalignmentfile.c:5835)
  File "pysam/libcalignmentfile.pyx", line 611, in pysam.libcalignmentfile.AlignmentFile._open (pysam/libcalignmentfile.c:8072)
ValueError: file has no sequences defined (mode='rb') - is it SAM/BAM format? Consider opening with check_sq=False

[1]+  Exit 1                  run_midas.py genes midas_output/sample_1 -1 example/sample_1.fq.gz

Thanks you!

Similar error occured when running

run_midas.py snps

@snayfach
Copy link
Owner

snayfach commented Aug 22, 2017 via email

@yingeddi2008
Copy link
Author

yingeddi2008 commented Aug 22, 2017

Sure, there is.

MIDAS: Metagenomic Intra-species Diversity Analysis System
version 1.3.0; github.com/snayfach/MIDAS
Copyright (C) 2015-2016 Stephen Nayfach
Freely distributed under the GNU General Public License (GPLv3)

===========Parameters===========
Command: /home/hulin/data/tools/midas/MIDAS/scripts/run_midas.py genes midas_output/sample_1 -1 example/sample_1.fq.gz
Script: run_midas.py genes
Database: /home/hulin/data/tools/midas/midas_db_v1.2
Output directory: midas_output/sample_1
Remove temporary files: False
Pipeline options:
build bowtie2 database of pangenomes
align reads to bowtie2 pangenome database
quantify coverage of pangenomes genes
Database options:
include all species with >=3.0X genome coverage
Read alignment options:
input reads (unpaired): example/sample_1.fq.gz
alignment speed/sensitivity: very-sensitive
number of reads to use from input: use all
number of threads for database search: 1
Gene coverage options:
minimum alignment percent identity: 94.0
minimum alignment coverage of reads: 0.75
minimum read quality score: 20
minimum mapping quality score: 0
trim 0 base-pairs from 3'/right end of read
================================

Building pangenome database
command: /home/hulin/data/tools/midas/MIDAS/bin/Linux/bowtie2-build --threads 1 midas_output/sample_1/genes/temp/pangenomes.fa midas_output/sample_1/genes/temp/pangenomes

Aligning reads to pangenomes
command: /home/hulin/data/tools/midas/MIDAS/bin/Linux/bowtie2 --no-unal -x midas_output/sample_1/genes/temp/pangenomes --very-sensitive-local --threads 1 -q -U example/sample_1.fq.gz | /home/hulin/data/tools/midas/MIDAS/bin/Linux/samtools view --threads 1 -b - > midas_output/sample_1/genes/temp/pangenomes.bam

Computing coverage of pangenomes

@snayfach
Copy link
Owner

snayfach commented Aug 22, 2017 via email

@yingeddi2008
Copy link
Author

yingeddi2008 commented Aug 22, 2017

Now I know, that's the problem with bowtie2, so no bam file is generated.

(ve) [hulin@HSDMongo1 midas]$ /home/hulin/data/tools/midas/MIDAS/bin/Linux/bowtie2 --no-unal -x midas_output/sample_1/genes/temp/pangenomes --very-sensitive-local --threads 1 -q -U example/sample_1.fq.gz | /home/hulin/data/tools/midas/MIDAS/bin/Linux/samtools view --threads 1 -b - > midas_output/sample_1/genes/temp/pangenomes.bam &
[1] 61561
(ve) [hulin@HSDMongo1 midas]$ /home/hulin/data/tools/midas/MIDAS/bin/Linux/bowtie2-align-s: symbol lookup error: /home/hulin/data/tools/midas/MIDAS/bin/Linux/bowtie2-align-s: undefined symbol: gzopen64
(ERR): bowtie2-align exited with value 127

@surh
Copy link

surh commented Aug 22, 2017

I had this problem with release 1.3.0. It is not a problem with MIDAS itself though, it is a problem with the samtools/bowtie binaries being compiled with a newer version of a couple of libraries. If you try to run the included binary samtools at /path/to/midas/bin/Linux/samtools --help you will get a library error.

You need to install the newest zlib and maybe bzlib.

Sur

@yingeddi2008
Copy link
Author

yingeddi2008 commented Aug 22, 2017 via email

@snayfach
Copy link
Owner

snayfach commented Aug 22, 2017 via email

snayfach pushed a commit that referenced this issue Nov 26, 2017
-see issues #73, #72, #69, #68
-added new samtools and bowtie2 binaries which should work on more systems
-added checks for samtools and bowtie2 binaries; if binaries don't work instructions for replacing them are provided
@snayfach
Copy link
Owner

I've fixed this issue in the latest release. If the problem persists please reopen this issue

@kaysahu
Copy link

kaysahu commented Jan 24, 2022

Hello,

I have installed MIDAS using conda command in an empty virtual environment. I also installed samtools under the same conda environment. The installation is a success. But when I execute the run_midas.py command, I get the samtool error like the following. I double-check the $PATH and samtools is included in the $PATH. Samtools can be also run separately under the conda environment, but it looks like MIDAS couldn't find it.

This is an example of my error:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants