Repeatmodeler package not working

[helltish]

Dear developer Team,

we have compared the bioconda Repeatmodeler package with a local installation of Repeatmodeler (with all dependencies).

The local installation of Repeatmodeler produced an output, which is reasonable for our input file. The conda package does not produce any output. Please find our logs to compare. The input data was the same in both cases.

Thanks a lot in advance!

[helltish]

RepeatModeler_conda.log
RepeatModeler_local_installation.log

[corburn]

NOTE: RepeatScout did not return any models.

[mbnmbn00]

Exactly same issue here. Manually installed RepeatModeler worked well on the same assembly.
I also noticed that nseg which is needed for RepeatScout was not installed.
Manually compiling and locating at bin directory didn't help.

<prefix>.fa.rscons.filtered file which is one of RepeatScout outputs was empty, too. So I assume that RepeatScout has failed at some point.

[astulaaa]

coda Repeatmodeler package still not working properly

[jebrosen]

I believe the underlying issue is here:
https://github.com/bioconda/bioconda-recipes/blob/master/recipes/repeatmodeler/build.sh#L23

The RepeatScout package in bioconda does not include filter-stage-1.prl in the bin directory, but that program is used by RepeatModeler.

[Juke34]

I'm working on it with this PR #19088. I guess it could work like it is in this PR (without nseg) but to be perfect it would be nice to include nseg. But there is currently no recipe for it. nseg is available here ftp://ftp.ncbi.nih.gov/pub/seg/nseg.

[Juke34]

In the build3 of RepeatScout nseg and filter-stage-1.prl are now included, but the result seems still the same. How can we check manually which step is failing? Maybe one of this tool do not behaves like they should

[jebrosen]

I believe at this point the remaining bug is in the RepModelConfig.pm and/or the RepeatModeler wrapper script shipped in bioconda: it sets $TRF_PRGM = $ENV{'TRF_DIR'}; which is something like .../conda/bin, but TRF_PRGM is supposed to be the path to the trf binary itself (.../conda/bin/trf) - the same goes for NSEG_PRGM. Either RepModelConfig.pm or the wrapper should be modified to set the correct paths.

There could be other problems in the custom RepModelConfig.pm that I haven't noticed, but those two are directly causing this particular issue.

[Juke34]

Thank you for your help, I succeeded to fix that today. Now this step is working fine.
I have now a problem later in the execution:

 -- Refining Family R=207 / 0 ( RS Elements: 2212, Using 100 ):
RepeatModeler: Could not open refined model /scratch/jacda119/RM_19538.FriDec61556042019/round-1/family-0.fa.refiner_cons!

[jebrosen]

@Juke34 Does that happen for every model or only some of them? I can try to reproduce that in a clean environment... probably next week.

[Juke34]

I found! It is because the PATH to RepeatClassifier, Refiner and TRFMask are wrong. They are called directly in the RepeatModeler folder (in share), that cause an error e.g:
'-bash: ./TRFMask: /u1/local/bin/perl: bad interpreter: No such file or directory'

while they have to be called as the other tools by the bin folder where there is a wrapper to call them like that:
perl path/to/share/RepeatModeler/TRFMask options

But I'm getting close, I found a (nasty) way to fix the problem by modifying the code during installation with a sed command. I'm trying locally and it seems to work ... will see if I face up another problem

[jebrosen]

./TRFMask: /u1/local/bin/perl: bad interpreter: No such file or directory

Yes... the configure script fixes all of those perl lines but bioconda does not use it. Hopefully with the newest version of RepeatModeler where configure supports command line arguments and does not need to be run interactively, bioconda can use configure instead.

[Juke34]

It run until the end now. Except at the very end I have this message

Missing /home/jacda119/anaconda3/envs/repeatmodeler/share/RepeatMasker/Libraries/RepeatPeps.lib.psq!
Please rerun the configure program in the RepeatModeler directory
before running this script.
  - Looking for similarity to known repeat proteins..
Classification Time: 00:00:26 (hh:mm:ss) Elapsed Time
Program Time: 12:19:15 (hh:mm:ss) Elapsed Time
Working directory:  /scratch/jacda119/RM_33119.FriDec62053432019
may be deleted unless there were problems with the run.

The results have been saved to:
[...]

Is this RepeatPeps.lib.psq important? What it is use for? Can we skip this step? If we need this file how can we include it in the package? I don't know from where it is supposed to come

[jebrosen]

I think that message is wrong - it should say to re-run configure in the RepeatMasker directory, which bioconda also does not do. This should only affect the classification step, but it's a pretty big part of it.

[Juke34]

That means I need to fix the repeatmasker recipe too to create this file. Could you tell me the steps needed to create this file without using the configure? There is a file called RepeatPeps.lib so I guess there is one step to create the RepeatPeps.lib.psq from it.

[jebrosen]

It used to be done by the package - https://github.com/bioconda/bioconda-recipes/blob/master/recipes/repeatmasker/build.sh#L22. Like RepeatModeler, the latest version of RepeatMasker has a configure script that can be run non-interactively so that should be easier in the future.

[Juke34]

Actually is just a file made by makeblastdb command. No need to touch the RepeatMasker recipe for that, I can fix it directly from the RepeatModeler recipe then.

[Juke34]

The recipe is now fixed in repeatmodeler-1.0.11 build pl526_2 (see #19137).

The only thing remaining is the last step (RepeatClassifier) that use RepeatMasker that will be skipped by default. You will get this message:

 Missing ${CONDA_PREFIX}/share/RepeatMasker/Libraries/RepeatMasker.lib.nsq!

This is because no nucleotide repeat library is included in RepeatMasker. So it is recommended to download the DB of your choice (get licence to use RepBase) to get this working properly.

   cp RepeatDB.fna ${CONDA_PREFIX}/share/RepeatMasker/Libraries/RepeatMasker.lib
   makeblastdb -dbtype nucl -in ${CONDA_PREFIX}/share/RepeatMasker/Libraries/RepeatMasker.lib

[Juke34]

@helltish you can close the issue now.

[cement-head]

Hello,

I just got this bug

RepeatClassifier Version 2.0.1
======================================
Search Engine = rmblast
  - Looking for Simple and Low Complexity sequences..
  - Looking for similarity to known repeat proteins..
Missing /home/cbfgws6/Programs/rpmskr/RepeatMasker//Libraries/RepeatPeps.lib.psq!
Please rerun the configure program in the RepeatModeler directory
before running this script.

How exactly should I fix it? Should I run this command:

$ makeblastdb -dbtype nucl -in /home/cbfgws6/Programs/rpmskr/RepeatMasker/Libraries/RepeatMasker.lib

So, running the above command generated the following three files:

RepeatMasker.lib.nsq
RepeatMasker.lib.nin
RepeatMasker.lib.nhr

And then restarting using the command (-recoverDir) results in this:

This directory ( /home/cbfgws6/Programs/rpmddlr/RepeatModeler/RM_9706.ThuSep171738172020 )
appears to contain a successful run of RepeatModeler.  If this
is not the case, please report this as a bug at the RepeatMasker
website ( www.repeatmasker.org )

So...am I good?

[Juke34]

Do conda list and check the following:
What version did you use?
What is the RepeatMasker version installed? (conda list)

The RepeatMasker.lib should be there $CONDA_PREFIX/share/RepeatMasker/Libraries/

[mbnmbn00]

Hello, I had the exact same issue.
What I had to do is run ./configure again. I think configure includes downloading the database and make a blast database.

cd $(dirname $(which RepeatMasker))/../share/RepeatMasker
# ./configure downloads required databases
echo -e "\n2\n$(dirname $(which rmblastn))\n\n5\n" > tmp && ./configure < tmp

It should look like this

ls $(dirname $(which RepeatMasker))/../share/RepeatMasker/Libraries
# Artefacts.embl  Dfam.hmm       RepeatAnnotationData.pm  RepeatMasker.lib.nin  RepeatPeps.lib      RepeatPeps.lib.psq
# CONS-Dfam_3.0   README.meta    RepeatMasker.lib         RepeatMasker.lib.nsq  RepeatPeps.lib.phr  RepeatPeps.readme
# Dfam.embl       RMRBMeta.embl  RepeatMasker.lib.nhr     RepeatMaskerLib.embl  RepeatPeps.lib.pin  taxonomy.dat

Hope it helps. (I'm using RepeatMasker 2.0.1)

repeatmasker 4.0.9_p2 pl526_2 bioconda
repeatmodeler 2.0.1 pl526_0 bioconda
repeatscout 1.0.6 h516909a_1 bioconda

[Juke34]

I checked repeatmasker 4.0.9_p2 and 4.1.0 and indeed the RepeatMaskerLib db in not set properly (on OSX at least)...

-rw-rw-r--  2 jacda119  wheel    18755326 Sep 15 21:14 RMRBMeta.embl
-rw-rw-r--  2 jacda119  wheel   113343436 Sep 15 21:14 taxonomy.dat
-rw-rw-r--  2 jacda119  wheel        5550 Sep 15 21:15 RepeatPeps.readme
-rw-rw-r--  2 jacda119  wheel    17979984 Sep 15 21:15 RepeatPeps.lib
-rwxrwxr-x  2 jacda119  wheel    22475384 Sep 15 21:15 RepeatAnnotationData.pm
-rw-rw-r--  2 jacda119  wheel         214 Sep 15 21:15 README.meta
-rw-rw-r--  2 jacda119  wheel  1869701327 Sep 15 21:15 Dfam.hmm
-rw-rw-r--  2 jacda119  wheel    24005361 Sep 15 21:15 Dfam.embl
-rwxrwxr-x  2 jacda119  wheel       25283 Sep 15 21:15 Artefacts.embl
-rw-rw-r--  2 jacda119  wheel    22661790 Sep 15 21:15 RepeatMaskerLib.embl
-rw-rw-r--  2 jacda119  wheel           0 Sep 15 21:15 RepeatMasker.lib
-rw-rw-r--  2 jacda119  wheel    16168295 Sep 15 21:15 RepeatPeps.lib.psq
-rw-rw-r--  2 jacda119  wheel     2931407 Sep 15 21:15 RepeatPeps.lib.phr
-rw-rw-r--  1 jacda119  wheel      144448 Sep 21 22:14 RepeatPeps.lib.pin

[Juke34]

For RepeatMasker version 4.0.9_p2 the easiest would be to do
makeblastdb -dbtype nucl -in $CONDA_PREFIX/share/RepeatMasker/Libraries/RepeatMasker.lib. This line should be added in the build.sh if we want to fix this version of the recipe.

For version 4.1.0 we use the following command:
perl ./configure -libdir ${RM_DIR}/Libraries -trf_prgm ${PREFIX}/bin/trf -rmblast_dir ${PREFIX}/bin/ -hmmer_dir ${PREFIX}/bin -abblast_dir ${PREFIX}/bin -crossmatch_dir ${PREFIX}/bin
@jebrosen is there any reason why the RepeatMasker.lib db is not set properly with the configure? It

[insectnate]

I am still getting this error
Missing /home1/miniconda3/share/RepeatMasker/Libraries/RepeatMasker.lib.nsq!
Please rerun the configure program in the RepeatModeler directory
before running this script.

when trying to generate the
consensi.fa.classified file
Using
RepeatClassifier -consensi consensi.fa

I don't quite follow all the above discussion on how to fix this. Where in the conda bin directory should the .lib.nsq file be placed?
Thanks for any help.

Nathan

[mudithekanayake]

Hello, I had the exact same issue.
What I had to do is run ./configure again. I think configure includes downloading the database and make a blast database.

cd $(dirname $(which RepeatMasker))/../share/RepeatMasker
# ./configure downloads required databases
echo -e "\n2\n$(dirname $(which rmblastn))\n\n5\n" > tmp && ./configure < tmp

It should look like this

ls $(dirname $(which RepeatMasker))/../share/RepeatMasker/Libraries
# Artefacts.embl  Dfam.hmm       RepeatAnnotationData.pm  RepeatMasker.lib.nin  RepeatPeps.lib      RepeatPeps.lib.psq
# CONS-Dfam_3.0   README.meta    RepeatMasker.lib         RepeatMasker.lib.nsq  RepeatPeps.lib.phr  RepeatPeps.readme
# Dfam.embl       RMRBMeta.embl  RepeatMasker.lib.nhr     RepeatMaskerLib.embl  RepeatPeps.lib.pin  taxonomy.dat

Hope it helps. (I'm using RepeatMasker 2.0.1)

repeatmasker 4.0.9_p2 pl526_2 bioconda
repeatmodeler 2.0.1 pl526_0 bioconda
repeatscout 1.0.6 h516909a_1 bioconda

Can you please describe this line? I'm new to this.
echo -e "\n2\n$(dirname $(which rmblastn))\n\n5\n" > tmp && ./configure < tmp

[insectnate]

So i ran the above code you suggest in the miniconda share dir.
cd $(dirname $(which RepeatMasker))/../share/RepeatMasker

./configure downloads required databases

echo -e "\n2\n$(dirname $(which rmblastn))\n\n5\n" > tmp && ./configure < tmp

But the directory still only looks like this.
Artefacts.embl README.meta RepeatMaskerLib.embl RepeatPeps.lib RepeatPeps.readme
Dfam.embl RepeatAnnotationData.pm RepeatMasker.lib.ndb RepeatPeps.lib.pdb RMRBMeta.embl
Dfam.hmm RepeatMasker.lib RepeatMasker.lib.ndb-lock RepeatPeps.lib.pdb-lock taxonomy.dat

[insectnate]

I configured it to use HMMER3.1 however. Would this be the problem?

[mbnmbn00]

@mudithekanayake,
If you run ./configure, the interactive configuration process will popup.
Five configurations required:
\n
2\n
$(dirname $(which rmblastn))\n
\n
5\n

For the first \n, you type the return key to use the default value.
For the second 2\n, you type 2 and the return key to choose 2.
... and so on.

Of course, you can do it in the interactive mode by setting one by one, but I was suggesting a kind of shortcut.

[mbnmbn00]

@insectnate I don't think that would be a problem.
Try it out!

[insectnate]

I did that and still get a Library directory of
Artefacts.embl README.meta RepeatMaskerLib.embl RepeatPeps.lib RepeatPeps.readme
Dfam.embl RepeatAnnotationData.pm RepeatMasker.lib.ndb RepeatPeps.lib.pdb RMRBMeta.embl
Dfam.hmm RepeatMasker.lib RepeatMasker.lib.ndb-lock RepeatPeps.lib.pdb-lock taxonomy.dat

[mbnmbn00]

Can you do the ./configure and configure the settings one by one?

[insectnate]

I did the ./configure where I am prompted to confirm the $PATH to each of the dependencies. I have tried it both with specifying RMBlast and HMMER3.1 but never get the makeblastdb files shown above. Is there another way to do configure that I am missing where there are more settings?

[mbnmbn00]

1. I recommend creating a new environment and try it again.
2. Actually, if you look at the above comments, makeblastdb does enough job for you.

 makeblastdb -dbtype nucl -in ${CONDA_PREFIX}/share/RepeatMasker/Libraries/RepeatMasker.lib

[insectnate]

Ok I will try that. By this do you mean deleting the conda install of RepeatMasker and installing again?

Thanks for all your help.

Nathan

[mbnmbn00]

Yes. Or, you could create another environment name.

[michaelkarlcoleman]

Possibly related, the script queryRepeatDatabase.pl seems not to run because FastaDB.pm is not in $PERL5LIB. (It's in .../share/RepeatMasker/.)

If I manually add that to $PERL5LIB, the script then fails with

No repeat libraries found!  At a minimum Dfam.embl, Dfam.hmm
or RepBase RepeatMasker Edition is required to run.  Please download
 and install the latest Dfam libraries.

Died at /packages/miniconda/20190102/envs/rm-edta-mcurry-20201021/share/RepeatMasker/LibraryUtils.pm line 386.

Note also for the above comments, I'm pretty sure it's not kosher to modify files/dirs in the conda tree after install by conda. I think this breaks conda.

[Masa918]

I had a same issue that RepeatMasker.lib and its blastdb were not provided by RepeatMasker configure. The solution I could was downloading the previous version-4.1.0 of RepeatMasker independently to conda then configured. Hope it works for you guys.

[jebrosen]

Now that #25163 is complete, I was able to download repeatmasker 4.1.1 from the bioconda repositories and RepeatModeler successfully finished, including the classification step, on a test sequence file. I think that was the last remaining issue reported in this thread; hopefully this update is also working well for others.

There are still a few reasons one might prefer a manual installation, for example the LTR structural search method introduced RepeatModeler 2.0 requires some dependencies that are not yet in bioconda.

[JuliaLopezDelgado]

I recently had the same issue as described above, with the error message "missing /RepeatMasker/Libraries/RepeatPeps.lib.psq" when I tried to run RepeatClassifier. Just in case anyone else experiences this issue, I solved it by re-configuring RepeatMasker and manually downloading all the libraries from https://home.cc.umanitoba.ca/~psgendb/doc/BIRCH/doc/local/pkg/RepeatMasker/Libraries/ to the directory RepeatMasker/Libraries.

[Juke34]

@JuliaLopezDelgado Please specify the version you have used. Because depending the version the problem is solved.

[JuliaLopezDelgado]

@JuliaLopezDelgado Please specify the version you have used. Because depending the version the problem is solved.

I am using RepeatMasker v.4.1.2 and RepeatModeler v.2.0.2

[jebrosen]

@JuliaLopezDelgado Bioconda had packages for RepeatMasker 4.1.1 (broken) and RepeatMasker 4.1.2-p1 (fixed), skipping over RepeatMasker 4.1.2 (FWIW, also fixed). Did you mean 4.1.2-p1?

The libraries available at home.cc.umanitoba.ca are over 10 years old. It is only by chance that they could even work today - and worse, old libraries might appear to work and silently fail or produce wrong/misleading results. For that reason I generally recommend against trying to "mix" files from new and old versions of RepeatMasker and RepeatModeler.

[JuliaLopezDelgado]

Thanks for letting me know @jebrosen !

I have rerun the RepeatMasker configure script and also ran: makeblastdb -dbtype nucl -in RepeatMasker/Libraries/RepeatMasker.lib but I keep getting the error: Missing RepeatMasker//Libraries/RepeatPeps.lib.psq!

This is my RepeatMasker/Libraries/ contents:
-rwx------ 1 bsjld fbs 25283 Apr 1 2021 Artefacts.embl
-rw------- 1 bsjld fbs 88115699680 Oct 13 11:01 Dfam.h5
drwx------ 4 bsjld fbs 4096 Oct 13 11:34 HMM-Dfam_3.2
-rw------- 1 bsjld fbs 214 Apr 1 2021 README.meta
-rwx------ 1 bsjld fbs 22475384 Apr 1 2021 RepeatAnnotationData.pm
-rw------- 1 bsjld fbs 11024741 Oct 13 11:21 RepeatMasker.lib
lrwxrwxrwx 1 bsjld fbs 66 Oct 13 11:09 RepeatMaskerLib.h5 -> /nobackup/bsjld/data/annotation/bin/RepeatMasker/Libraries/Dfam.h5
-rw------- 1 bsjld fbs 8192 Oct 13 11:21 RepeatMasker.lib.ndb
-rw------- 1 bsjld fbs 8192 Oct 13 11:21 RepeatMasker.lib.ndb-lock
-rw------- 1 bsjld fbs 744110 Oct 13 11:48 RepeatMasker.lib.nhr
-rw------- 1 bsjld fbs 83584 Oct 13 11:48 RepeatMasker.lib.nin
-rw------- 1 bsjld fbs 3095721 Oct 13 11:48 RepeatMasker.lib.nsq
-rw------- 1 bsjld fbs 17979785 Apr 1 2021 RepeatPeps.lib
-rw------- 1 bsjld fbs 8192 Oct 13 11:21 RepeatPeps.lib.pdb
-rw------- 1 bsjld fbs 8192 Oct 13 11:21 RepeatPeps.lib.pdb-lock
-rw------- 1 bsjld fbs 5550 Apr 1 2021 RepeatPeps.readme
-rw------- 1 bsjld fbs 18752245 Apr 1 2021 RMRBMeta.embl
-rw------- 1 bsjld fbs 113343436 Apr 1 2021 taxonomy.dat

Do you have any suggestions on how to fix this?

I have manually installed RepeatMasker v.4.1.2 and RepeatModeler v.2.0.2.

Thanks in advance,
Julia

[jebrosen]

@JuliaLopezDelgado It looks like you are missing the file RepeatPeps.lib.psq, but you ran a command to generate RepeatMasker.lib.nsq. Try this one instead: makeblastdb -dbtype prot -in RepeatMasker/Libraries/RepeatPeps.lib. I hope this resolves your problem!

I noticed in your file list that there are files from multiple versions of RepeatMasker. Mixing files from different versions can cause confusion and sometimes errors, so we usually recommend installing updates to RepeatMasker and RepeatModeler in new directories.

If you continue experiencing issues with a manual installation of RepeatMasker/RepeatModeler, please see also our GitHub pages and issue trackers for these projects, which are not specific to bioconda: https://github.com/rmhubley/RepeatMasker/issues, https://github.com/Dfam-consortium/RepeatModeler/issues.

[JuliaLopezDelgado]

Thank you @jebrosen! The makeblastdb command generated the .psq file and RepeatClassifier now works.

I didn't install multiple versions of RepeatMasker, so I'm unsure why there are files from different versions. These are the commands I used to install it:
wget http://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.2-p1.tar.gz
gunzip RepeatMasker-4.1.2-p1.tar.gz
tar xvf RepeatMasker-4.1.2-p1.tar.gz
wget https://www.dfam.org/releases/Dfam_3.2/families/Dfam.h5.gz
gunzip Dfam.h5.gz
mv Dfam.h5 ./RepeatMasker/Libraries
cd ./RepeatMasker
perl ./configure
makeblastdb -dbtype prot -in ./Libraries/RepeatPeps.lib

Does this look okay?

Thank you!
Julia

[jebrosen]

Sorry @JuliaLopezDelgado, after a re-read I think I got a few version numbers confused. That looks fine to me, although the latest version of Dfam is now up to Dfam 3.5.

[duoqingxiaotaiyang]

but when i type makeblastdb -dbtype prot -in RepeatMasker/Libraries/RepeatPeps.lib, it shows that "Error: NCBI C++ Exception:
T0 "/opt/conda/conda-bld/rmblast_1648559906758/work/c++/src/objtools/blast/seqdb_writer/build_db.cpp", line 1019: Error: (CMultisourceException::eInvalid) BLASTDB::ncbi::CBuildDatabase::CreateDirectories() - Failed to create directory 'Libraries'
",how can i solve this problem?

[Juke34]

The path to the lib is wrong try:
makeblastdb -dbtype prot -in $(dirname $(which RepeatMasker))/../share/RepeatMasker/Libraries/RepeatPeps.lib

Before you run the command check if the path is correct:
ls -l $(dirname $(which RepeatMasker))/../share/RepeatMasker/Libraries/

[duoqingxiaotaiyang]

Thank you so much for letting me know !! now repeatmodeler can work properly