unable to run conda as standalone application

[suminwei2772]

Hi, I'm trying to run dropSeqPipe, and have followed the steps to install miniconda3. When I try to run the following error:

snakemake --use-conda --cores 4 --directory /projects/lwei_prj/Dolomite

Building DAG of jobs...
CreateCondaEnvironmentException:
Unable to check conda version:
ERROR: The install method you used for conda--probably either pip install conda
or easy_install conda--is not compatible with using conda as an application.
If your intention is to install conda as a standalone application, currently
supported install methods include the Anaconda installer and the miniconda
installer. You can download the miniconda installer from
https://conda.io/miniconda.html.

File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 232, in create
File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 343, in new
File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 356, in init
File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 411, in _check

Would I need to install conda another way?

[seb-mueller]

It look like the installed conda is not in your PATH and you might have another version of conda which is in the PATH. Could you try echo $PATH? Does it include the conda installation directory? Also, could you try follow the suggestions here for troubleshooting?

[suminwei2772]

Did echo $PATH, and looks like it does include the conda installation directory (/projects/lwei_prj/Dolomite/miniconda3/bin):

/projects/lwei_prj/python_virtual_env/bin:/projects/lwei_prj/Dolomite/miniconda3/bin:/projects/lwei_prj/Dolomite/miniconda3/condabin:.:/projects/marralab_prj/lwei/AAML1031/scripts/samtools-1.9:.:/projects/lwei_prj/python_virtual_env_arboreto/bin/python:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/projects/lwei_prj/Dolomite/miniconda3/bin

[seb-mueller]

Indeed, the path seems correct. Strange. I still think there is something wrong with conda in general.Could you try to install something in conda, e.g. conda install samtools or so?
Failing this, could you have a look what conda info gives you.

[suminwei2772]

Hi @seb-mueller,
So with some help from Richard Corbett (also posted an issue) I managed to get conda working by adding the miniconda3 install (/projects/lwei_prj/Dolomite/miniconda3/bin) to the beginning of my PATH, and doing the following:

conda create -n dropSeqPipe
conda install -c bioconda -c conda-forge snakemake
conda activate dropSeqPipe

I re-started the run, and I'm now getting the following error when running
snakemake --use-conda --cores 20 --directory /projects/lwei_prj/Dolomite/dropSeqPipe2

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 20
Rules claiming more threads will be scaled down.
Job counts:
count jobs
4 DetectBeadSubstitutionErrors
3 MergeBamAlignment
3 STAR_align
4 SingleCellRnaSeqMetricsCollector
4 TagReadWithGeneExon
1 all
4 bam_hist
4 bead_errors_metrics
8 convert_long_to_mtx
4 extract_reads_expression
4 extract_umi_expression
2 merge_long
1 multiqc_star
3 pigz_unmapped
1 plot_adapter_content
4 plot_knee_plot
4 plot_rna_metrics
1 plot_yield
4 repair_barcodes
1 summary_stats
1 violine_plots
65

[Tue Feb 11 23:01:58 2020]
rule repair_barcodes:
input: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/Aligned.merged.bam, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/barcode_ref.pkl, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/barcode_ext_ref.pkl, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/empty_barcode_mapping.pkl
output: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/Aligned.repaired.bam, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/barcode_mapping_counts.pkl
jobid: 108
wildcards: results_dir=/projects/lwei_prj/Dolomite/dropSeqPipe2/results, sample=sample4

[Tue Feb 11 23:01:58 2020]
localrule plot_adapter_content:
input: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample1.clean_qc.csv, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample2.clean_qc.csv, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample3.clean_qc.csv, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample4.clean_qc.csv
output: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/plots/adapter_content.pdf
jobid: 4
wildcards: results_dir=/projects/lwei_prj/Dolomite/dropSeqPipe2/results

Activating conda environment: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/c7fcf3fb
Activating conda environment: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

filter, lag

The following objects are masked from ‘package:base’:

intersect, setdiff, setequal, union

Loading required package: viridisLite
Error in mutate_impl(.data, dots) :
Evaluation error: as_dictionary() is defunct as of rlang 0.3.0.
Please use as_data_pronoun() instead.
Calls: %>% ... -> mutate -> mutate.tbl_df -> mutate_impl -> .Call
Execution halted
[Tue Feb 11 23:02:14 2020]
Error in rule plot_adapter_content:
jobid: 4
output: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/plots/adapter_content.pdf
conda-env: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b

RuleException:
CalledProcessError in line 120 of /projects/lwei_prj/Dolomite/dropSeqPipe2/rules/filter.smk:
Command 'source /projects/lwei_prj/Dolomite/miniconda3/bin/activate '/projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b'; set -euo pipefail; Rscript --vanilla /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/scripts/tmpi_2c5y5g.plot_adapter_content.R' returned non-zero exit status 1.
File "/projects/lwei_prj/Dolomite/dropSeqPipe2/rules/filter.smk", line 120, in __rule_plot_adapter_content
File "/projects/lwei_prj/Dolomite/miniconda3/envs/dropSeqPipe/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Removing temporary output file /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/Aligned.merged.bam.
[Tue Feb 11 23:12:13 2020]
Finished job 108.
1 of 65 steps (2%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/log/2020-02-11T230153.534247.snakemake.log

Do you know what might be wrong?
Thank you,
Lisa

[seb-mueller]

First of all, conda seem to work now, and this seems to be another error.

This is a bit unexpected and probably has to do how conda resolves package dependencies (which we don't have full control over). I suspect the dplyr and rlang versions don't like each other, could you try activating the environment in question:

conda active /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b
# start R (which is part of the environment)
R

within R, run the following and report back the results:

library(ggplot2)
library(dplyr)
library(viridis)
sessionInfo()

Also, which version of dropseqpipe do you use?

[suminwei2772]

I typed in
conda activate /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b (/projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b) [lwei@marrahost02 dropSeqPipe2]$ R
I then get the following:

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(ggplot2)

> library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

> library(viridis)
Loading required package: viridisLite
> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS release 6.7 (Final)

Matrix products: default
BLAS: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b/lib/R/lib/libRblas.so
LAPACK: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] viridis_0.5.1     viridisLite_0.3.0 dplyr_0.7.4       ggplot2_3.1.0    

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.15     withr_2.1.2      crayon_1.3.4     assertthat_0.2.0
 [5] grid_3.4.1       plyr_1.8.4       R6_2.2.2         gtable_0.2.0    
 [9] magrittr_1.5     scales_1.0.0     pillar_1.3.0     rlang_0.3.0.1   
[13] lazyeval_0.2.1   bindrcpp_0.2     glue_1.3.0       munsell_0.5.0   
[17] compiler_3.4.1   pkgconfig_2.0.2  colorspace_1.3-2 bindr_0.1.1     
[21] gridExtra_2.3    tibble_1.4.2

[suminwei2772]

Also, I cloned the dropSeqPipe from Github using
git clone https://github.com/Hoohm/dropSeqPipe.git
I'm not sure how I may be able to find out which version this is?

[Hoohm]

Hello @suminwei2772
would you be ok to try the singularity approach instead of the conda one?

[seb-mueller]

I'm pretty sure you are on version 0.4, i.e. the current master. It is concerning that this is not working for you. Strangely, your dplyr version is 0.7.4 even though it should be 0.7.6. Not sure why this is, maybe it is interfering with a R local install, you could check by typing .libPaths() in the session above. Otherwise, I would also suggest, failing @Hoohm suggestion, to try out the develop version, i.e. git checkout develop and rerun the pipeline.