osx perl-json-xs (and other packages) install perl-threaded

[nathanweeks]

On macOS, perl-json-xs (at least) depends on the deprecated perl-threaded, which is Perl 5.22.0. When perl-threaded is installed, it causes $CONDA_PREFIX/bin/perl to refer to its version of perl, causing problems when other perl modules that depend on the conda-forge 5.26.2 are installed (e.g., perl-perlio-gzip).

Pristine Miniconda3 installation w/ bioconda channels configured per http://bioconda.github.io/:

$ conda install perl-json
...
The following NEW packages will be INSTALLED:

    perl:              5.26.2-h16c6ff1_0 conda-forge
    perl-common-sense: 3.74-pl526_1      bioconda
    perl-json:         2.90-pl526_2      bioconda
    perl-json-xs:      2.34-0            bioconda
    perl-threaded:     5.22.0-10         bioconda
...
$ conda install perl-perlio-gzip
...
The following NEW packages will be INSTALLED:

    perl:             5.26.2-h16c6ff1_0    conda-forge
    perl-perlio-gzip: 0.19-pl526h1341992_3 bioconda

Installing both:

$ conda install perl-json perl-perlio-gzip
The following NEW packages will be INSTALLED:

    perl:              5.26.2-h16c6ff1_0    conda-forge
    perl-common-sense: 3.74-pl526_1         bioconda
    perl-json:         2.90-pl526_2         bioconda
    perl-json-xs:      2.34-0               bioconda
    perl-perlio-gzip:  0.19-pl526h1341992_3 bioconda
    perl-threaded:     5.22.0-10            bioconda

$CONDA_PREFIX/bin/perl (perl-threaded) can use the installed perl-json module, but not perl-perlio-gzip. Conversely, $CONDA_PREFIX/bin/perl5.26.2 can use perl-perlio-gzip, but not perl-json:

$ perl -v | head -n 2

This is perl 5, version 22, subversion 0 (v5.22.0) built for darwin-thread-multi-2level
$ perl5.26.2 -MPerlIO::gzip -e ''
$ perl -MPerlIO::gzip -e ''
Can't locate PerlIO/gzip.pm in @INC (you may need to install the PerlIO::gzip module) (@INC contains: /Volumes/scratch/miniconda3/lib/perl5/site_perl/5.22.0/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/perl5/site_perl/5.22.0 /Volumes/scratch/miniconda3/lib/perl5/5.22.0/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/perl5/5.22.0 .).
$ perl -MJSON -e ''
$ perl5.26.2 -MJSON -e ''
Can't locate JSON.pm in @INC (you may need to install the JSON module) (@INC contains: /Volumes/scratch/miniconda3/lib/site_perl/5.26.2/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/site_perl/5.26.2 /Volumes/scratch/miniconda3/lib/5.26.2/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/5.26.2 .).
BEGIN failed--compilation aborted.

[nathanweeks]

This affects at least some Linux packages as well; e.g., perl-perlio-gzip and perl-bioperl (the latter of which has a dependency on perl-threaded somewhere).

In this case, $CONDA_PREFIX/bin/perl is perl5.26.2, and there isn't an executable for Perl 5.22.0. So PerlIO::gzip can be used, but bioperl cannot...

$ conda install perl-perlio-gzip perl-bioperl
...
$ perl --version | head -n 2

This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-thread-multi
$ perl -MPerlIO::gzip -e ''
$ perl -MBio::DB::SeqFeature::Store -e ''
Can't locate Bio/DB/SeqFeature/Store.pm in @INC (you may need to install the Bio::DB::SeqFeature::Store module) (@INC contains: /project/test/miniconda/lib/site_perl/5.26.2/x86_64-linux-thread-multi /project/test/miniconda/lib/site_perl/5.26.2 /project/test/miniconda/lib/5.26.2/x86_64-linux-thread-multi /project/test/miniconda/lib/5.26.2 .).
BEGIN failed--compilation aborted.
$ find $CONDA_PREFIX -name Store.pm
/project/test/miniconda/lib/perl5/site_perl/5.22.0/Bio/DB/SeqFeature/Store.pm

[acaprez]

Yeah, I noticed this awhile back as well. I've been slowly working through small batches of packages trying to fix things.

[acaprez]

It's even worse, as some packages were rebuilt after the switch to 5.26.2 and perl-threaded was pulled in. So they are tagged as 5.26 builds, but the package contents are installed in the wrong (5.22.0) location. Example of that here.

[nathanweeks]

Thanks @acaprez , I appreciate your efforts to go clean this up!

[bgruening]

Sorry, to be late here. @acaprez I thought I removed all occurrences of perl-threaded a while back. Are there still packages depending on it? Should we release a new/last version of perl-threaded pointing to the new perl release? Will this help?

[acaprez]

No worries @bgruening. Yes, there are still perl packages that use it - for example, perl-xml-twig and perl-xml-path. I suspect what happened is some packages were added to the blacklist and so were never rebuilt with perl 5.26.2, but were still pulled in as dependencies for other packages, thus pulling in perl-threaded.

I'd be hesitant to try anything too clever with the perl-threaded package. In hindsight, using that name wasn't the best idea, as conda build will happily install perl-threaded and perl in the same environment. My suggestion I suppose would be to remove the perl-threaded package completely from the bioconda channel (or at least the OS X packages), and then do a force rebuilding of all the perl packages on OS X. Anything that still tries to pull in perl-threaded will fail immediately then with a dependency error, and be easy to spot. This will also make creation of any environment still using perl-threaded fail, but IMHO it's better to have a hard failure immediately than successfully create an env that is half-broken. Ideally we could create a dependency DAG of all the perl packages and rebuild them in batches according to their level in the graph, but I haven't looked into how difficult it would be to generate that graph.

[druvus]

We spent time to rebuild failing perl osx packages so this should be solved. Please reopen if the issue is still showing up.