Make test issue

[valentinavan]

Hi,

I am a newby and I was trying to install katuali following the instruction. I think I got it right until the setup of the environment. I was then running the make test but I get lots of "warning" and "error" messages.
The problem might be a dependencies installation issue? I have tried to install what I need however, for some tools the installation did not work: for example, pomoxis installation did not work with bioconda nor pip.
Any help will be appreciated.
Thanks

(nano) (katuali) linnaeus@Carls-Mac-Pro katuali % make test
. ./venv/bin/activate && python setup.py develop
running develop
running egg_info
writing katuali.egg-info/PKG-INFO
writing dependency_links to katuali.egg-info/dependency_links.txt
writing entry points to katuali.egg-info/entry_points.txt
writing requirements to katuali.egg-info/requires.txt
writing top-level names to katuali.egg-info/top_level.txt
reading manifest file 'katuali.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
no previously-included directories found matching 'docs'
writing manifest file 'katuali.egg-info/SOURCES.txt'
running build_ext
Creating /Users/linnaeus/katuali/venv/lib/python3.6/site-packages/katuali.egg-link (link to .)
katuali 0.3.2 is already the active version in easy-install.pth
Installing check_files_exist script to /Users/linnaeus/katuali/venv/bin
Installing katuali_config script to /Users/linnaeus/katuali/venv/bin
Installing katuali_datafile script to /Users/linnaeus/katuali/venv/bin
Installing pick_gpu script to /Users/linnaeus/katuali/venv/bin
Installing process_katuali_config script to /Users/linnaeus/katuali/venv/bin
Installing katuali script to /Users/linnaeus/katuali/venv/bin

Installed /Users/linnaeus/katuali
Processing dependencies for katuali==0.3.2
Searching for pysam==0.15.2
Reading https://pypi.org/simple/pysam/
Downloading https://files.pythonhosted.org/packages/15/f6/ce0611aaa1865a616f7dc164fbf046eaf38f2b17c6d404403c56250beb93/pysam-0.15.2.tar.gz#sha256=d049efd91ed5b1af515aa30280bc9cb46a92ddd15d546c9b21ee68a6ed4055d9
Best match: pysam 0.15.2
Processing pysam-0.15.2.tar.gz
Writing /var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/setup.cfg
Running pysam-0.15.2/setup.py -q bdist_egg --dist-dir /var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/egg-dist-tmp-a4yfet1v
# pysam: no cython available - using pre-compiled C
# pysam: htslib mode is shared
# pysam: HTSLIB_CONFIGURE_OPTIONS=None
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for ranlib... ranlib
checking for grep that handles long lines and -e... /usr/bin/grep
checking for C compiler warning flags... -Wall
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for _LARGEFILE_SOURCE value needed for large files... no
checking shared library type for unknown-Darwin... Darwin dylib
checking how to run the C preprocessor... gcc -E
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for getpagesize... yes
checking for working mmap... yes
checking for gmtime_r... yes
checking for fsync... yes
checking for drand48... yes
checking whether fdatasync is declared... no
checking for library containing log... none required
checking for zlib.h... yes
checking for inflate in -lz... yes
checking for library containing recv... none required
checking for bzlib.h... yes
checking for BZ2_bzBuffToBuffCompress in -lbz2... yes
checking for lzma.h... no
checking for lzma_easy_buffer_encode in -llzma... yes
checking for libdeflate.h... no
checking for libdeflate_deflate_compress in -ldeflate... no
checking for curl_easy_pause in -lcurl... yes
checking for CCHmac... yes
checking whether PTHREAD_MUTEX_RECURSIVE is declared... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating htslib.pc.tmp
config.status: creating config.h
# pysam: htslib configure options: --enable-libcurl
make[1]: ./version.sh: Command not found
make[1]: ./version.sh: Command not found
# pysam: htslib_config LDFLAGS=
# pysam: htslib_config LIBHTS_OBJS=kfunc.o knetfile.o kstring.o bcf_sr_sort.o bgzf.o errmod.o faidx.o hfile.o hfile_net.o hts.o hts_os.o md5.o multipart.o probaln.o realn.o regidx.o sam.o synced_bcf_reader.o vcf_sweep.o tbx.o textutils.o thread_pool.o vcf.o vcfutils.o cram/cram_codecs.o cram/cram_decode.o cram/cram_encode.o cram/cram_external.o cram/cram_index.o cram/cram_io.o cram/cram_samtools.o cram/cram_stats.o cram/files.o cram/mFILE.o cram/open_trace_file.o cram/pooled_alloc.o cram/rANS_static.o cram/sam_header.o cram/string_alloc.o hfile_libcurl.o hfile_gcs.o hfile_s3.o
# pysam: htslib_config LIBS=-llzma -lbz2 -lz -lcurl
# pysam: htslib_config PLATFORM=Darwin
# pysam: config_option: ENABLE_PLUGINS=0
# pysam: config_option: HAVE_COMMONCRYPTO=1
# pysam: config_option: HAVE_GMTIME_R=1
# pysam: config_option: HAVE_HMAC=0
# pysam: config_option: HAVE_IRODS=0
# pysam: config_option: HAVE_LIBCURL=1
# pysam: config_option: HAVE_MMAP=1
package init file 'samtools/__init__.py' not found (or not a regular file)
package init file 'bcftools/__init__.py' not found (or not a regular file)
package init file 'samtools/win32/__init__.py' not found (or not a regular file)
package init file 'htslib/__init__.py' not found (or not a regular file)
package init file 'htslib/htslib/__init__.py' not found (or not a regular file)
warning: no files found matching 'KNOWN_BUGS'
warning: no files found matching 'THANKS'
no previously-included directories found matching 'tests/'
warning: no files found matching 'samtools/configure'
warning: no files found matching 'samtools/config.mk.in'
warning: no files found matching 'samtools/config.h.in'
/Users/linnaeus/katuali/venv/lib/python3.6/site-packages/setuptools/lib2to3_ex.py:41: SetuptoolsDeprecationWarning: 2to3 support is deprecated. If the project still requires Python 2 support, please migrate to a single-codebase solution or employ an independent conversion process.
  SetuptoolsDeprecationWarning)
htslib/thread_pool.c:642:9: warning: code will never be executed [-Wunreachable-code]
        printf("%d waiting, %d running, %d output, %d, arun %d => %d\t", p->njobs,
        ^~~~~~
htslib/thread_pool.c:641:9: note: silence by adding parentheses to mark code as explicitly dead
    if (0) {
        ^
        /* DISABLES CODE */ ( )
1 warning generated.
htslib/cram/cram_encode.c:1605:13: warning: code will never be executed [-Wunreachable-code]
            cram_byte_array_len_encoder e;
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
htslib/cram/cram_encode.c:1595:13: note: silence by adding parentheses to mark code as explicitly dead
        if (1) {
            ^
            /* DISABLES CODE */ ( )
htslib/cram/cram_encode.c:2684:40: warning: code will never be executed [-Wunreachable-code]
                                int ol=l;
                                       ^
htslib/cram/cram_encode.c:2681:33: note: silence by adding parentheses to mark code as explicitly dead
                            if (0 && CRAM_MAJOR_VERS(fd->version) >= 3) {
                                ^
                                /* DISABLES CODE */ ( )
htslib/cram/cram_encode.c:2656:17: warning: code will never be executed [-Wunreachable-code]
                int l;
                ^~~~~~
3 warnings generated.
bcftools/convert.c.pysam.c:446:66: warning: code will never be executed [-Wunreachable-code]
        default: error("Unexpected type: %d\n", fmt->fmt->type); exit(1); break;
                                                                 ^~~~
1 warning generated.
bcftools/tabix.c.pysam.c:57:22: warning: code will never be executed [-Wunreachable-code]
            detect = 0;
                     ^
1 warning generated.
bcftools/vcfroh.c.pysam.c:764:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:754:34: note: expanded from macro 'BRANCH'
                prob[0] = p[irr] < 256 ? args->pl2p[ p[irr] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:764:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:755:34: note: expanded from macro 'BRANCH'
                prob[1] = p[ira] < 256 ? args->pl2p[ p[ira] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:764:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:756:34: note: expanded from macro 'BRANCH'
                prob[2] = p[iaa] < 256 ? args->pl2p[ p[iaa] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:794:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:784:34: note: expanded from macro 'BRANCH'
                prob[0] = p[irr] < 256 ? args->pl2p[ p[irr] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:794:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:785:34: note: expanded from macro 'BRANCH'
                prob[1] = p[ira] < 256 ? args->pl2p[ p[ira] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:794:32: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
            case BCF_BT_INT8:  BRANCH(int8_t); break;
                               ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:786:34: note: expanded from macro 'BRANCH'
                prob[2] = p[iaa] < 256 ? args->pl2p[ p[iaa] ] : args->pl2p[255]; \
                          ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:936:36: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
                case BCF_BT_INT8:  BRANCH(int8_t); break;
                                   ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:931:33: note: expanded from macro 'BRANCH'
                pdg[0] = p[irr] < 256 ? args->pl2p[ p[irr] ] : args->pl2p[255]; \
                         ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:936:36: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
                case BCF_BT_INT8:  BRANCH(int8_t); break;
                                   ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:932:33: note: expanded from macro 'BRANCH'
                pdg[1] = p[ira] < 256 ? args->pl2p[ p[ira] ] : args->pl2p[255]; \
                         ~~~~~~ ^ ~~~
bcftools/vcfroh.c.pysam.c:936:36: warning: result of comparison of constant 256 with expression of type 'int8_t'
      (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]
                case BCF_BT_INT8:  BRANCH(int8_t); break;
                                   ^~~~~~~~~~~~~~
bcftools/vcfroh.c.pysam.c:933:33: note: expanded from macro 'BRANCH'
                pdg[2] = p[iaa] < 256 ? args->pl2p[ p[iaa] ] : args->pl2p[255]; \
                         ~~~~~~ ^ ~~~
9 warnings generated.
bcftools/filter.c.pysam.c:2755:12: warning: code will never be executed [-Wunreachable-code]
    if (0) filter_debug_print(out, NULL, nout);
           ^~~~~~~~~~~~~~~~~~
bcftools/filter.c.pysam.c:2755:9: note: silence by adding parentheses to mark code as explicitly dead
    if (0) filter_debug_print(out, NULL, nout);
        ^
        /* DISABLES CODE */ ( )
1 warning generated.
pysam/libcutils.c:6272:3: error: implicit declaration of function 'bcftools_set_stderr' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
  bcftools_set_stderr(__pyx_t_10);
  ^
pysam/libcutils.c:6272:3: note: did you mean 'samtools_set_stderr'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:16:8: note: 
      'samtools_set_stderr' declared here
FILE * samtools_set_stderr(int fd);
       ^
pysam/libcutils.c:6444:5: error: implicit declaration of function 'bcftools_set_stdout_fn' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_stdout_fn(__pyx_t_13);
    ^
pysam/libcutils.c:6444:5: note: did you mean 'samtools_set_stdout_fn'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:27:6: note: 
      'samtools_set_stdout_fn' declared here
void samtools_set_stdout_fn(const char * fn);
     ^
pysam/libcutils.c:6455:5: error: implicit declaration of function 'bcftools_set_stdout' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_stdout(__pyx_t_10);
    ^
pysam/libcutils.c:6851:7: error: implicit declaration of function 'bcftools_set_stdout_fn' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
      bcftools_set_stdout_fn(__pyx_t_13);
      ^
pysam/libcutils.c:6991:7: error: implicit declaration of function 'bcftools_set_stdout' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
      bcftools_set_stdout(__pyx_t_10);
      ^
pysam/libcutils.c:7022:5: error: implicit declaration of function 'bcftools_set_stdout_fn' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_stdout_fn(((char const *)"-"));
    ^
pysam/libcutils.c:7287:5: error: implicit declaration of function 'bcftools_set_optind' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_optind(1);
    ^
pysam/libcutils.c:7287:5: note: did you mean 'samtools_set_optind'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:45:6: note: 
      'samtools_set_optind' declared here
void samtools_set_optind(int);
     ^
pysam/libcutils.c:7316:5: error: implicit declaration of function 'bcftools_set_optind' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_set_optind(0);
    ^
pysam/libcutils.c:7366:22: error: implicit declaration of function 'bcftools_main' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    __pyx_v_retval = bcftools_main((__pyx_v_n + 2), __pyx_v_cargs);
                     ^
pysam/libcutils.c:7366:22: note: did you mean 'samtools_main'?
/private/var/folders/vq/mc2zrs352hl8pykqzn0nt25r0000gn/T/easy_install-oi3q60rr/pysam-0.15.2/samtools/samtools.pysam.h:47:12: note: 
      'samtools_main' declared here
extern int samtools_main(int argc, char *argv[]);
           ^
pysam/libcutils.c:7435:3: error: implicit declaration of function 'bcftools_unset_stderr' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
  bcftools_unset_stderr();
  ^
pysam/libcutils.c:7435:3: note: did you mean 'bcftools_set_stderr'?
pysam/libcutils.c:6272:3: note: 'bcftools_set_stderr' declared here
  bcftools_set_stderr(__pyx_t_10);
  ^
pysam/libcutils.c:7471:5: error: implicit declaration of function 'bcftools_unset_stdout' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
    bcftools_unset_stdout();
    ^
11 errors generated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
make: *** [install] Error 1

[cjw85]

Hi @valentinavan,

The error above all seems to stem from when the installation is attempting to install the python module pysam.

If you forget about katuali for the moment and simply try to install pysam using

pip install pysam

I suspect you will run into a similar issue. Can you confirm? If this is the case I would suggest reading the pysam documentation for help.

Unrelated to the precise error above, I notice that you appear to be using macOS. Whilst I suspect that most of the tools that katuali uses can run under macOS, it is not something we typically test (there shouldn't be any issue with the katuali code itself).

[valentinavan]

Hi thanks for replying.

So, first of all, I got to a point where I did so many things that I lost track of them so I fresh started again by uninstalling everything.

Pysam:

1) it is recommended to install it through bioconda channel (being in the virtual env), which I did but even thought the installation was successful, I still get warnings/errors.
Plus, If I type "pysam" or "which pysam" I get the "not found" message but if I do "conda list", pysam is listed there (not in the "physical" folder, though).

2) So, I looked around on the internet and I tried to re-do the installation by using pip and giving the python path:
/Users/linnaeus/katuali/venv/bin/python -m pip install pysam
Got this messages:

Collecting pysam
Using cached pysam-0.16.0.1.tar.gz (3.6 MB)
Building wheels for collected packages: pysam
Building wheel for pysam (setup.py) ... done
Created wheel for pysam: filename=pysam-0.16.0.1-cp39-cp39-macosx_10_15_x86_64.whl size=2726557 sha256=a4adf37ed381f0810e49aa34034f4e45bb95e9f586511b521054d61b502fca13
Stored in directory: /Users/linnaeus/Library/Caches/pip/wheels/fb/96/5b/d62b12ccae437c41f0368bca424aa34b6d3f9fef4a38640edc
Successfully built pysam
Installing collected packages: pysam
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
katuali 0.3.2 requires drmaa, which is not installed.
katuali 0.3.2 requires gpustat, which is not installed.
katuali 0.3.2 requires pandas, which is not installed.
katuali 0.3.2 requires pysam==0.15.2, but you have pysam 0.16.0.1 which is incompatible.
Successfully installed pysam-0.16.0.1

3) I then installed, via conda, all these dependencies.
4) Checked for pysam. Cant find it. But with "conda list" is still there (still not in the "physical" folder, though). Plus is the 0.15.2 version and not the 0.16.0.1 like the message said.

Do not what to do... :)

Thanks in advance