Installing from conda is a total turkey #660

valeriupredoi · 2020-06-02T17:29:47Z

conda install esmvaltool -c esmvalgroup -c conda-forge

results in successive failures of conda to solve the environment and piling up memory consumption in the process (on my puny 4GB RAM machine the conda python process reached 3GB and went straight to swap twice, forcing me to restart the whole machine twice). I am aware this is partly conda's fault for trying to be something that'll never be but it's also our fault for not optimizing the package; we need to fix this since people are trying to install it this way and not from source and giving up in the process (one case reported to me in the last 24h); I will look into this tomorrow but I ask @bouweandela and @bjoernbroetz to have a think too 🍺

bouweandela · 2020-06-02T18:44:24Z

Maybe you could open an issue about this with conda? https://github.com/conda/conda/issues

It looks like similar issues have been reported before, e.g. conda/conda#7787, conda/conda#8650

bouweandela · 2020-06-02T20:07:32Z

My installation finished:

$ \time -v conda create -n esmvaltool_deps -c esmvalgroup -c conda-forge esmvaltool

resulted in

	Command being timed: "conda create -n esmvaltool_deps -c esmvalgroup -c conda-forge esmvaltool"
	User time (seconds): 4681.48
	System time (seconds): 80.85
	Percent of CPU this job got: 109%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:12:41
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7334184
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1168
	Minor (reclaiming a frame) page faults: 23161833
	Voluntary context switches: 294555
	Involuntary context switches: 455039
	Swaps: 0
	File system inputs: 1657920
	File system outputs: 4395656
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

i.e. about 1 hour to solve the environment and around 7GB or RAM.

valeriupredoi · 2020-06-02T20:10:23Z

tiptop, mate, tiptop 🤣

valeriupredoi · 2020-06-02T20:11:46Z

I can't see your comment about the env solving quickly w/o ncl - did you remove it?

bouweandela · 2020-06-03T15:22:34Z

Sorry, yes, because it was confusing, e.g.
conda create -n test -c esmvalgroup -c conda-forge esmvaltool-r imagemagick ncl takes about a minute to resolve and 1.2 GB or RAM and installs the exact same packages as conda create -n test -c esmvalgroup -c conda-forge esmvaltool, which takes an hour and needs 7GB RAM.

valeriupredoi · 2020-06-04T12:19:43Z

what does -r do?

valeriupredoi · 2020-06-04T12:56:20Z

OK @bouweandela some progress twoards understanding this:

conda env create ... is actually very quick and only taking 1.5G which is normal; why is conda create ... so bulky is puzzling;
installing just esmvalcore fails:

(base) valeriu@valeriu:~/ESMValTool$ \time conda install -c esmvalgroup esmvalcore
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                               

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Command exited with non-zero status 1
31.94user 0.84system 0:39.12elapsed 83%CPU (0avgtext+0avgdata 212256maxresident)k
104568inputs+8outputs (48major+73946minor)pagefaults 0swaps

-> it would be very useful if you could create an up to date esmvalcore package so we can see if the issues with installing esmvaltool stem from conflicts with esmvalcore deps - usually conda decides it can still solve an env that is otherwise insolvable (if fewer deps were in the list) and goes on forever with no avail

valeriupredoi · 2020-06-04T12:58:54Z

UPDATE: \time conda install -c esmvalgroup -c conda-forge esmvalcore works (duh!) we need to change the instructions at https://anaconda.org/ESMValGroup/esmvalcore

stats: 85.24user 8.15system 3:10.18elapsed 49%CPU (0avgtext+0avgdata 504072maxresident)k
721960inputs+2167944outputs (94major+418051minor)pagefaults 0swaps

valeriupredoi · 2020-06-04T13:09:42Z

OK we really need a new esmvalcore package, am hitting this (proj-related issue that we solved a month or so ago):

(base) valeriu@valeriu:~/ESMValTool$ esmvaltool --help
ERROR 1: PROJ: proj_create_from_database: Open of /home/valeriu/miniconda3/share/proj failed
PROJ: proj_create_from_database: Cannot find proj.db
Traceback (most recent call last):
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS
Traceback (most recent call last):
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/valeriu/miniconda3/bin/esmvaltool", line 11, in <module>
    load_entry_point('ESMValCore==2.0.0b9', 'console_scripts', 'esmvaltool')()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
    return ep.load()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2445, in load
    return self.resolve()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2451, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_main.py", line 42, in <module>
    from ._recipe import TASKSEP, read_recipe_file
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe.py", line 14, in <module>
    from . import _recipe_checks as check
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe_checks.py", line 12, in <module>
    from .preprocessor import PreprocessingTask, TIME_PREPROCESSORS
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/__init__.py", line 13, in <module>
    from ._derive import derive
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 32, in <module>
    ALL_DERIVED_VARIABLES = _get_all_derived_variables()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 27, in _get_all_derived_variables
    f'esmvalcore.preprocessor._derive.{short_name}')
  File "/home/valeriu/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/sispeed.py", line 6, in <module>
    from .._regrid import regrid
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid.py", line 16, in <module>
    from ._regrid_esmpy import ESMF_REGRID_METHODS
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 4, in <module>
    import ESMF
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/__init__.py", line 70, in <module>
    from ESMF.api.esmpymanager import *
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/api/esmpymanager.py", line 11, in <module>
    from ESMF.interface.cbindings import *
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/cbindings.py", line 13, in <module>
    from ESMF.interface.loadESMF import _ESMF
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 131, in <module>
    raise ImportError('The ESMF shared library did not load properly.')
ImportError: The ESMF shared library did not load properly.

valeriupredoi · 2020-06-04T13:12:23Z

@bouweandela can you pls produce a new esmvalcore package, ship it to esmvalgroup, change the install instructions on the page and let me know when you've done that, please mate? I will test the installation from conda right away that's done, I suspect our conda woes will be fixed by doing this 🍺

bouweandela · 2020-06-04T13:59:08Z

what does -r do?

It installs only the esmvaltool-r subpackage, as described in our installation instructions.

we need to change the instructions at https://anaconda.org/ESMValGroup/esmvalcore

I've done that a few times already, but these instructions get overwritten with the default every time I upload a new package, so I'm not going to do that anymore. I reckon people should just use the installation instructions from our documentation.

OK we really need a new esmvalcore package, am hitting this (proj-related issue that we solved a month or so ago):

Could you please share all commands you ran to create this environment, so I can try to reproduce your issue?

valeriupredoi · 2020-06-08T10:18:51Z

conda install -c conda-forge -c esmvalgroup esmvalcore
esmvaltool --help

am trying it on Jasmin as well at the mo, will report back 🍺

bouweandela · 2020-06-08T13:14:17Z

Could you also share the commands you used to create the environment that you're installing into?

valeriupredoi · 2020-06-08T13:21:20Z

no other commands man, installing in the conda (base) - ideally when a user does conda install ... they should not create any environment prior to 🍺

bouweandela · 2020-06-08T13:39:17Z

I just downloaded conda (version 4.8.3) and ran

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# chose ~/conda-test as an installation directory
. conda-test/etc/profile.d/conda.sh
conda activate base
conda install -c conda-forge -c esmvalgroup esmvalcore
esmvaltool --help

and everything worked perfectly. What version of conda are you using? Can you provide a link for downloading from the archive: https://repo.anaconda.com/miniconda/

bouweandela · 2020-06-08T13:40:52Z

Or maybe a link to a docker container that runs the conda version that has the problem?

valeriupredoi · 2020-06-08T15:08:02Z

here are the traceback steps Monsieur Le Bouwe:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
conda install -c conda-forge -c esmvalgroup esmvalcore

note that I accepted conda to prepend the installation and activate base right away, equivalent to the snippet in .bashrc:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/valeriu/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/valeriu/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/valeriu/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/valeriu/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

and for the big finale:

(base) valeriu@valeriu:~$ esmvaltool --help
Traceback (most recent call last):
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS
Traceback (most recent call last):
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/valeriu/miniconda3/bin/esmvaltool", line 11, in <module>
    load_entry_point('ESMValCore==2.0.0b9', 'console_scripts', 'esmvaltool')()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
    return ep.load()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2445, in load
    return self.resolve()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2451, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_main.py", line 42, in <module>
    from ._recipe import TASKSEP, read_recipe_file
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe.py", line 14, in <module>
    from . import _recipe_checks as check
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe_checks.py", line 12, in <module>
    from .preprocessor import PreprocessingTask, TIME_PREPROCESSORS
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/__init__.py", line 13, in <module>
    from ._derive import derive
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 32, in <module>
    ALL_DERIVED_VARIABLES = _get_all_derived_variables()
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 27, in _get_all_derived_variables
    f'esmvalcore.preprocessor._derive.{short_name}')
  File "/home/valeriu/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/sispeed.py", line 6, in <module>
    from .._regrid import regrid
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid.py", line 16, in <module>
    from ._regrid_esmpy import ESMF_REGRID_METHODS
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 4, in <module>
    import ESMF
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/__init__.py", line 70, in <module>
    from ESMF.api.esmpymanager import *
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/api/esmpymanager.py", line 11, in <module>
    from ESMF.interface.cbindings import *
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/cbindings.py", line 13, in <module>
    from ESMF.interface.loadESMF import _ESMF
  File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 131, in <module>
    raise ImportError('The ESMF shared library did not load properly.')
ImportError: The ESMF shared library did not load properly.

a few pointers:

(base) valeriu@valeriu:~$ which conda
/home/valeriu/miniconda3/bin/conda
(base) valeriu@valeriu:~$ conda --version
conda 4.8.3
(base) valeriu@valeriu:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.6 LTS
Release:	14.04
Codename:	trusty
(base) valeriu@valeriu:~$ conda list esmvalcore
# packages in environment at /home/valeriu/miniconda3:
#
# Name                    Version                   Build  Channel
esmvalcore                2.0.0b9                    py_0    esmvalgroup

valeriupredoi · 2020-06-08T15:10:48Z

moar pointers:

(base) valeriu@valeriu:~$ conda list python
# packages in environment at /home/valeriu/miniconda3:
#
# Name                    Version                   Build  Channel
antlr-python-runtime      4.7.2                 py37_1001    conda-forge
msgpack-python            1.0.0            py37h99015e2_1    conda-forge
python                    3.7.6           cpython_h8356626_6    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-stratify           0.1.1           py37h03ebfcd_1002    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
(base) valeriu@valeriu:~$ conda list esmf
# packages in environment at /home/valeriu/miniconda3:
#
# Name                    Version                   Build  Channel
esmf                      8.0.1           nompi_hbeb3ca6_0    conda-forge

Note that on Jasmin the install didn't even finish, it was going in circles trying to solve the environment, on my laptop it was very snappy (as pointed above)

valeriupredoi · 2020-06-08T15:19:21Z

Massive hint: when you uninstall esmvalcore and install esmf(since it'll be removed with the uninstall of esmvalcore) that one is compiled correctly against mpi with mpich:

(base) valeriu@valeriu:~$ conda list esmf
# packages in environment at /home/valeriu/miniconda3:
#
# Name                    Version                   Build  Channel
esmf                      8.0.1           mpi_mpich_h213fab7_100    conda-forge

right after this if you re-install esmvalcore, then it'll work nicely. You need to check that the requirement for esmf in the conda recipe for esmvalcore is set correctly for mpi with mpich 🍺

valeriupredoi · 2020-06-08T16:07:53Z

strange, I have repeated the install steps on Jasmin and all works fine there out the box with

(base) [valeriu@jasmin-sci2 ~]$ conda list esmf
# packages in environment at /home/users/valeriu/miniconda3:
#
# Name                    Version                   Build  Channel
esmf                      8.0.1           nompi_hbeb3ca6_0    conda-forge

I still reckon it's best to run a few installation tests on different OS's after the package gets built. It might be that my OS is old (Ubuntu 14 is ancient, 6 years old) but even so it shouldn't matter

jasmin's OS:

(base) [valeriu@jasmin-sci2 ~]$ lsb_release -a
LSB Version:	:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 6.10 (Santiago)
Release:	6.10
Codename:	Santiago

bouweandela · 2020-06-08T16:10:58Z

It also works fine on my laptop and on CircleCI, so I wouldn't be suprised if it's somehow related to your OS or user profile.

valeriupredoi · 2020-06-08T16:15:51Z

what are you running on your laptop - if it's a Debian-based OS then am OK to call this solved (well, the installation of esmvalcore that is, for now, I am still struggling with esmvaltool) 🍺

bouweandela · 2020-06-08T16:17:14Z

I'm running ubuntu 20.04. Trying to reproduce your issue now with ubuntu 14.04.

bouweandela · 2020-06-08T16:18:50Z

Yes, I managed to reproduce it!

valeriupredoi · 2020-06-08T16:25:11Z

excellent! did you build a container with that ye olde OS? Got good news - I managed to install esmvaltool on Jasmin straight from conda (well, it's installing now, but the env solving lasted about a smoke's time 😁 ) - this was after I've installed esmvalcore from conda so it may be that either this is a workaround or conda fixed their troubles (I'll uninstall them both and try installing esmvaltool from scratch)

valeriupredoi · 2020-06-08T16:26:55Z

I'm running ubuntu 20.04. Trying to reproduce your issue now with ubuntu 14.04.

way to go updated OS, man! What's that one called, Flying Dutchman? 😁

bouweandela · 2020-06-08T16:32:55Z

Indeed I did

docker pull ubuntu:14.04
docker run -it ubuntu:14.04

and that reproduced the problem. I also found lots of other people having the same issue: pytorch/pytorch#2575 (comment)

It looks like the publicly available Ubuntu 14.04 is no longer maintained: https://ubuntu.com/about/release-cycle, so maybe now would be a good time to upgrade?

valeriupredoi · 2020-06-08T16:36:24Z

aw mate, 14.04 hasn't been supported since April last year 😁 but you know, updating an OS that is 11 version increments old is probably suicidal. Nah, it's fine, we should log this somewhere and if some oldtimer like me complains in a future issue we will know what he problem is

valeriupredoi · 2020-06-08T16:48:08Z

new roadblock; while installing esmvaltool from conda (conda install -c conda-forge -c esmvalgroup esmvaltool) it complains about Julia:

ERROR conda.core.link:_execute(700): An error occurred while installing package 'esmvalgroup::esmvaltool-julia-2.0.0b4-0'.
Rolling back transaction: \

valeriupredoi · 2020-06-08T17:01:09Z

full output:

Rolling back transaction: - 
done

LinkError: post-link script failed for package esmvalgroup::esmvaltool-julia-2.0.0b4-0
location of failed script: /home/users/valeriu/miniconda3/bin/.esmvaltool-julia-post-link.sh
==> script messages <==
Executable julia not found! Exiting...

==> script output <==
stdout: 
stderr: 
return code: 1

()

2005.28user 528.46system 42:49.91elapsed 98%CPU (0avgtext+0avgdata 1311888maxresident)k
2302216inputs+9258752outputs (1063major+34605267minor)pagefaults 0swaps

and esmvaltool --help dont work 😢

valeriupredoi · 2020-06-08T17:04:33Z

that's it for me for today, off to make dinner 🍕

bouweandela · 2020-06-09T12:59:48Z

new roadblock; while installing esmvaltool from conda (conda install -c conda-forge -c esmvalgroup esmvaltool) it complains about Julia:

It would be really helpful if you could read the installation instructions. Quoting from there:

In order to install the Conda package, you will need both conda and Julia pre-installed, this is because Julia cannot be installed from conda.

valeriupredoi · 2020-06-09T13:10:56Z

you reckon I don't know that? 😁 I am not happy with the conda installation to fail because of that - we need to make it install and throw a big warning at the end saying this and that Julia but not to fail altogether; this is in line with that R diagnostic test that is still failing if R is not installed (bump: approve that PR, mate) 🍺

bouweandela · 2020-06-09T13:46:26Z

we need to make it install and throw a big warning at the end saying this and that Julia

Maybe you could open a pull request to improve the error message?

valeriupredoi · 2020-06-09T16:03:25Z

oh I will and it'll contain a few changes:

you do realize that just by pointing the user to the Julia download page will not cut the mustard - that page is cryptic to start with and the actual installation procedures specific for a conda env to be correctly built are not there;
can we default the installation from conda (conda install -c conda-forge -c esmvalgroup esmvaltool) to the installation without needing Julia? That would be absolutely awesome because it'll be very straightforward

valeriupredoi · 2020-06-09T16:41:59Z

OK I managed to catch this loose tiger that conda pretends to be: if you want to install esmvaltool from conda you will end up spending the good part of your day looking at that spindle spin and not solve the environment - if you install esmvalcore first, esmvaltool env will be solved in 3min and chuck at most 2G of mem. I will make a PR with updated instructions including how to install and softlink julia before installing 🍺

bouweandela · 2020-06-11T15:13:44Z

can we default the installation from conda (conda install -c conda-forge -c esmvalgroup esmvaltool) to the installation without needing Julia? That would be absolutely awesome because it'll be very straightforward

No, I think not, because then it will not work if people try to run a Julia diagnostic. For advanced users that know they do not want Julia diagnostics, we have the subpackages, e.g. conda install esmvaltool-python esmvaltool-r esmvaltool-ncl will get you esmvaltool without Julia. Maybe we could make it possible to install the julia dependencies later, once we have the new command line interface. Or we could try to change the script so it installs Julia as well as the Julia dependencies, but I'm afraid that might be bit error prone.

valeriupredoi · 2020-06-11T15:40:36Z

OK let's keep it this way then, I am happier now that we are actually telling the user how to install Julia and make the executable available. I will try today or tomorrow see if we can get Julia back inside the conda environment, although doubt much has changes in the meantime 🍺

valeriupredoi · 2020-06-30T16:24:41Z

closing this since it works now, beware a reopen 😁

valeriupredoi · 2020-07-28T12:07:06Z

ah the dreaded reopen 😁 As per the instructions installing from conda via simple conda invocation gets stuck in conflict zone:

conda install -c esmvalgroup -c conda-forge esmvalcore
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.

on an anaconda install, in a base environment:

valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ conda -V
conda 4.8.3
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ python -V
Python 3.8.3

I am adding the conda install test in #700 now and we can see this distributed to both Linux and OSX for different Python versions 🍺

valeriupredoi · 2020-07-28T16:13:13Z

too much fluff here, closing this and reopening a new, more precise one 🍺

valeriupredoi added the installation Installation problem label Jun 2, 2020

valeriupredoi assigned bouweandela, bjoernbroetz and valeriupredoi Jun 2, 2020

valeriupredoi mentioned this issue Jun 9, 2020

added optimal install instructions for install from conda ESMValGroup/ESMValTool#1675

Closed

18 tasks

bouweandela mentioned this issue Jun 10, 2020

Conda package 2.0.0b4 broken? ESMValGroup/ESMValTool#1676

Closed

bouweandela mentioned this issue Jun 12, 2020

Speed up conda installation ESMValGroup/ESMValTool#1677

Merged

5 tasks

valeriupredoi closed this as completed Jun 30, 2020

valeriupredoi reopened this Jul 28, 2020

valeriupredoi closed this as completed Jul 28, 2020

Installing from conda is a total turkey #660

Installing from conda is a total turkey #660

Comments

valeriupredoi commented Jun 2, 2020

bouweandela commented Jun 2, 2020

bouweandela commented Jun 2, 2020

valeriupredoi commented Jun 2, 2020

valeriupredoi commented Jun 2, 2020

bouweandela commented Jun 3, 2020

valeriupredoi commented Jun 4, 2020

valeriupredoi commented Jun 4, 2020

valeriupredoi commented Jun 4, 2020 • edited Loading

valeriupredoi commented Jun 4, 2020

valeriupredoi commented Jun 4, 2020

bouweandela commented Jun 4, 2020

valeriupredoi commented Jun 8, 2020

bouweandela commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

bouweandela commented Jun 8, 2020 • edited Loading

bouweandela commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020 • edited Loading

bouweandela commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

bouweandela commented Jun 8, 2020

bouweandela commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020 • edited Loading

valeriupredoi commented Jun 8, 2020

bouweandela commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

valeriupredoi commented Jun 8, 2020

bouweandela commented Jun 9, 2020

valeriupredoi commented Jun 9, 2020

bouweandela commented Jun 9, 2020

valeriupredoi commented Jun 9, 2020

valeriupredoi commented Jun 9, 2020 • edited Loading

bouweandela commented Jun 11, 2020

valeriupredoi commented Jun 11, 2020

valeriupredoi commented Jun 30, 2020

valeriupredoi commented Jul 28, 2020 • edited Loading

valeriupredoi commented Jul 28, 2020

valeriupredoi commented Jun 4, 2020 •

edited

Loading

bouweandela commented Jun 8, 2020 •

edited

Loading

valeriupredoi commented Jun 8, 2020 •

edited

Loading

valeriupredoi commented Jun 8, 2020 •

edited

Loading

valeriupredoi commented Jun 9, 2020 •

edited

Loading

valeriupredoi commented Jul 28, 2020 •

edited

Loading