-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update docker recipe #603
Update docker recipe #603
Conversation
Hi Javier, Thanks for picking this up! What is your plan with the docker images? How many with what content? I thought we might build images for releases and maybe some other images called something like esmvalcore-dev and esmvaltool-dev with only the dependencies we can use instead of the plain conda docker image to speed up the tests on CircleCI? The links to ci/dockercloud to not seem to work, is this expected? |
That is a good idea. So we keep three images for each repository:
And the links failing is not expected, no |
Unfortunately, pip has no way to install only the dependencies required for a package. What I am doing for the dev container is to install ESMValCore in develop mode and remove it afterwards. I have seen other solutions (e.g. creating the egg and looking for its requirements.txt) but I thought they were more confusing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I think this is a very good start. Could you please also
- Document how users can use the
esmvaltool
command from the container, e.g. here https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/quickstart/install.html or https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/quickstart/run.html? The markdown file in the docker directory can be removed. - Document how to build the container locally (e.g. in a comment at the top of the dockerfile)
- It would be good to empty the conda package cache after building with
conda clean --all
, to reduce the size - It would be good to remove the /source directory in both images after building to reduce the size
- If you provide a relative path for a recipe/config user file, it is relative to /source, maybe it would be nicer if this was e.g. the home directory of root or some other sensible directory
It does make sense to change the starting image for other jobs? I doubt it make sense for |
I agree
It would be best to leave that unchanged too, because now it mimics the steps followed by readthedocs. |
…ent.yml Change linux compilers to OS independent compilers
I added the |
Co-authored-by: Bouwe Andela <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mattiarighi @valeriupredoi Could you please test?
Cheers muchly @jvegasbsc 🍺 - my sincere apols for being so late testing this! I have actually postponed it back when you asked me then I completely forgot! My bit of testing: installed docker (Ubuntu/trusty) and checked: (base) valeriu@valeriu:~$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly. but next thing hit the wall: (base) valeriu@valeriu:~$ sudo docker pull esmvalgroup/esmvaltool:2.0
Error response from daemon: manifest for esmvalgroup/esmvaltool:2.0 not found also worth mentioning that So the image I am pulling is OK - it installed: Digest: sha256:6959d905c82b30d73805acfabc05709376d7b9bfd82f397b75f5d247332a3e45
Status: Downloaded newer image for esmvalgroup/esmvaltool:2.0.0b3 Help works but with a couple warnings: (base) valeriu@valeriu:~$ sudo docker run esmvalgroup/esmvaltool:2.0.0b3 --help
[sudo] password for valeriu:
Warning 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with a non-EPSG compliant axis order.
ERROR 1: PROJ: proj_create: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_create: cannot expand +init=epsg:4326 +type=crs
[esmvaltool help follows] Then it comes to this: (base) valeriu@valeriu:~/ESMValCore$ sudo docker run -v /home/valeriu/:/data/ esmvalgroup/esmvaltool:2.0.0b3 ~/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_test.yml
...
FileNotFoundError: [Errno 2] Specified recipe file does not exist: '/home/valeriu/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_test.yml' but that unfortunately exists. Also why are using How do you tell it where to get the recipes from? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pls see me comments 🍺
correction: as per the new instructions: sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:/data esmvalgroup/esmvaltool:2.0.0b3 ~/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_test.yml runs fine (I got no data, but it's complaining exactly bc of that, it finds the recipe). Can you pls remove the obsolete |
I am still very concerned about the size of the container, can we do something about that? |
Hi V, Please note that this pull request is for the docker container for ESMValCore (not ESMValTool). The tags
|
If you build the documentation locally, you should see a nice webpage explaining this, as part of the ESMValCore installation instructions. |
running it now |
it's fine, Javi's new instructions are spot-on! Just got confused by the old instructions file |
yarp! works fine! 🍺 Am gonna approve this since I was testing the wrong container but we need to be careful about |
The |
Very stupi error. Already changed to master |
We use it for running the unit tests: ESMValCore/.circleci/config.yml Lines 19 to 28 in 950cf40
They now take less than 3 minutes to complete: https://app.circleci.com/pipelines/github/ESMValGroup/ESMValCore?branch=master |
Updated docker recipe to install from source. Allows automation on the master branch
Tasks
yamllint
to check that your YAML files do not contain mistakesIf you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.
Closes #566