Test if multi-stage build can optimize rocker image

[BretFisher]

Right now, due to the limits of how docker works, our custom rocker (rocker/verse plus nonmem plus psn) works, we have to manually install nonmem and psn again on top of FROM rocker/verse to get them all in one image.

This is a duplicate effort from what we already do in the nonmem and psn images.

It may be possible to copy over at least some of nonmem/psn from the existing images into our custom rocker image with multi-stage builds, so I'll create a custom image (for now) so we can test.

[dpastoor]

@BretFisher is my understanding correct that docker compose does not build images sequentially? Namely, if you have the PsN image, and the Rocker image, it does not layer PsN and Rocker together, rather they are just orchestrated together - there is no way to touch PsN (or any CLI in another image) from the Rocker image? It seems that would break the whole encapsulation abstraction, for better or worse.

[dpastoor]

One nascent thought is that, Nonmem and PsN are both pretty static. Given that we might rebuild them every 3-6 months, its really the later R portion that changes, and where the most customization may occur. I am thinking of even something like a simple bash script that just cats the separate images together. If the nonmem image is the first part of the dockerfile, it shouldn't change, hence those layers would remain cached, and the build would really just handle installing the latest version of packages/rstudio/etc.

[dpastoor]

The other thing, is generally in our community, the space of these images is rarely going to be a bottleneck. I would encourage not being concerned trying to shave image size, rather the thing that would be amazing to 'figure out' is some of the downstream points that don't have elegant solutions at this point - using different versions of Rstudio (preview vs stable release), using tip vs stable (CRAN) version of packages. Having a way of generating such customization in a user specified way would go a long long way towards community adoption, because people could mimic their environment and preferences. For example, a huge win for the nightlies of Rstudio right now is they have incorporated a built in terminal, which has been amazing as no longer need to keep a second terminal open and exec'd in just for occasional bash/git commands. Likewise, though the OSS CI systems are pretty nice, being able to generate a matrix of package configurations for testing or debugging would be great for local package development and maintenance, and could potentially incentivize the broader community to take a look at this project. Just some food for thought

[BretFisher]

My motivation is to simplify Dockerfile construction and duplicate effort/code (as we have today) and not speed of build (they build in background on Docker Cloud) or image size (they are using Docker's single-instance-storage so agreed on not caring so much). All the images are lean except the rocker one.

I'm not sure how to answer your first question, but if you look at psn image Dockerfile, it's FROM the nonmem image, so yes they are layered in that sense. Today our rocker image is FROM rocker/verse, and then it re-installs nonmem and psn on top of that. I can't really "merge two images" like the psn + rocker/verse, but we can get closer with the new multi-stage builds, which is what this GHIssue is for.

Yes we have to combine products into one image. You can see in the table of this repo's README what each image containers (nonmem image is just nonmem, psn image has nonmem plus psn installed, and so on).

In terms of their storage in a Docker image cache (e.g. on your machine, or a server) they are mostly not duplicating storage because of their FROM being each other. Since the psn image is FROM osmosisfoundation/nonmem, it's technically only storing the difference between the two on disk (assuming you have both in your image list).

Yes having them access each other at runtime is not generally possible, and if it were (via file system), it would break lots of concepts around isolated containers. In Swarm we generally can't expect two containers to always be on the same node anyway.

Remember that generally, users will not be building these images themselves, just using the images we provide via dockerhub: https://hub.docker.com/u/osmosisfoundation/ and our docker-compose.yml in this repo helps running those much easier for them, but not required if they are advanced.

On the preview/stable and daily releases, that's good feedback and likely should be a new GHIssue with details for each specific product we want to make more releases available for, including some background on those releases and use cases (since I'm the one that doesn't know anything about these products, I only know the needs when someone gives me specifics on user stories 😄 ).