Version 2 of the r image

.gitignore

@@ -2,6 +2,8 @@
 renv.lock.bak
 renv/
 .tests.R
+.tests-base.R
+pkg.lock.bak
 
 # rstudio-server directories
 *.config

README.md

@@ -8,19 +8,28 @@ Docker image for running R code in OpenSAFELY, both locally and in production.
 * docker-compose
 * [just](https://github.com/casey/just)
 
+And the tests additionally require
+
+* curl
+* python3
+
 ## Building
 
 ```sh
-just build
+just build VERSION
 ```
 
-Under the hood, this builds `./Dockerfile` using docker-compose and buildkit.
+where `VERSION` is either v1 or v2.
+
+Under the hood, this builds `VERSION/Dockerfile` using docker-compose and buildkit.
 
-We currently build a lot of packages, so an initial build on a fresh checkout
+In v1, we currently build a lot of packages, so an initial build on a fresh checkout
 can take a long time (e.g. an hour).  However, to alleviate this, the
-Dockerfile is carefully designed to use local buildkit cache, so subequent
+v1/Dockerfile is carefully designed to use local buildkit cache, so subequent
 rebuilds should be very fast.
 
+In v2, where possible we install binary R packages for Linux from the Posit Public Package Manager.
+
 ## Adding new packages
 
 :warning: To do this you will need:
@@ -41,10 +50,18 @@ experience to approve the package.
 
 ### Install the package within Docker
 
+#### Under v1
+
 To add a package, by default it will be installed from CRAN.
 
 ```sh
-just add-package PACKAGE
+just add-package-v1 PACKAGE
+```
+
+If you need to install a package from another CRAN-like repository, specify its URL as the REPOS argument.
+
+```sh
+just add-package-v1 PACKAGE REPOS
 ```
 
 If you need to install a package from another CRAN-like repository, specify its URL as the REPOS argument.
@@ -54,14 +71,26 @@ just add-package PACKAGE REPOS
 ```
 
 This will attempt to install and build the package and its dependencies, and
-update the `renv.lock`. It will then rebuild the R image with the new lock file
+update the `v1/renv.lock`. It will then rebuild the R image with the new lock file
 and test it.
 
 Note that the first time you do this it will need to compile every
 included R package (because you won't have the R package builds cached
 locally). This can take **several hours**. (When we solve the caching
 problem here we'll be able to do this all in CI.)
 
+#### Under v2
+
+Add the package to _v2/packages.csv_. In this file, the first column is the package name, and the second column is for the repository. If the package is on CRAN simply leave the second column blank after the comma. If the package is not on CRAN please add it to the <https://opensafely-core.r-universe.dev> by adding it to the _packages.json_ file in <https://github.com/opensafely-core/opensafely-core.r-universe.dev>, then enter the relevant Linux binary package URL, likely `https://opensafely-core.r-universe.dev/bin/linux/noble/4.4/`.
+
+If the package requires any runtime dependencies add those to _v2/dependencies.txt_
+
+Then build the v2 image.
+
+```sh
+just build v2
+```
+
 ### Push the new Docker image to Github Container Registry
 
 You will need to configure authentication to GitHub's container registry first.
@@ -70,13 +99,13 @@ See [GitHub's documentation](https://docs.github.com/en/packages/working-with-a-
 When you have authentication configured, run:
 
 ```sh
-just publish
+just publish VERSION
 ```
 
 ### Commit changes to this repository
 
 Commit and push the small resulting change (should only be a few extra
-lines in `packages.csv` and `renv.lock`) to a branch, then get the changes
+lines in `VERSION/packages.csv`, `VERSION/packages.md`, and `VERSION/renv.lock`) to a branch, then get the changes
 merged via pull request.
 
 The review is a trivial exercise because the Docker image has already been
@@ -93,37 +122,82 @@ separately in the tech team manual. If you don't have access, ask in
 #### System dependencies
 
 If the package requires any system build dependencies (e.g. -dev packages with
-headers), they should be added to `build-dependencies.txt`. If it requires
-runtime dependencies, they should be added to `dependencies.txt`. Packages
+headers), they should be added to `VERSION/build-dependencies.txt`. If it requires
+runtime dependencies, they should be added to `VERSION/dependencies.txt`. Packages
 don't advertise their system dependencies, so you may need to figure them out
 by trying to add the package and reading any error output on failure.
 
-#### Installing an older version
+#### Installing an older version in the v1 image only
 
 If the package still fails to build, you may be able to install an older version.
 
 Find a previous version at `https://cran.r-project.org/src/contrib/Archive/{PACKAGE}/`, and attempt to install it specifically with
 
 ```sh
-just add-package PACKAGE@VERSION
+just add-package v1 PACKAGE@VERSION
 ```
 
 ## Building, testing, and publishing the rstudio image
 
 The rstudio image is based on the r image including rstudio-server. To build run
 
 ```sh
-just build-rstudio
+just build-rstudio VERSION
 ```
 
 To test that rstudio-server appears at `http://localhost:8787` run
 
 ```sh
-just test-rstudio
+just test-rstudio VERSION
 ```
 
 And then push the new rstudio image to the GitHub container registry with
 
 ```sh
-just publish-rstudio
+just publish-rstudio VERSION
+```
+
+## How to update the version of R and the packages
+
+In v2, we choose a date from which to install the packages from CRAN, we strongly recommend that the version of R in the image was the release version of R on this date. R release dates can be found on the [R wikipedia page](https://en.wikipedia.org/wiki/R_(programming_language)#Version_names).
+
+In v2, when installing packages we use a Posit Public Package Manager (PPPM) snapshot repository on the chosen `CRAN_DATE`.
+
+We use a fixed date because CRAN follows a rolling release model.
+As such we know that on a particular date CRAN has tested these package versions with the release version of R.
+Hence this is an extremely stable approach to choosing a set of package versions.
+And we can add additional packages at their versions on this date reliably (and without updating dependency packages already included in the image).
+
+The CRAN apt repository for R is available [here](https://cran.r-project.org/bin/linux/ubuntu/noble-cran40/) (note you may need to amend the Ubuntu codename in the URL if using a newer base image), find the package number you require and edit the number in _v2/dependencies.txt_ and _v2/build-dependencies.txt_.
+
+Then amend the `CRAN_DATE` and `REPOS` arguments in _v2/env_.
+
+To update run
+
+```sh
+just build v2
+```
+
+To test the updated image run
+
+```sh
+just test v2
 ```
+
+### How to choose a version of R and CRAN date
+
+Choose a version of R.
+
+Choose a CRAN date when that version of R.
+
+Essentially we follow a very similar approach to the versioned stack of the Rocker project. They list their R versions and CRAN dates on their [wiki](https://github.com/rocker-org/rocker-versioned2/wiki/Versions).
+
+We recommend not choosing a date within the first week of a new version of R being released, because there may be alot of packages updated on CRAN during this time.
+
+You then need to check that a PPPM snapshot repository exists for your chosen date. Navigate to <https://p3m.dev/client/#/repos/cran/setup> and inspect your chosen date. Set this as the `REPOS` argument in _v2/env_.
+
+If you choose a version of R that is not the current version of R we recommend following the rocker approach and choosing the CRAN date as the day before the next version of R was released. For example, if choosing R 4.4.1, R 4.4.2 was released on 2024-10-31 and so we would choose 2024-10-30 as the CRAN date.
+
+You can find out when the next release of R is scheduled for on the [R developer page](https://developer.r-project.org/).
+
+We set the `HTTPUserAgent` in the appropriate places so that we obtain binary R packages for Linux from the PPPM. There is additional information about this on the [PPPM website](https://p3m.dev/__docs__/admin/serving-binaries/#binary-user-agents).

docker-compose.yaml

@@ -1,20 +1,25 @@
 services:
   # used to build the production image
   r:
-    image: r
+    image: r:${MAJOR_VERSION}
     build:
       context: .
+      dockerfile: ${MAJOR_VERSION}/Dockerfile
       target: r
       cache_from:  # should speed up the build in CI, where we have a cold cache
-        - ghcr.io/opensafely-core/base-docker
-        - ghcr.io/opensafely-core/r
+        - ghcr.io/opensafely-core/base-docker:${BASE}
+        - ghcr.io/opensafely-core/r:${MAJOR_VERSION}
       args:
         # this makes the image work for later cache_from: usage
         - BUILDKIT_INLINE_CACHE=1
         # env vars supplied by make/just
         - BUILD_DATE
         - REVISION
         - VERSION
+        - BASE
+        - MAJOR_VERSION
+        - CRAN_DATE
+        - REPOS
     init: true
     platform: linux/amd64
   add-package:
@@ -28,7 +33,7 @@ services:
     platform: linux/amd64
   rstudio:
     extends: r
-    image: rstudio
+    image: rstudio:${MAJOR_VERSION}
     build:
       target: rstudio
       args:

justfile

@@ -5,40 +5,55 @@ export DOCKER_BUILDKIT := "1"
 export COMPOSE_DOCKER_CLI_BUILD := "1"
 
 # build the R image locally
-build: 
+build version:
     #!/usr/bin/env bash
     set -euo pipefail
 
+    source {{ version }}/env
 
     # set build args for prod builds
     export BUILD_DATE=$(date -u +'%y-%m-%dT%H:%M:%SZ')
     export GITREF=$(git rev-parse --short HEAD)
 
     # build the thing
-    docker-compose build --pull r
+    docker-compose --env-file {{ version }}/env build --pull r
 
+    if [ "{{ version }}" = "v1" ]; then
+      # update renv.lock
+      cp ${MAJOR_VERSION}/renv.lock ${MAJOR_VERSION}/renv.lock.bak
+      # cannot use docker-compose run as it mangles the output
+      docker run --platform linux/amd64 --rm r:{{ version }} cat /renv/renv.lock > ${MAJOR_VERSION}/renv.lock
+    elif [ "{{ version }}" = "v2" ]; then
+      # update pkg.lock
+      cp ${MAJOR_VERSION}/pkg.lock ${MAJOR_VERSION}/pkg.lock.bak
+      # cannot use docker-compose run as it mangles the output
+      docker run --platform linux/amd64 --rm r:{{ version }} cat /pkg.lock > ${MAJOR_VERSION}/pkg.lock
+    fi
 
+    # render the packages.md file
+    {{ just_executable() }} render {{ version }}
+
+# render the version/packages.md file
+render version:
+    docker run --platform linux/amd64 --env-file {{ version }}/env --entrypoint bash --rm -v "/$PWD:/out" -v "$PWD/scripts:/out/scripts" r:{{ version }} "/out/scripts/render.sh"
+
 # build and add a package and its dependencies to the image
-add-package package repos="NULL":
-    bash ./add-package.sh {{ package }} {{ repos }}
+add-package-v1 package repos="NULL":
+    bash v1/scripts/add-package.sh {{ package }} {{ repos }}
 
 # r image containing rstudio-server
-build-rstudio:
-    #!/usr/bin/env bash
-    set -euo pipefail
-
-    # Set RStudio Server .deb filename
-    export RSTUDIO_BASE_URL=https://download2.rstudio.org/server/focal/amd64/
-    export RSTUDIO_DEB=rstudio-server-2024.09.0-375-amd64.deb
-    docker-compose build --pull rstudio
+build-rstudio version:
+    docker-compose --env-file {{ version }}/env build --pull rstudio
 
 # test the locally built image
-test image="r": build
-    bash ./test.sh "{{ image }}"
+test version:
+    #!/usr/bin/env bash
+    source {{ version }}/env
+    bash tests/test.sh {{ version }}
 
 # test rstudio-server launches
-test-rstudio: _env
-    bash ./test-rstudio.sh
+test-rstudio version: _env
+    bash tests/test-rstudio.sh {{ version }}
 
 _env:
     #!/bin/bash
@@ -47,14 +62,24 @@ _env:
     echo "HOSTPLATFORM=$(docker info -f '{{{{ lower .ClientInfo.Os }}')" >> .env
 
 # lint source code
-lint:
+lint version:
     docker pull hadolint/hadolint
-    docker run --rm -i hadolint/hadolint < Dockerfile
+    docker run --rm -i hadolint/hadolint < {{ version }}/Dockerfile
 
-publish:
-    docker tag r ghcr.io/opensafely-core/r:latest
-    docker push ghcr.io/opensafely-core/r:latest
+publish version:
+    #!/usr/bin/env bash
+    docker tag r:{{ version }} ghcr.io/opensafely-core/r:{{ version }}
+    docker push ghcr.io/opensafely-core/r:{{ version }}
+    if [ "{{ version }}" = "v1" ]; then
+      docker tag r:{{ version }} ghcr.io/opensafely-core/r:latest
+      docker push ghcr.io/opensafely-core/r:latest
+    fi
 
-publish-rstudio:
-    docker tag rstudio ghcr.io/opensafely-core/rstudio:latest
-    docker push ghcr.io/opensafely-core/rstudio:latest
+publish-rstudio version:
+    #!/usr/bin/env bash
+    docker tag rstudio:{{ version }} ghcr.io/opensafely-core/rstudio:{{ version }}
+    docker push ghcr.io/opensafely-core/rstudio:{{ version }}
+    if [ "{{ version }}" = "v1" ]; then
+      docker tag rstudio:{{ version }} ghcr.io/opensafely-core/rstudio:latest
+      docker push ghcr.io/opensafely-core/rstudio:latest
+    fi