Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added dockerfile #177

Merged
merged 47 commits into from
May 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
0431d13
added dockerfile
thawn Apr 24, 2024
bdfa394
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 24, 2024
1a4aecd
Merge branch 'main' into docker
Intron7 May 2, 2024
ebe4997
install dependencies into separate container
thawn May 3, 2024
32d7b4d
fix rapids_version
thawn May 3, 2024
cf921c5
do not update apt packages to avoid updating the nvidia cuda installa…
thawn May 3, 2024
dafa76b
split build into conda and pip part to make debugging faster
thawn May 3, 2024
e90570b
made docker-push.sh executable
thawn May 3, 2024
27ffd78
fix CUDA_ERROR_OPERATING_SYSTEM
thawn May 3, 2024
9a107c8
first attempt at a github CI workflow that builds the dependency cont…
thawn May 3, 2024
11cac6e
attempt to prevent running out of disk space
thawn May 3, 2024
aae7329
increase space reserved on root
thawn May 3, 2024
56780fa
increase space on root
thawn May 3, 2024
17b0fc8
shorten name, increase space
thawn May 3, 2024
6583f08
reduced space
thawn May 3, 2024
3ef5cc6
less verbose apt-get output
thawn May 6, 2024
352a5a9
disable Updating database messages from apt
thawn May 6, 2024
bfe93ca
switch to noninteractive debian frontend
thawn May 6, 2024
e87a013
apparently DEBIAN_FRONTEND=noninteractive is not sufficient
thawn May 6, 2024
0efce32
build both deps and final image
thawn May 6, 2024
8abf2f7
improved documentation and naming
thawn May 6, 2024
eaf50de
rename docker github action yml file
thawn May 6, 2024
61b0fac
added release note
thawn May 6, 2024
ace3eed
fix trailing white space
thawn May 6, 2024
3b0f1ac
fix trailing white space
thawn May 6, 2024
ebe25d9
more verbose job step name
thawn May 6, 2024
7ada944
Merge branch 'main' into docker
Intron7 May 6, 2024
0c65944
use admin setting
flying-sheep May 7, 2024
3d02a28
Merge branch 'main' into docker
flying-sheep May 7, 2024
dc78669
worth a try
flying-sheep May 7, 2024
e24dc0c
trigger GitHub actions
thawn May 7, 2024
e40fd3a
Merge branch 'docker' of github.com:thawn/rapids_singlecell into docker
thawn May 7, 2024
087d213
Try to increase disk space more
thawn May 7, 2024
1edf8f7
reserved disk space was too much
thawn May 7, 2024
01530b6
try to reduce swap size
thawn May 7, 2024
144a58b
try smaller rootfs size
thawn May 7, 2024
d21cb37
fix cache repo
thawn May 7, 2024
2827ff3
push only releases to registry
thawn May 8, 2024
f227e83
try without build context
thawn May 8, 2024
dcff42a
try to use digest as build context
thawn May 8, 2024
0fd91eb
try another build context
thawn May 8, 2024
f400e5b
try another build context
thawn May 8, 2024
4e66815
trigger GitHub actions
thawn May 8, 2024
bbab508
list available docker images
thawn May 8, 2024
475b4aa
another build context
thawn May 8, 2024
65e621a
moved listing docker images to the end
thawn May 8, 2024
1ca2899
Safer script
flying-sheep May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# This workflow will build two Docker image and push then to GitHub Packages Container registry:
# - a base image with the dependencies
# - a main image with the application code

name: Docker

on:
push:
branches: [main]
paths:
- 'docker/**'
- '.github/workflows/docker-base.yml'
- 'conda/*.yml'
- 'pyproject.toml'
pull_request:
branches: [main]
paths:
- 'docker/**'
- '.github/workflows/docker-base.yml'
- 'conda/*.yml'
- 'pyproject.toml'
release:
types: [published]

jobs:
build_docker_images:
strategy:
matrix:
RAPIDS_VER:
- 24.04
name: Build Docker images
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
attestations: write
steps:
- name: Maximize build disk space
uses: easimon/maximize-build-space@v10
with:
remove-dotnet: true
remove-android: true
remove-haskell: true
remove-codeql: true
root-reserve-mb: 35840
swap-size-mb: 1048

- name: Check out the repo
uses: actions/checkout@v4

- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker base image
id: meta-base
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/${{ github.repository }}-deps

- name: create yaml file for conda environment
run: |
grep -v -- '- rapids-singlecell' conda/rsc_rapids_${{ matrix.RAPIDS_VER }}.yml > docker/rsc_rapids.yml
shell: bash

- name: Build and push Docker base images
id: push-base
uses: docker/build-push-action@v5
with:
context: ./docker/
file: ./docker/Dockerfile.deps
push: ${{ github.event_name == 'release' }}
tags: ${{ steps.meta-base.outputs.tags }}
labels: ${{ steps.meta-base.outputs.labels }}
#cache-from: type=registry,ref=ghcr.io/${{ github.repository }}-deps

- name: Generate artifact attestation for base image
if: github.event_name == 'release'
uses: actions/attest-build-provenance@v1
with:
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
subject-digest: ${{ steps.push-base.outputs.digest }}
push-to-registry: true

- name: Extract metadata (tags, labels) for main Docker image
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/${{ github.repository }}

- name: Build and push main Docker images
id: push
uses: docker/build-push-action@v5
with:
context: ./docker/
file: ./docker/Dockerfile
push: ${{ github.event_name == 'release' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
#cache-from: type=registry,ref=ghcr.io/${{ github.repository }}
build-contexts: |
rapids-singlecell-deps=docker-image://${{ steps.meta-base.outputs.tags }}

- name: Generate artifact attestation for main image
if: github.event_name == 'release'
uses: actions/attest-build-provenance@v1
with:
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true

- name: list docker images
run: |
docker image ls -a
shell: bash
20 changes: 20 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM rapids-singlecell-deps

ARG GIT_ID=main

SHELL ["/bin/bash", "-euo", "pipefail", "-c"]

ENV PATH=/opt/conda/bin:$PATH

RUN <<EOF
# install rapids_singlecell from source
set -x
mkdir /src
cd /src
git clone https://github.com/scverse/rapids_singlecell.git
cd rapids_singlecell
git checkout ${GIT_ID}
/opt/conda/bin/python -m pip install --no-cache-dir -e .
EOF

ENTRYPOINT ["/opt/conda/bin/ipython"]
40 changes: 40 additions & 0 deletions docker/Dockerfile.deps
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
ARG CUDA_VER=11.8.0
ARG LINUX_VER=ubuntu22.04

FROM nvidia/cuda:${CUDA_VER}-base-${LINUX_VER}

SHELL ["/bin/bash", "-euo", "pipefail", "-c"]

ARG PYTHON_VER=3.11

ENV PATH=/opt/conda/bin:$PATH
ENV PYTHON_VERSION=${PYTHON_VER}

COPY --from=condaforge/miniforge3:24.3.0-0 /opt/conda /opt/conda

COPY rsc_rapids.yml rsc_rapids.yml

ARG GIT_ID=main
ARG DEBIAN_FRONTEND=noninteractive

RUN <<EOF
# install conda environment
set -x
apt-get -qq update
apt-get -q -o=Dpkg::Use-Pty=0 -y dist-upgrade
apt-get -q install -y -o=Dpkg::Use-Pty=0 git
apt-get -q clean -y
mamba env update -n base -f rsc_rapids.yml
mamba install -y -n base pytest -c conda-forge
mamba clean -afy
EOF

RUN <<EOF
# install rapids_singlecell dependencies
set -x
/opt/conda/bin/python -m pip install --no-cache-dir git+https://github.com/scverse/rapids_singlecell.git@${GIT_ID}
/opt/conda/bin/python -m pip uninstall -y --no-cache-dir rapids-singlecell
/opt/conda/bin/python -m pip cache purge
EOF

ENTRYPOINT ["/opt/conda/bin/ipython"]
12 changes: 12 additions & 0 deletions docker/docker-push.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/bash
set -euxo pipefail

docker_account=scverse
rapids_version=24.04
grep -v -- '- rapids-singlecell' conda/rsc_rapids_${rapids_version}.yml > rsc_rapids.yml
docker build -t rapids-singlecell-deps:latest -f docker/Dockerfile.deps .
rm rsc_rapids.yml
docker build -t rapids-singlecell:latest -f docker/Dockerfile .
latest_id=$( docker images |grep -e "rapids-singlecell[ \t]*latest"|head -n1|awk '{print $3}' )
#docker tag $latest_id $docker_account/rapids-singlecell:latest
#docker push $docker_account/rapids-singlecell:latest
2 changes: 2 additions & 0 deletions docs/release-notes/0.10.3.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
```{rubric} Features
```
* adds the flavor `seurat_v3_paper` to `pp.highly_variable_genes` {pr}`170` {smaller}`S Dicks`
* build docker images in github CI {pr}`177` {smaller}`T Korten (HZDR/Helmholtz AI)`

```{rubric} Performance
```
Expand All @@ -12,6 +13,7 @@
* updates `pp.highly_variable_genes` for the flavors `seurat` and `cell_ranger` {pr}`183` {smaller}`S Dicks`
* `pp.scale` now uses `RawKernels` for dense and csr {pr}`185` {smaller}`S Dicks`


```{rubric} Bug fixes
```

Expand Down