-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add images with VS Code Remote Dev Setup (#4)
* add sleap-vscode-connect * use sleap-cuda base image * update VNC dockerfile without redundant layers from base image * make container with VScode tunnel * add workflows for vscode tunnel containers * need curl and tar * need vncserver tools * make devcontainer for VScode tunnel * modify build context * need devcontainer files remotely * reorganize remote files * try hardcoding these X server values * can't copy temp files * GUI display will not work without mount * add link to docker repo * update readme * add devcontainer * update readme * update README and devcontainer * remote devcontainer at base of repo * add forward port to enable graphical applications in VS code * generalize docs * redo main readme * add devcontainer * fix readme for repo * update container readmes
- Loading branch information
Showing
11 changed files
with
584 additions
and
153 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| name: Build and Push sleap-vscode-tunnel (Production Workflow) | ||
|
|
||
| # Run on push to main branch after testing is complete | ||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - sleap_vscode_tunnel/** # Only run on changes to sleap_vscode_tunnel | ||
| - .github/workflows/sleap_vscode_tunnel_production.yml # Only run on changes to this workflow | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest # Only build on Ubuntu for now since Docker is not available on macOS runners | ||
| strategy: | ||
| matrix: | ||
| platform: [linux/amd64] # Only build amd64 for now | ||
| max-parallel: 2 # Build both architectures in parallel (if more than one) | ||
| outputs: | ||
| git_sha: ${{ steps.get_sha.outputs.sha }} | ||
| sanitized_platform: ${{ steps.sanitize_platform.outputs.sanitized_platform }} | ||
| steps: | ||
| - name: Checkout code | ||
| # https://github.com/actions/checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Get Git SHA | ||
| id: get_sha | ||
| run: echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Debug Git SHA | ||
| run: echo "Git SHA ${{ steps.get_sha.outputs.sha }}" | ||
|
|
||
| # Generate a sanitized platform string with slashes replaced by dashes | ||
| - name: Sanitize platform name | ||
| id: sanitize_platform | ||
| run: | | ||
| sanitized_platform="${{ matrix.platform }}" # Copy platform value | ||
| sanitized_platform="${sanitized_platform/\//-}" # Replace / with - | ||
| echo "sanitized_platform=$sanitized_platform" >> $GITHUB_OUTPUT | ||
| - name: Set up Docker Buildx | ||
| # https://github.com/docker/setup-buildx-action | ||
| uses: docker/setup-buildx-action@v3 | ||
| with: | ||
| driver: docker-container # Use a container driver for Buildx (default) | ||
|
|
||
| - name: Log in to Docker Hub | ||
| # https://github.com/docker/login-action | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ vars.DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.DOCKERHUB_TOKEN }} | ||
|
|
||
| - name: Build and push Docker image | ||
| # https://github.com/docker/build-push-action | ||
| uses: docker/build-push-action@v6 | ||
| with: | ||
| context: ./sleap_vscode_tunnel # Build context wrt the root of the repository | ||
| file: ./sleap_vscode_tunnel/Dockerfile # Path to Dockerfile wrt the root of the repository | ||
| platforms: ${{ matrix.platform }} | ||
| push: true # Push the image to Docker Hub | ||
| # Tags for the production images, including the "latest" tag | ||
| tags: | | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:latest | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }} | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-nvidia-cuda-11.3.1-cudnn8-runtime-ubuntu20.04 | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-sleap-1.3.4 | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-${{ steps.get_sha.outputs.sha }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| name: Build and Push sleap-vscode-tunnel (Test Workflow) | ||
|
|
||
| # Run on push to branches other than main for sleap_vscode_tunnel | ||
| on: | ||
| push: | ||
| branches-ignore: | ||
| - main | ||
| paths: | ||
| - sleap_vscode_tunnel/** # Only run on changes to sleap_vscode_tunnel | ||
| - .github/workflows/sleap_vscode_tunnel_test.yml # Only run on changes to this workflow | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest # Only build on Ubuntu for now since Docker is not available on macOS runners | ||
| strategy: | ||
| matrix: | ||
| platform: [linux/amd64] # Only build amd64 for now | ||
| max-parallel: 2 # Build both architectures in parallel (if more than one) | ||
| outputs: | ||
| git_sha: ${{ steps.get_sha.outputs.sha }} | ||
| sanitized_platform: ${{ steps.sanitize_platform.outputs.sanitized_platform }} | ||
| steps: | ||
| - name: Checkout code | ||
| # https://github.com/actions/checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Get Git SHA | ||
| id: get_sha | ||
| run: echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Debug Git SHA | ||
| run: echo "Git SHA ${{ steps.get_sha.outputs.sha }}" | ||
|
|
||
| # Generate a sanitized platform string with slashes replaced by dashes | ||
| - name: Sanitize platform name | ||
| id: sanitize_platform | ||
| run: | | ||
| sanitized_platform="${{ matrix.platform }}" # Copy platform value | ||
| sanitized_platform="${sanitized_platform/\//-}" # Replace / with - | ||
| echo "sanitized_platform=$sanitized_platform" >> $GITHUB_OUTPUT | ||
| - name: Set up Docker Buildx | ||
| # https://github.com/docker/setup-buildx-action | ||
| uses: docker/setup-buildx-action@v3 | ||
| with: | ||
| driver: docker-container # Use a container driver for Buildx (default) | ||
|
|
||
| - name: Log in to Docker Hub | ||
| # https://github.com/docker/login-action | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ vars.DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.DOCKERHUB_TOKEN }} | ||
|
|
||
| - name: Build and push Docker image | ||
| # https://github.com/docker/build-push-action | ||
| uses: docker/build-push-action@v6 | ||
| with: | ||
| context: ./sleap_vscode_tunnel # Build context wrt the root of the repository | ||
| file: ./sleap_vscode_tunnel/Dockerfile # Path to Dockerfile wrt the root of the repository | ||
| platforms: ${{ matrix.platform }} | ||
| push: true # Push the image to Docker Hub | ||
| # Tags all include "-test" to differentiate from production images | ||
| tags: | | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-test | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-nvidia-cuda-11.3.1-cudnn8-runtime-ubuntu20.04-test | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-sleap-1.3.4-test | ||
| ${{ vars.DOCKERHUB_USERNAME }}/sleap-vscode-tunnel:${{ steps.sanitize_platform.outputs.sanitized_platform }}-${{ steps.get_sha.outputs.sha }}-test |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,111 +1,113 @@ | ||
| # sleap-cuda-container | ||
|
|
||
|
|
||
| ## Description | ||
| This repo contains a DockerFile for a lightweight container (~2.79 GB) with the PyPI installation of SLEAP and all of its dependencies. The container repository is located at [https://hub.docker.com/repository/docker/eberrigan/sleap-cuda/general](https://hub.docker.com/repository/docker/eberrigan/sleap-cuda/general). | ||
|
|
||
| The base image used is [nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu20.04](https://hub.docker.com/layers/nvidia/cuda/11.3.1-cudnn8-runtime-ubuntu20.04/images/sha256-025a321d3131b688f4ac09d80e9af6221f2d1568b4f9ea6e45a698beebb439c0). | ||
| - The Dockerfile is located at `docker/Dockerfile`. | ||
| - The repo has CI set up in `.github/workflows` for building and pushing the image when making changes. | ||
| - The workflow uses the linux/amd64 platform to build. | ||
| - `.devcontainer/devcontainer.json` is convenient for developing inside a container made with the DockerFile using Visual Studio Code. | ||
| - Test data for training is located in `tests/data`. | ||
|
|
||
|
|
||
| ## Installation | ||
|
|
||
| **Make sure to have Docker Daemon running first** | ||
|
|
||
|
|
||
| You can pull the image if you don't have it built locally, or need to update the latest, with | ||
|
|
||
| ``` | ||
| # SLEAP CUDA Container Repository | ||
|
|
||
| This repository provides multiple Docker configurations for running **SLEAP** (a machine learning-based tool for animal pose estimation) in different environments, leveraging GPU support and Ubuntu. These configurations enable different ways to interact with the containerized SLEAP environment, including VNC and VS Code Tunnels. | ||
|
|
||
| --- | ||
|
|
||
| ## Repository Structure | ||
|
|
||
| The repository is organized as follows: | ||
|
|
||
| ``` | ||
| sleap-cuda-container (repo) | ||
| │ | ||
| ├── .github | ||
| ├── sleap_cuda | ||
| │ ├── README.md | ||
| │ ├── Dockerfile | ||
| │ ├── .dockerignore | ||
| │ ├── .devcontainer | ||
| │ | ||
| ├── sleap_vnc_connect | ||
| │ ├── README.md | ||
| │ ├── Dockerfile | ||
| │ ├── .dockerignore | ||
| │ ├── .devcontainer | ||
| │ | ||
| ├── sleap_vscode_tunnel | ||
| │ ├── README.md | ||
| │ ├── Dockerfile | ||
| │ ├── .dockerignore | ||
| │ ├── .devcontainer | ||
| │ | ||
| ├── tests | ||
| ├── README.md | ||
| ├── .gitignore | ||
| └── LICENSE | ||
| ``` | ||
|
|
||
| ### Subfolders and Containers | ||
|
|
||
| - **`sleap_cuda/`**: | ||
| - Contains the base image for SLEAP with GPU support and Ubuntu. | ||
| - This image is the foundation for other container configurations. | ||
| - **[Docker Hub Link](https://hub.docker.com/repository/docker/eberrigan/sleap-cuda/general)**. | ||
|
|
||
| - **`sleap_vnc_connect/`**: | ||
| - Adds a **VNC server** to the base image, allowing you to connect to the container using a local VNC client. | ||
| - Useful for graphical interaction with the SLEAP GUI. | ||
| - See the `README.md` in this folder for setup and connection instructions. | ||
|
|
||
| - **`sleap_vscode_tunnel/`**: | ||
| - Adds the **VS Code CLI** for establishing a remote tunnel connection to the container. | ||
| - Enables headless interaction with the container through VS Code. | ||
| - Note: **VS Code Tunnels do not yet support port forwarding**, so graphical applications (like the SLEAP GUI) cannot be used with this container. | ||
|
|
||
| - **`tests/`**: | ||
| - Contains test scripts for validating the functionality of the Docker configurations. | ||
|
|
||
| --- | ||
|
|
||
| ## Running the Containers | ||
|
|
||
| ### 1. **Base SLEAP CUDA Container** | ||
| To use the base container: | ||
| ```bash | ||
| docker pull eberrigan/sleap-cuda:latest | ||
| ``` | ||
|
|
||
| ## Usage | ||
| ### 2. **SLEAP VNC Connect** | ||
| The `sleap_vnc_connect` container starts a **VNC server**, enabling a graphical interface for SLEAP. | ||
|
|
||
| Then, to run the image with gpus interactively: | ||
| 1. **Build and Run**: | ||
| ```bash | ||
| docker build -t sleap-vnc ./sleap_vnc_connect | ||
| docker run --rm -it -p 5901:5901 --gpus all sleap-vnc | ||
| ``` | ||
|
|
||
| ``` | ||
| docker run --gpus all -it eberrigan/sleap-cuda:latest bash | ||
| ``` | ||
| 2. **Connect to the VNC Server**: | ||
| - Use a VNC client and connect to `localhost:5901`. | ||
| - Ensure your display environment variables are correctly set. | ||
|
|
||
| and test with | ||
| ### 3. **SLEAP VS Code Tunnel** | ||
| The `sleap_vscode_tunnel` container allows **remote development** using VS Code tunnels. | ||
|
|
||
| ``` | ||
| python -c "import sleap; sleap.versions()" && nvidia-smi | ||
| ``` | ||
| 1. **Build and Run**: | ||
| ```bash | ||
| docker build -t sleap-tunnel ./sleap_vscode_tunnel | ||
| docker run --rm -it --gpus all sleap-tunnel | ||
| ``` | ||
|
|
||
| In general, use the syntax | ||
| 2. **Connect to the Tunnel**: | ||
| - The VS Code CLI in the container starts a tunnel. | ||
| - Use the provided tunnel URL in VS Code to connect and work remotely. | ||
|
|
||
| ``` | ||
| docker run -v /path/on/host:/path/in/container [other options] image_name [command] | ||
| ``` | ||
| --- | ||
|
|
||
| Note that host paths are absolute. | ||
| ## Notes and Limitations | ||
|
|
||
| - **Remote Tunnels**: | ||
| - VS Code Remote Tunnels do not currently support port forwarding. As a result, graphical applications like the SLEAP GUI cannot run in this container. | ||
|
|
||
| Use this syntax to give host permissions to mounted volumes | ||
| ``` | ||
| docker run -u $(id -u):$(id -g) -v /your/host/directory:/container/directory [options] your-image-name [command] | ||
| ``` | ||
| - **VNC**: | ||
| - The VNC Connect container requires port `5901` to be exposed. Use the `-p` flag to map this port when running the container. | ||
|
|
||
| ``` | ||
| docker run -u $(id -u):$(id -g) -v ./tests/data:/workspace/tests/data --gpus all -it eberrigan/sleap-cuda:latest bash | ||
| ``` | ||
| - **X11 Forwarding**: | ||
| - If graphical interaction is required over an SSH connection, ensure proper X11 forwarding is set up. | ||
|
|
||
| Test: | ||
| --- | ||
|
|
||
| ``` | ||
| python3 -c "import sleap; print('SLEAP version:', sleap.__version__)" | ||
| nvidia-smi # Check that the GPUs are disvoerable | ||
| ps aux | grep Xtightvnc # Check that the VNC server is running | ||
| sleap-train "tests/data/initial_config.json" "tests/data/dance.mp4.labels.slp" --video-paths "tests/data/dance.mp4" | ||
| ``` | ||
|
|
||
| **Notes:** | ||
|
|
||
| - The `eberrigan/sleap-cuda` is the Docker registry where the images are pulled from. This is only used when pulling images from the cloud, and not necesary when building/running locally. | ||
| - `-it` ensures that you get an interactive terminal. The `i` stands for interactive, and `t` allocates a pseudo-TTY, which is what allows you to interact with the bash shell inside the container. | ||
| - The `-v` or `--volume` option mounts the specified directory with the same level of access as the directory has on the host. | ||
| - `bash` is the command that gets executed inside the container, which in this case is to start the bash shell. | ||
| - Order of operations is 1. Pull (if needed): Get a pre-built image from a registry. 2. Run: Start a container from an image. | ||
|
|
||
| ## Contributing | ||
|
|
||
| - Use the `devcontainer.json` to open the repo in a dev container using VS Code. | ||
| - There is some test data in the `tests` directory that will be automatically mounted for use since the working directory is the workspace. | ||
| - Rebuild the container when you make changes using `Dev Container: Rebuild Container`. | ||
|
|
||
| - Please make a new branch, starting with your name, with any changes, and request a reviewer before merging with the main branch since this image will be used by others. | ||
| - Please document using the same conventions (docstrings for each function and class, typing-hints, informative comments). | ||
| - Tests are written in the pytest framework. Data used in the tests are defined as fixtures in `tests/fixtures/data.py` (https://docs.pytest.org/en/6.2.x/reference.html#fixtures-api). | ||
|
|
||
|
|
||
| ## Build | ||
| To build and push via automated CI, just push changes to a branch. | ||
| - Pushes to `main` result in an image with the tag `latest`. | ||
| - Pushes to other branches have tags with `-test` appended. | ||
| - See `.github/workflows` for testing and production workflows. | ||
|
|
||
| To test `test` images locally use after pushing the `test` images via CI: | ||
|
|
||
| ``` | ||
| docker pull eberrigan/sleap-cuda:linux-amd64-test | ||
| ``` | ||
|
|
||
| then | ||
|
|
||
| ``` | ||
| docker run -v ./tests/data:/workspace/tests/data --gpus all -it eberrigan/sleap-cuda:linux-amd64-test bash | ||
| ``` | ||
|
|
||
| To build locally for testing you can use the command: | ||
|
|
||
| ``` | ||
| docker build --platform linux/amd64 ./docker | ||
| ``` | ||
| ## License | ||
| This repository is licensed under the **MIT License**. See the `LICENSE` file for more details. | ||
|
|
||
| ## Support | ||
| contact Elizabeth at [email protected] | ||
| For additional questions or issues, please open an issue in this repository. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.