RHOAIENG-10783: fix(rocm): de-vendor the bundled rocm libraries from pytorch #652

jiridanek · 2024-07-31T15:06:07Z

RHOAIENG-10783 Optimize ROCm images to reduce the size so they can be built on OpenShift CI

Use the script from instructlab to remove the duplicate copy of rocm libs from the image. This will make the image significantly smaller.

Script lives at https://github.com/tiran/instructlab-containers/blob/main/containers/rocm/de-vendor-torch.sh

Description

This gives me

rocm-jupyter-pytorch-ubi9-python-3.9-2024a_20240731               ab8eaa47fff8  34 seconds ago  35.7 GB

While on master we currently have

rocm-jupyter-pytorch-ubi9-python-3.9-main_37df13c29fdde3fa4f8e455d30b5bf39d80f6dfb      9a4a772dbee7  4 days ago   46.4 GB

This way the torch image size is in line with tensorflow image size

rocm-jupyter-tensorflow-ubi9-python-3.9-main_37df13c29fdde3fa4f8e455d30b5bf39d80f6dfb   465069df6467  4 days ago   35.5 GB

In conclusion, 10 GB was saved.

How Has This Been Tested?

CONTAINER_BUILD_CACHE_ARGS="--platform=linux/amd64  --cache-from ghcr.io/jiridanek/notebooks/workbench-images/build-cache" make rocm-jupyter-pytorch-ubi9-python-3.9

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

openshift-ci · 2024-07-31T15:06:12Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

caponetto · 2024-07-31T16:02:36Z

Awesome!

/lgtm

jupyter/rocm/pytorch/ubi9-python-3.9/de-vendor-torch.sh

jstourac · 2024-07-31T16:09:22Z

/lgtm

Nicely done! 🙂

Update: per the comments from the RHEL AI member on the slack, this script hasn't been tested ever... so I suppose we truly need to assure that we don't break anything with this!

jiridanek · 2024-07-31T16:18:40Z

Lets see if prow is able to build this without running of of ephemeral-storage; if I can't get a build, I can't test it.

harshad16 · 2024-07-31T20:10:35Z

/hold

awesome work in tracking and implementing this.

Kept this on hold,
please some-one test this image in a cluster, before unholding this PR.
quay.io/opendatahub/workbench-images:rocm-jupyter-pytorch-ubi9-python-3.9-pr-652

caponetto · 2024-08-01T09:49:06Z

Can we do the same thing for rocm-runtime-pytorch-ubi9-python-3_9?

…ytorch Use the script from instructlab to remove the duplicate copy of rocm libs from the image. This will make the image significantly smaller. Script lives at https://github.com/tiran/instructlab-containers/blob/main/containers/rocm/de-vendor-torch.sh

jiridanek · 2024-08-02T10:41:20Z

[jdanek@nvd-srv-05 ~]$ podman run --entrypoint /bin/bash --device=/dev/kfd --device=/dev/dri --ipc=host  --rm -it ghcr.io/jiridanek/notebooks/workbench-images:rocm-runtime-pytorch-ubi9-python-3.9-jd_devendor_aaacb974de5617e89cb945892d3801c831b37a44
(app-root) bash-5.1$ python
Python 3.9.18 (main, Jul  3 2024, 00:00:00) 
[GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.get_device_name(0)
'AMD Instinct MI210'
>>> torch.rand(10).device
device(type='cpu')
>>> torch.set_default_tensor_type(torch.cuda.FloatTensor)
/opt/app-root/lib64/python3.9/site-packages/torch/__init__.py:955: UserWarning: torch.set_default_tensor_type() is deprecated as of PyTorch 2.1, please use torch.set_default_dtype() and torch.set_default_device() as alternatives. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:432.)
  _C._set_default_tensor_type(t)
>>> torch.rand(10).device
device(type='cuda', index=0)

(app-root) bash-5.1$ git clone https://github.com/pytorch/examples.git
Cloning into 'examples'...
remote: Enumerating objects: 4296, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 4296 (delta 2), reused 6 (delta 0), pack-reused 4283
Receiving objects: 100% (4296/4296), 41.36 MiB | 25.79 MiB/s, done.
Resolving deltas: 100% (2149/2149), done.
(app-root) bash-5.1$ cd examples/mnist
(app-root) bash-5.1$ python main.py 
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz

...

Train Epoch: 14 [58240/60000 (97%)]	Loss: 0.035374
Train Epoch: 14 [58880/60000 (98%)]	Loss: 0.003282
Train Epoch: 14 [59520/60000 (99%)]	Loss: 0.001743

Test set: Average loss: 0.0274, Accuracy: 9915/10000 (99%)

and it is the same thing on

ghcr.io/jiridanek/notebooks/workbench-images:rocm-jupyter-pytorch-ubi9-python-3.9-jd_devendor_aaacb974de5617e89cb945892d3801c831b37a44

/lgtm

openshift-ci · 2024-08-02T10:41:22Z

@jiridanek: you cannot LGTM your own PR.

In response to this:

[jdanek@nvd-srv-05 ~]$ podman run --entrypoint /bin/bash --device=/dev/kfd --device=/dev/dri --ipc=host  --rm -it ghcr.io/jiridanek/notebooks/workbench-images:rocm-runtime-pytorch-ubi9-python-3.9-jd_devendor_aaacb974de5617e89cb945892d3801c831b37a44
(app-root) bash-5.1$ python
Python 3.9.18 (main, Jul  3 2024, 00:00:00) 
[GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.get_device_name(0)
'AMD Instinct MI210'
>>> torch.rand(10).device
device(type='cpu')
>>> torch.set_default_tensor_type(torch.cuda.FloatTensor)
/opt/app-root/lib64/python3.9/site-packages/torch/__init__.py:955: UserWarning: torch.set_default_tensor_type() is deprecated as of PyTorch 2.1, please use torch.set_default_dtype() and torch.set_default_device() as alternatives. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:432.)
 _C._set_default_tensor_type(t)
>>> torch.rand(10).device
device(type='cuda', index=0)

(app-root) bash-5.1$ git clone https://github.com/pytorch/examples.git
Cloning into 'examples'...
remote: Enumerating objects: 4296, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 4296 (delta 2), reused 6 (delta 0), pack-reused 4283
Receiving objects: 100% (4296/4296), 41.36 MiB | 25.79 MiB/s, done.
Resolving deltas: 100% (2149/2149), done.
(app-root) bash-5.1$ cd examples/mnist
(app-root) bash-5.1$ python main.py 
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz

...

Train Epoch: 14 [58240/60000 (97%)]	Loss: 0.035374
Train Epoch: 14 [58880/60000 (98%)]	Loss: 0.003282
Train Epoch: 14 [59520/60000 (99%)]	Loss: 0.001743

Test set: Average loss: 0.0274, Accuracy: 9915/10000 (99%)

and it is the same thing on

ghcr.io/jiridanek/notebooks/workbench-images:rocm-jupyter-pytorch-ubi9-python-3.9-jd_devendor_aaacb974de5617e89cb945892d3801c831b37a44

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-08-05T07:47:56Z

/unhold
I checked this works fine

caponetto · 2024-08-05T17:27:33Z

/lgtm

jstourac · 2024-08-06T08:59:15Z

/lgtm

jiridanek · 2024-08-06T09:11:07Z

/approved
💯

/override ci/prow/runtime-rocm-pytorch-ubi9-python-3-9-pr-image-mirror
/override ci/prow/rocm-runtimes-ubi9-e2e-tests
/override ci/prow/rocm-notebooks-e2e-tests
/override
ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror
/override ci/prow/images

build failed on openshift-ci, this is known issue

error: build error: committing container for step {Env:[STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i APP_ROOT=/opt/app-root HOME=/opt/app-root/src PLATFORM=el9 NODEJS_VER=20 PYTHON_VERSION=3.9 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/app-root/src/.local/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PYTHONUNBUFFERED=1 PYTHONIOENCODING=UTF-8 LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 PIP_NO_CACHE_DIR=off NAME=python3 VERSION=0 ARCH=x86_64 SUMMARY=Platform for building and running Python 3.9 applications DESCRIPTION=Python 3.9 available as container is a base platform for building and running various Python 3.9 applications and frameworks. Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python's elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms. BASH_ENV=/opt/app-root/bin/activate ENV=/opt/app-root/bin/activate PROMPT_COMMAND=. /opt/app-root/bin/activate BUILD_LOGLEVEL=0 OPENSHIFT_BUILD_NAME=cuda-c9s-python-3.9-amd64 OPENSHIFT_BUILD_NAMESPACE=ci-op-ckwlrcfk NVARCH=x86_64 NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 NV_CUDA_CUDART_VERSION=12.1.105-1 CUDA_VERSION=12.1.1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility NV_CUDA_LIB_VERSION=12.1.1-1 NV_NVTX_VERSION=12.1.105-1 NV_LIBNPP_VERSION=12.1.0.40-1 NV_LIBNPP_PACKAGE=libnpp-12-1-12.1.0.40-1 NV_LIBCUBLAS_VERSION=12.1.3.1-1 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 NV_LIBNCCL_VERSION=2.17.1 NCCL_VERSION=2.17.1 NV_LIBNCCL_PACKAGE=libnccl-2.17.1-1+cuda12.1 XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda NV_NVPROF_VERSION=12.1.105-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1-12.1.105-1 NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 NV_NVML_DEV_VERSION=12.1.105-1 NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 NV_LIBNPP_DEV_VERSION=12.1.0.40-1 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-1-12.1.0.40-1 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.17.1-1+cuda12.1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1-12.1.1-1 LIBRARY_PATH=/usr/local/cuda/lib64/stubs NV_CUDNN_VERSION=8.9.0.131-1 NV_CUDNN_PACKAGE=libcudnn8-8.9.0.131-1.cuda12.1 NV_CUDNN_PACKAGE_DEV=libcudnn8-devel-8.9.0.131-1.cuda12.1] Command:label Args:[io.openshift.build.name cuda-c9s-python-3.9-amd64 io.openshift.build.namespace ci-op-ckwlrcfk io.openshift.build.commit.author  io.openshift.build.commit.date  io.openshift.build.commit.id  io.openshift.build.commit.message  io.openshift.build.commit.ref  io.openshift.build.name  io.openshift.build.namespace  io.openshift.build.source-context-dir  io.openshift.build.source-location  io.openshift.ci.from.base-c9s-python-3.9 sha256:04a2c872bf092df215ed42cbf9ae55bc5d870aab67677b7df2cf40ca0b7f3174 vcs-ref  vcs-type  vcs-url ] Flags:[] Attrs:map[] Message:LABEL "io.openshift.build.name" "cuda-c9s-python-3.9-amd64" "io.openshift.build.namespace" "ci-op-ckwlrcfk" "io.openshift.build.commit.author" "" "io.openshift.build.commit.date" "" "io.openshift.build.commit.id" "" "io.openshift.build.commit.message" "" "io.openshift.build.commit.ref" "" "io.openshift.build.name" "" "io.openshift.build.namespace" "" "io.openshift.build.source-context-dir" "" "io.openshift.build.source-location" "" "io.openshift.ci.from.base-c9s-python-3.9" "sha256:04a2c872bf092df215ed42cbf9ae55bc5d870aab67677b7df2cf40ca0b7f3174" "vcs-ref" "" "vcs-type" "" "vcs-url" "" Heredocs:[] Original:LABEL "io.openshift.build.name"="cuda-c9s-python-3.9-amd64" "io.openshift.build.namespace"="ci-op-ckwlrcfk" "io.openshift.build.commit.author"="" "io.openshift.build.commit.date"="" "io.openshift.build.commit.id"="" "io.openshift.build.commit.message"="" "io.openshift.build.commit.ref"="" "io.openshift.build.name"="" "io.openshift.build.namespace"="" "io.openshift.build.source-context-dir"="" "io.openshift.build.source-location"="" "io.openshift.ci.from.base-c9s-python-3.9"="sha256:04a2c872bf092df215ed42cbf9ae55bc5d870aab67677b7df2cf40ca0b7f3174" "vcs-ref"="" "vcs-type"="" "vcs-url"=""}: copying layers and metadata for container "b8399a96772b12e3e3dd6074228e2dd8747d1807d0a202f8cf199c1ddbc4a5f2": initializing source containers-storage:image-registry.openshift-image-registry.svc-working-container-1: error creating new image source "containers-storage:image-registry.openshift-image-registry.svc-working-container-1": storing layer "42d894d1519f04a040d38a1885e6317c4654536d45d65cf2a41ba51649298a96" to file: io: read/write on closed pipe

openshift-ci · 2024-08-06T09:11:16Z

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/images, ci/prow/rocm-notebooks-e2e-tests, ci/prow/rocm-runtimes-ubi9-e2e-tests, ci/prow/runtime-rocm-pytorch-ubi9-python-3-9-pr-image-mirror

In response to this:

/approved
💯

/override ci/prow/runtime-rocm-pytorch-ubi9-python-3-9-pr-image-mirror
/override ci/prow/rocm-runtimes-ubi9-e2e-tests
/override ci/prow/rocm-notebooks-e2e-tests
/override
ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror
/override ci/prow/images

build failed on openshift-ci, this is known issue

error: build error: committing container for step {Env:[STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i APP_ROOT=/opt/app-root HOME=/opt/app-root/src PLATFORM=el9 NODEJS_VER=20 PYTHON_VERSION=3.9 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/app-root/src/.local/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PYTHONUNBUFFERED=1 PYTHONIOENCODING=UTF-8 LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 PIP_NO_CACHE_DIR=off NAME=python3 VERSION=0 ARCH=x86_64 SUMMARY=Platform for building and running Python 3.9 applications DESCRIPTION=Python 3.9 available as container is a base platform for building and running various Python 3.9 applications and frameworks. Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python's elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms. BASH_ENV=/opt/app-root/bin/activate ENV=/opt/app-root/bin/activate PROMPT_COMMAND=. /opt/app-root/bin/activate BUILD_LOGLEVEL=0 OPENSHIFT_BUILD_NAME=cuda-c9s-python-3.9-amd64 OPENSHIFT_BUILD_NAMESPACE=ci-op-ckwlrcfk NVARCH=x86_64 NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 NV_CUDA_CUDART_VERSION=12.1.105-1 CUDA_VERSION=12.1.1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility NV_CUDA_LIB_VERSION=12.1.1-1 NV_NVTX_VERSION=12.1.105-1 NV_LIBNPP_VERSION=12.1.0.40-1 NV_LIBNPP_PACKAGE=libnpp-12-1-12.1.0.40-1 NV_LIBCUBLAS_VERSION=12.1.3.1-1 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 NV_LIBNCCL_VERSION=2.17.1 NCCL_VERSION=2.17.1 NV_LIBNCCL_PACKAGE=libnccl-2.17.1-1+cuda12.1 XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda NV_NVPROF_VERSION=12.1.105-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1-12.1.105-1 NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 NV_NVML_DEV_VERSION=12.1.105-1 NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 NV_LIBNPP_DEV_VERSION=12.1.0.40-1 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-1-12.1.0.40-1 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.17.1-1+cuda12.1 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1-12.1.1-1 LIBRARY_PATH=/usr/local/cuda/lib64/stubs NV_CUDNN_VERSION=8.9.0.131-1 NV_CUDNN_PACKAGE=libcudnn8-8.9.0.131-1.cuda12.1 NV_CUDNN_PACKAGE_DEV=libcudnn8-devel-8.9.0.131-1.cuda12.1] Command:label Args:[io.openshift.build.name cuda-c9s-python-3.9-amd64 io.openshift.build.namespace ci-op-ckwlrcfk io.openshift.build.commit.author  io.openshift.build.commit.date  io.openshift.build.commit.id  io.openshift.build.commit.message  io.openshift.build.commit.ref  io.openshift.build.name  io.openshift.build.namespace  io.openshift.build.source-context-dir  io.openshift.build.source-location  io.openshift.ci.from.base-c9s-python-3.9 sha256:04a2c872bf092df215ed42cbf9ae55bc5d870aab67677b7df2cf40ca0b7f3174 vcs-ref  vcs-type  vcs-url ] Flags:[] Attrs:map[] Message:LABEL "io.openshift.build.name" "cuda-c9s-python-3.9-amd64" "io.openshift.build.namespace" "ci-op-ckwlrcfk" "io.openshift.build.commit.author" "" "io.openshift.build.commit.date" "" "io.openshift.build.commit.id" "" "io.openshift.build.commit.message" "" "io.openshift.build.commit.ref" "" "io.openshift.build.name" "" "io.openshift.build.namespace" "" "io.openshift.build.source-context-dir" "" "io.openshift.build.source-location" "" "io.openshift.ci.from.base-c9s-python-3.9" "sha256:04a2c872bf092df215ed42cbf9ae55bc5d870aab67677b7df2cf40ca0b7f3174" "vcs-ref" "" "vcs-type" "" "vcs-url" "" Heredocs:[] Original:LABEL "io.openshift.build.name"="cuda-c9s-python-3.9-amd64" "io.openshift.build.namespace"="ci-op-ckwlrcfk" "io.openshift.build.commit.author"="" "io.openshift.build.commit.date"="" "io.openshift.build.commit.id"="" "io.openshift.build.commit.message"="" "io.openshift.build.commit.ref"="" "io.openshift.build.name"="" "io.openshift.build.namespace"="" "io.openshift.build.source-context-dir"="" "io.openshift.build.source-location"="" "io.openshift.ci.from.base-c9s-python-3.9"="sha256:04a2c872bf092df215ed42cbf9ae55bc5d870aab67677b7df2cf40ca0b7f3174" "vcs-ref"="" "vcs-type"="" "vcs-url"=""}: copying layers and metadata for container "b8399a96772b12e3e3dd6074228e2dd8747d1807d0a202f8cf199c1ddbc4a5f2": initializing source containers-storage:image-registry.openshift-image-registry.svc-working-container-1: error creating new image source "containers-storage:image-registry.openshift-image-registry.svc-working-container-1": storing layer "42d894d1519f04a040d38a1885e6317c4654536d45d65cf2a41ba51649298a96" to file: io: read/write on closed pipe

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-08-06T09:11:43Z

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror

openshift-ci · 2024-08-06T09:11:49Z

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror

In response to this:

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2024-08-06T09:12:07Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Add images based on python 3.11 * Apply #656 to Python 3.11 images * Fix expected TF vesion on the test file * Fix labels for Python 3.11 * Apply #652 to Python 3.11 images * Update lock to fix debugpy package version * Apply #635 to Python 3.11 images * Replace 3-9 -> 3-11 leftovers * Fix runtime rocm image name according to openshift/release * Apply #667 to Python 3.11 images * Adapt test code for Python 3.11 images

openshift-ci bot added the do-not-merge/work-in-progress label Jul 31, 2024

jiridanek marked this pull request as ready for review July 31, 2024 15:48

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 31, 2024

openshift-ci bot requested review from caponetto and harshad16 July 31, 2024 15:49

jiridanek requested review from jstourac and removed request for caponetto July 31, 2024 15:49

jiridanek added the tide/merge-method-squash label Jul 31, 2024

openshift-ci bot assigned caponetto Jul 31, 2024

openshift-ci bot added the lgtm label Jul 31, 2024

jstourac reviewed Jul 31, 2024

View reviewed changes

jupyter/rocm/pytorch/ubi9-python-3.9/de-vendor-torch.sh Show resolved Hide resolved

jiridanek mentioned this pull request Jul 31, 2024

RHOAIENG-10783: fix(rocm): remove more files that instructlab also removes #653

Closed

3 tasks

openshift-ci bot assigned jstourac Jul 31, 2024

openshift-ci bot added the do-not-merge/hold label Jul 31, 2024

jstourac mentioned this pull request Aug 1, 2024

Update the ROCm versions installed from 6.0 to 6.1 to make it in sync with what we use in Dockerfile #656

Merged

3 tasks

jiridanek added 2 commits August 2, 2024 10:55

add a selftest

aaacb97

jiridanek force-pushed the jd_devendor branch from daab953 to aaacb97 Compare August 2, 2024 08:56

openshift-ci bot removed the lgtm label Aug 2, 2024

jiridanek changed the title ~~RHOAIENG-9853: fix(rocm): de-vendor the bundled rocm libraries from pytorch~~ RHOAIENG-10783: fix(rocm): de-vendor the bundled rocm libraries from pytorch Aug 2, 2024

openshift-ci bot removed the do-not-merge/hold label Aug 5, 2024

openshift-ci bot added the lgtm label Aug 5, 2024

jiridanek added the approved label Aug 6, 2024

openshift-merge-bot bot merged commit ec45de1 into opendatahub-io:main Aug 6, 2024
12 checks passed

caponetto added a commit to caponetto/opendatahub-io-notebooks that referenced this pull request Aug 6, 2024

Apply opendatahub-io#652 to Python 3.11 images

f303351

caponetto added a commit to caponetto/opendatahub-io-notebooks that referenced this pull request Aug 6, 2024

Apply opendatahub-io#652 to Python 3.11 images

c95d164

caponetto mentioned this pull request Aug 7, 2024

RHOAIENG-10783: fix(Dockerfiles): combine fix-permissions RUN with the preceding RUN when possible #635

Merged

3 tasks

caponetto added a commit to caponetto/opendatahub-io-notebooks that referenced this pull request Aug 7, 2024

Apply opendatahub-io#652 to Python 3.11 images

507e486

caponetto added a commit to caponetto/opendatahub-io-notebooks that referenced this pull request Aug 8, 2024

Apply opendatahub-io#652 to Python 3.11 images

88a8440

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RHOAIENG-10783: fix(rocm): de-vendor the bundled rocm libraries from pytorch #652

RHOAIENG-10783: fix(rocm): de-vendor the bundled rocm libraries from pytorch #652

jiridanek commented Jul 31, 2024 •

edited

Loading

openshift-ci bot commented Jul 31, 2024

caponetto commented Jul 31, 2024

jstourac commented Jul 31, 2024 •

edited

Loading

jiridanek commented Jul 31, 2024

harshad16 commented Jul 31, 2024 •

edited

Loading

caponetto commented Aug 1, 2024

jiridanek commented Aug 2, 2024

openshift-ci bot commented Aug 2, 2024

jiridanek commented Aug 5, 2024

caponetto commented Aug 5, 2024

jstourac commented Aug 6, 2024

jiridanek commented Aug 6, 2024

openshift-ci bot commented Aug 6, 2024

jiridanek commented Aug 6, 2024

openshift-ci bot commented Aug 6, 2024

openshift-ci bot commented Aug 6, 2024

RHOAIENG-10783: fix(rocm): de-vendor the bundled rocm libraries from pytorch #652

RHOAIENG-10783: fix(rocm): de-vendor the bundled rocm libraries from pytorch #652

Conversation

jiridanek commented Jul 31, 2024 • edited Loading

Description

How Has This Been Tested?

Merge criteria:

openshift-ci bot commented Jul 31, 2024

caponetto commented Jul 31, 2024

jstourac commented Jul 31, 2024 • edited Loading

jiridanek commented Jul 31, 2024

harshad16 commented Jul 31, 2024 • edited Loading

caponetto commented Aug 1, 2024

jiridanek commented Aug 2, 2024

openshift-ci bot commented Aug 2, 2024

jiridanek commented Aug 5, 2024

caponetto commented Aug 5, 2024

jstourac commented Aug 6, 2024

jiridanek commented Aug 6, 2024

openshift-ci bot commented Aug 6, 2024

jiridanek commented Aug 6, 2024

openshift-ci bot commented Aug 6, 2024

openshift-ci bot commented Aug 6, 2024

jiridanek commented Jul 31, 2024 •

edited

Loading

jstourac commented Jul 31, 2024 •

edited

Loading

harshad16 commented Jul 31, 2024 •

edited

Loading