Skip to content

Commit

Permalink
[KYUUBI #6047] [CI] Free up disk space
Browse files Browse the repository at this point in the history
I saw [the K8s IT](https://github.com/apache/kyuubi/actions/runs/7776268590/job/21203214430?pr=6045) failed with

```
Error: No space left on device : '/home/runner/runners/2.312.0/_diag/pages/131e7057-723d-4281-b296-b876237f4318_21774d93-c106-57cd-810b-f0e6bf96d865_1.log'
```

Borrow the solution from Apache Spark, just uninstall the un-used large pre-installed packages.

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

Wait CI report

---

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6047 from pan3793/ci-space.

Closes #6047

272df27 [Cheng Pan] nit
8c78115 [Cheng Pan] nit
a9c8b03 [Cheng Pan] nit
ab4c3e4 [Cheng Pan] keep jdk8
9d1ec4e [Cheng Pan] extend
47f90b9 [Cheng Pan] nit
30b82cd [Cheng Pan] fix
352689f [Cheng Pan] [CI] Free up disk space for K8s IT

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 5f99ec8)
Signed-off-by: Cheng Pan <[email protected]>
  • Loading branch information
pan3793 committed Feb 5, 2024
1 parent 3383c2e commit 06cc3db
Show file tree
Hide file tree
Showing 4 changed files with 81 additions and 8 deletions.
55 changes: 55 additions & 0 deletions .github/scripts/free_disk_space.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# This script is inspired by Apache Spark

echo "=================================="
echo "Free up disk space on CI system"
echo "=================================="

echo "Listing 100 largest packages"
dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
df -h

echo "Removing large packages"
sudo rm -rf /usr/share/dotnet/
sudo rm -rf /usr/share/php/
sudo rm -rf /usr/local/graalvm/
sudo rm -rf /usr/local/.ghcup/
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/local/share/chromium
sudo rm -rf /usr/local/lib/android
sudo rm -rf /usr/local/lib/node_modules

sudo rm -rf /opt/az
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo rm -rf /opt/hostedtoolcache/go
sudo rm -rf /opt/hostedtoolcache/node

sudo apt-get remove --purge -y '^aspnet.*'
sudo apt-get remove --purge -y '^dotnet-.*'
sudo apt-get remove --purge -y '^llvm-.*'
sudo apt-get remove --purge -y 'php.*'
sudo apt-get remove --purge -y '^temurin-\d{n,}.*'
sudo apt-get remove --purge -y snapd google-chrome-stable microsoft-edge-stable firefox
sudo apt-get remove --purge -y azure-cli google-cloud-sdk mono-devel msbuild powershell libgl1-mesa-dri
sudo apt-get autoremove --purge -y
sudo apt-get clean

df -h
24 changes: 20 additions & 4 deletions .github/workflows/master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ jobs:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
Expand Down Expand Up @@ -120,6 +122,8 @@ jobs:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
Expand Down Expand Up @@ -169,6 +173,8 @@ jobs:
comment: 'verify-on-flink-1.15-binary'
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
Expand Down Expand Up @@ -213,6 +219,8 @@ jobs:
comment: [ "normal" ]
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
Expand Down Expand Up @@ -248,6 +256,8 @@ jobs:
comment: [ "normal" ]
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
Expand Down Expand Up @@ -283,6 +293,8 @@ jobs:
comment: [ "normal" ]
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
Expand Down Expand Up @@ -313,6 +325,8 @@ jobs:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK 8
Expand All @@ -334,8 +348,9 @@ jobs:
name: Kyuubi Server On Kubernetes Integration Test
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v3
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
# https://github.com/docker/build-push-action
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
Expand Down Expand Up @@ -397,8 +412,9 @@ jobs:
name: Spark Engine On Kubernetes Integration Test
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v3
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Setup Minikube
run: |
# https://minikube.sigs.k8s.io/docs/start/
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/publish-snapshot-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,9 @@ jobs:
if: ${{ startsWith(github.repository, 'apache/') }}
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v3
- uses: actions/checkout@v3
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/publish-snapshot-nexus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,11 @@ jobs:
- branch: branch-1.6
profiles: -Pflink-provided,spark-provided,hive-provided,spark-3.3
steps:
- name: Checkout repository
uses: actions/checkout@v3
- uses: actions/checkout@v3
with:
ref: ${{ matrix.branch }}
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Setup JDK 8
uses: actions/setup-java@v3
with:
Expand Down

0 comments on commit 06cc3db

Please sign in to comment.