IBM hosted docker containers offline

[richardlau]

All of the softlayer (IBM) hosted docker containers are currently offline. It looks like we've run out of disk space.

https://ci.nodejs.org/computer/

[richardlau]

iojs@test-softlayer-ubuntu1804-docker-x64-1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  1.6M  3.2G   1% /run
/dev/xvda2       99G   94G   16M 100% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/xvda1      240M  137M   91M  61% /boot
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/de91f608a9b532b8b55d5734caf2091ae2178d0cea094a61b9fa50e0bad18fcb/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/8967b934f69ddc566576abe34a4f054b9bacac0a9ae7a1afa985818e5ed95079/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/a66eeaca38645222ce7d322390a5c24fccf2ab850ebcf974c3f120ca504bcea4/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/042b6fcd71e01c8d07c6eada4c230e128f64907fe7902d6541ff3ffee25cdfa0/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/a85c96d672fc61940197a0e7a085a4a73ae44355f3dacbbfc39b401cc6556aa9/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/fd08e8b8f6747377c52560023c6a4757be11d53f95a74f1a5b35a6ec8bfb278a/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/eb14b654e7c578dba3696183a04e7455100bd7d9820e15ce3d2fa6f0e7ea0c5f/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/c208a1594517b91eb74131d5e1446fa8464c69f6654210370d09c26dc51ef638/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/1a3931e363c807f38fb8e80f8e4444ebaef9d663adfc33249bdb99c92bbed59a/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/2786aa9b6e8955a15b06550014517aad7aef59492e5636ca45f8ca1b7e540839/merged
overlay          99G   94G   16M 100% /var/lib/docker/overlay2/823a012ffb357e1fea25c1a4bb99e2c1fa5a5eec2294cf66797955bf94c72f02/merged
tmpfs           3.2G     0  3.2G   0% /run/user/0
iojs@test-softlayer-ubuntu1804-docker-x64-1:~$

iojs@test-softlayer-ubuntu1804-docker-x64-1:~$ du -hs /home/iojs/
du: cannot access '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-4/tmp/.tmp.2130N0UDsX/middle/leaf': Permission denied
du: cannot access '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/node-tmp/.tmp.2047eB1isj/middle/leaf': Permission denied
du: cannot access '/home/iojs/test-softlayer-ubi81_container-x64-1/node-tmp/.tmp.20478LZJ1N/middle/leaf': Permission denied
du: cannot read directory '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-5/tmp/.tmp.2183JNkJXS/middle': Permission denied
du: cannot access '/home/iojs/test-softlayer-alpine311_container-x64-1/node-tmp/.tmp.2047IpbCEh/middle/leaf': Permission denied
du: cannot access '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/node-tmp/.tmp.2047YdGl5b/middle/leaf': Permission denied
66G     /home/iojs/
iojs@test-softlayer-ubuntu1804-docker-x64-1:~$ du -hs /home/iojs/*
0       /home/iojs/�
du: cannot access '/home/iojs/test-softlayer-alpine311_container-x64-1/node-tmp/.tmp.2047IpbCEh/middle/leaf': Permission denied
2.8G    /home/iojs/test-softlayer-alpine311_container-x64-1
2.7G    /home/iojs/test-softlayer-alpine312_container-x64-1
du: cannot access '/home/iojs/test-softlayer-ubi81_container-x64-1/node-tmp/.tmp.20478LZJ1N/middle/leaf': Permission denied
2.7G    /home/iojs/test-softlayer-ubi81_container-x64-1
298M    /home/iojs/test-softlayer-ubuntu1604_arm_cross_container-x64-1
15G     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1
312M    /home/iojs/test-softlayer-ubuntu1804_container-x64-1
du: cannot access '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/node-tmp/.tmp.2047eB1isj/middle/leaf': Permission denied
11G     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1
2.7G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-2
du: cannot access '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/node-tmp/.tmp.2047YdGl5b/middle/leaf': Permission denied
2.7G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3
du: cannot access '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-4/tmp/.tmp.2130N0UDsX/middle/leaf': Permission denied
2.7G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-4
du: cannot read directory '/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-5/tmp/.tmp.2183JNkJXS/middle': Permission denied
2.4G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-5
iojs@test-softlayer-ubuntu1804-docker-x64-1:~$

of those the outliers are

15G     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1
11G     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1

test-softlayer-ubuntu1804_sharedlibs_container-x64-1 is a debug build, with about 1 G in .git (which seems in line with the other container workspaces on the machine) and 9 G in out (which is not out of the ordinary for a debug build) for its workspace.

iojs@test-softlayer-ubuntu1804-docker-x64-1:~$ du -hs /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/.git /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/*
953M    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/.git
124K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/AUTHORS
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/BSDmakefile
36K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/BUILDING.md
56K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/CHANGELOG.md
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/CODE_OF_CONDUCT.md
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/CONTRIBUTING.md
8.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/GOVERNANCE.md
84K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/LICENSE
52K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/Makefile
32K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/README.md
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/SECURITY.md
52K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/__pycache__
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/android-configure
1.6M    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/benchmark
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/codecov.yml
20K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/common.gypi
24K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/config.gypi
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/config.mk
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/config.status
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/configure
72K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/configure.py
396M    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/deps
11M     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/doc
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/glossary.md
72K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/icu_config.gypi
3.3M    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/lib
0       /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/node
44K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/node.gyp
12K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/node.gypi
16K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/onboarding.md
9.1G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/out
8.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/select-compiler.sh
4.6M    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/src
48M     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/test
30M     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/tools
4.0K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/tsconfig.json
104K    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/typings
36K     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1/build/workspace/node-test-commit-linux-containered/vcbuild.bat
iojs@test-softlayer-ubuntu1804-docker-x64-1:~$

on the otherhand, test-softlayer-ubuntu1804_arm_cross_container-x64-1 has 13 G in .git and 1 G in out for its workspace.

iojs@test-softlayer-ubuntu1804-docker-x64-1:~$ du -hs /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/.git
 /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/*
13G     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/.git
124K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/AUTHORS
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/BSDmakefile
36K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/BUILDING.md
56K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/CHANGELOG.md
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/CODE_OF_CONDUCT.md
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/CONTRIBUTING.md
8.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/GOVERNANCE.md
84K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/LICENSE
52K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/Makefile
32K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/README.md
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/SECURITY.md
52K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/__pycache__
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/android-configure
1.6M    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/benchmark
72M     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/binary
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/codecov.yml
20K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/common.gypi
24K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/config.gypi
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/config.mk
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/config.status
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/configure
72K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/configure.py
396M    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/deps
11M     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/doc
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/glossary.md
72K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/icu_config.gypi
3.3M    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/lib
0       /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/node
44K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/node.gyp
12K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/node.gypi
16K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/onboarding.md
875M    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/out
4.6M    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/src
48M     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/test
30M     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/tools
4.0K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/tsconfig.json
104K    /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/typings
36K     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/vcbuild.bat
iojs@test-softlayer-ubuntu1804-docker-x64-1:~$

[richardlau]

I ran git prune in /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1/build/workspace/node-cross-compile/.git and that's dropped the space used by .git to 5 G. Running a subsequent git gc in the same directory drops the space used to 1 G (I wasn't able to run git gc before git prune as it needs to write things to the disk and the disk was full).

There's now 13 G free on the disk

iojs@test-softlayer-ubuntu1804-docker-x64-1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  1.6M  3.2G   1% /run
/dev/xvda2       99G   82G   13G  87% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/xvda1      240M  137M   91M  61% /boot
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/de91f608a9b532b8b55d5734caf2091ae2178d0cea094a61b9fa50e0bad18fcb/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/8967b934f69ddc566576abe34a4f054b9bacac0a9ae7a1afa985818e5ed95079/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/a66eeaca38645222ce7d322390a5c24fccf2ab850ebcf974c3f120ca504bcea4/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/042b6fcd71e01c8d07c6eada4c230e128f64907fe7902d6541ff3ffee25cdfa0/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/a85c96d672fc61940197a0e7a085a4a73ae44355f3dacbbfc39b401cc6556aa9/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/fd08e8b8f6747377c52560023c6a4757be11d53f95a74f1a5b35a6ec8bfb278a/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/eb14b654e7c578dba3696183a04e7455100bd7d9820e15ce3d2fa6f0e7ea0c5f/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/c208a1594517b91eb74131d5e1446fa8464c69f6654210370d09c26dc51ef638/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/1a3931e363c807f38fb8e80f8e4444ebaef9d663adfc33249bdb99c92bbed59a/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/2786aa9b6e8955a15b06550014517aad7aef59492e5636ca45f8ca1b7e540839/merged
overlay          99G   82G   13G  87% /var/lib/docker/overlay2/823a012ffb357e1fea25c1a4bb99e2c1fa5a5eec2294cf66797955bf94c72f02/merged
tmpfs           3.2G     0  3.2G   0% /run/user/0
iojs@test-softlayer-ubuntu1804-docker-x64-1:~$

Looking at the cross compile job part of the job adds the compiled binaries to a temporary branch and pushes to the temp binary git repo:

git add binary/binary.tar.xz
git checkout -B $TEMP_PUSH_BRANCH/$git_branch
...
git commit -m "added binaries"
...
ssh-agent sh -c "ssh-add $JENKINS_TMP_KEY && git push $TEMP_REPO +$TEMP_PUSH_BRANCH/$git_branch"
rm -f $JENKINS_TMP_KEY

git reset --mixed HEAD^1
git checkout jenkins_tmp/_jenkins_local_branch
git branch -D $TEMP_PUSH_BRANCH/$git_branch

I've added a git gc after the above (audit https://github.com/nodejs/jenkins-config-test/commit/b2700490c65c03985ee869ca7809c3d07c03ff41) which should hopefully mean we don't have to periodically manually prune/gc the workspace.