osx can't build due to no space left on device

[jimmyliao]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

g-extensions-3.7.4.3 urllib3-1.26.7 wrapt-1.12.1 zipp-3.5.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Error: error committing container for step {Env:[PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D PYTHON_VERSION=3.7.12 PYTHON_PIP_VERSION=21.2.4 PYTHON_SETUPTOOLS_VERSION=57.5.0 PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/c20b0cfd643cd4a19246ccf204e2997af70f6b21/public/get-pip.py PYTHON_GET_PIP_SHA256=fa6f3fb93cce234cd4e8dd2beb54a51ab9c247653b52855a48dd44e6b21ff28b] Command:run Args:[pip install -r requirements.txt] Flags:[] Attrs:map[] Message:RUN pip install -r requirements.txt Original:RUN pip install -r requirements.txt}: error copying layers and metadata for container "80ffc4528da65d5f2c7ef516711fe94b0be9908a65b9386cfcdc1c95a56a3368": writing blob: adding layer with blob "sha256:9e9e4dd78a45a27380d9df62bd5187c9d094a636c2c7d97dbc2a3829bda71df0": Error processing tar file(exit status 1): open /usr/local/lib/python3.7/site-packages/tensorflow/python/training/checkpoint_state_pb2.py: no space left on device

Steps to reproduce the issue:

1. Dockerfile

FROM python:3.7-slim
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt

2. requirements.txt

seldon-core==1.6.0
tensorflow
Pillow

3. podman build . -t myname

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Tue Aug 31 03:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Tue Aug 31 04:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.16-200.fc34.x86_64
  linkmode: dynamic
  memFree: 1700859904
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 20m 4.01s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 20:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

I would guess the storage on your system is running out of space?

[rhatdan]

Could /var/tmp be loaded up?

[rhatdan]

You can change the location using TMPDIR=$HOME/tmp.

[Luap99]

The coreos vm only has 10 GB by default, @baude @ashley-cui should the size be increased?

[baude]

The default could be ... you can also increase disk size as an option to init. To me, this screams that we need a machine section in common with all this stuff.

[ashley-cui]

@baude if we want to do a [machine] section in containers.conf we should do that asap, since machine_image and helper_binaries are already in common, but not yet in a podman release, so we can move them without breaking people. We can also add the disk sizeoption there too. machine_enabled has been released and is in the [engine] table, im not sure if we should move that into a machine table though. If this sounds good, I can take it.

[Luap99]

helper_binaries is general, we will need this for other podman binaries. Other than that an extra [machine] section for the fields SGTM.

[baude]

@ashley-cui knock that out and we can review it as part of the regular review process. thanks

[rhatdan]

I think the default should be 20GB and be configurable. I just got a new laptop that defaults to 1TB sdd.

[mheon]

Are we sure that this is an issue with legitimate depletion of space on the machine? I'm a little worried we might have a storage leak from not cleaning up temporary files from builds, or similar.

[afbjorklund]

We used 20G for "old" machine, and people still wanted that to be bumped...

$  podman-machine create --driver qemu --help
--qemu-cpu-count "1" Number of CPUs
--qemu-memory "1024" Size of memory for host in MB
--qemu-disk-size "20000" Size of disk for host in MB

EDIT: Actually it was 20,000M which always raised some eyebrows (~19.5G)

The filesystem has some "tax" as well, so the usable amount is somewhat lower.

---

With CoreOS, it seems to be about 18G left from an initial value of 20G.

$ podman machine ssh df -h /sysroot
Could not create directory '/root/.ssh'.
Warning: Permanently added '[localhost]:41299' (ECDSA) to the list of known hosts.
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda4        20G  1.6G   18G   9% /sysroot

[baude]

I dont mind 20G as a default. Also, it will be configurable in 3.4+. My feeling is just choose something and users can adjust if they have large requirements.

[vrothberg]

It's now configurable in containers.conf (and merged into Podman).

@ashley-cui, @baude, I think it would be nice to have machine data in podman info. This way, we'd get size, cpus and memory directly in the reports which can come in handy.

[rhatdan]

Which machine? All of them?

[afbjorklund]

I thought it already had cpus and memTotal ? But maybe there is a way to wire in the "disk free" too (but: which disk, etc)

Possibly the graphRoot location. But then it would be available for all podman users, and not just for machine I suppose ?

$ podman machine ssh df -h /var/lib/containers/storage
Could not create directory '/root/.ssh'.
Warning: Permanently added '[localhost]:45825' (ECDSA) to the list of known hosts.
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda4        20G  3.5G   17G  18% /var

So like "diskFree" and "diskTotal" ?

[ashley-cui]

@vrothberg We don't have a machine info but some info should be available via list (#11691). Do you think we should also create a machine info as well? @baude

[afbjorklund]

There is no good GUI tool for qemu, is there ? I tried to find something, but there was only QMP and virt-manager

Otherwise that could be a good place to get some graphs over the current machine usage versus the total available.

[vrothberg]

@vrothberg We don't have a machine info but some info should be available via list (#11691). Do you think we should also create a machine info as well? @baude

podman machine list already gives the info, but I want that info in podman info as well. That's what we request in issue and BZs and having data on (running) machines can help speed things up.

[vrothberg]

Which machine? All of them?

Yes, all of them. Similar to what we're already doing with registries.

[rhatdan]

SGTM

[rhatdan]

Does podman -remote info print information about local environment now, or is it all from remote? I think we print version info.

[ashley-cui]

@rhatdan it prints local info, but if remote is not set up correctly, it errors and prints out nothing. Maybe we should print out local info regardless if the remote call fails?

[rhatdan]

I think we had a request for this earlier that @vrothberg was looking at, but that sounds good to me.

[voquangphu]

Just found out this thread and this helped me a lot by allocating more disk space during init.
That being said comparing to Docker I see that Podman cache and image size are still too big.
Docker with nearly 10 images took up only 25GBs while only 3 Podman images already takes up more than 50GBs.

[rhatdan]

That does not make much sense. Could you give us the output of podman info and podman images.

[matejvasek]

while only 3 Podman images already takes up more than 50GBs.

That's really odd, I have on my machine (Fedora) over 60 images and they take only 20GB.

Try running podman system df.

[voquangphu]

christopher.vo@P81579 hotelinfo % podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         27          0           57.04GB     57.04GB (100%)
Containers     0           0           0B          0B (0%)
Local Volumes  0           0           0B          0B (0%)

There you can see how much space it actually takes, so I had to re-configure VM size to allow these.
Here is the actual size of the images (I deleted 2, they used to take up about 3GB each).
I think the rest of the space is for build caching since every time I tried to reclaim all available space and re-build images the cache disappeared.

christopher.vo@P81579 hotelinfo % podman images
REPOSITORY             TAG         IMAGE ID      CREATED       SIZE
localhost/hotelinfo    latest      f6abd8db23a1  39 hours ago  3.38 GB
quay.io/centos/centos  6.6         368c96d786ae  2 years ago   213 MB

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Where do we stand on this, is this just the podman-machine was not big enough to start? Is there something that Podman can fix?

[mheon]

Make the machine bigger by default? That seems to be what the competing solutions have done.

[baude]

we did, we made it 20GB by default and it is configurable on the command line.

[vrothberg]

Are we good to close the issue?

[Luap99]

Looking at the code the default is still 10 GB, so no. https://github.com/containers/common/blob/77aa0316f20c77d094bc3205aaad3377640f2e39/pkg/config/default.go#L225-L232

[vrothberg]

Always good to have you watch our backs, thanks @Luap99

[vrothberg]

containers/common#832

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since common PR got merged. I am going to close.