Cannot build Singularity on a Pi with 512MB RAM - out of ram

[dtrudg]

Version of Singularity:

3.2.1

Expected behavior

make succeeds and builds Singularity - or perhaps a warning if builds in less than x MB RAM aren't feasible.

Actual behavior

I have a variety of fun cheap Pi's / clones on my desk and was trying to get latest Singularity on each one. The only failure was on a Pi Zero with 512MB RAM....

10:14 pm $ make --debug
GNU Make 4.1
Built for arm-unknown-linux-gnueabihf
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
build flag -mod=vendor only valid when using modules
build flag -mod=vendor only valid when using modules
Updating goal targets....
 File 'all' does not exist.
   File 'singularity' does not exist.
     Prerequisite 'config.h' is newer than target '/home/pi/go/src/github.com/sylabs/singularity/internal/pkg/buildcfg/config.go'.
    Must remake target '/home/pi/go/src/github.com/sylabs/singularity/internal/pkg/buildcfg/config.go'.
    Successfully remade target file '/home/pi/go/src/github.com/sylabs/singularity/internal/pkg/buildcfg/config.go'.
  Must remake target 'singularity'.
 GO singularity
    [+] GO_TAGS "containers_image_openpgp sylog apparmor selinux seccomp"
# github.com/sylabs/singularity/cmd/singularity
/opt/go/pkg/tool/linux_arm/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory

Makefile:154: recipe for target 'singularity' failed
make: *** [singularity] Error 2

Adding -x to the go build in the generated Makefile reveals not much more info...

mkdir -p $WORK/b001/exe/
cd .
/opt/go/pkg/tool/linux_arm/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=FMIs8d7E8epJ0-mVPDgn/HXLuklx0CBHXnkJ2PA-n/jkDDQU2Pv3scjuSpvg-F/FMIs8d7E8epJ0-mVPDgn -B 0xd371fecdc73323aa3e9a6088c2e904693462a5db -extld=gcc /home/pi/.cache/go-build/cd/cde567f245f86211b7dfe09661b308f3a174e91bb55fe0fe09366f99b1144274-d
# github.com/sylabs/singularity/cmd/singularity
/opt/go/pkg/tool/linux_arm/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory

Workaround for me was to increase the size of the swapfile to 512MB under /etc/dphys-swapfile and go make a coffee while it builds. Could have also built on an arm board with more RAM and transferred.

Steps to reproduce behavior

Try to build current Singularity from source on an arm system (and maybe other architectures) with 512MB RAM and small (<<RAM) swapfile.

[jscook2345]

I think we should start by finding a good way to profile the memory usage during the build. We may also want to profile some tests to verify a minimum memory amount.

[dtrudg]

There is actually an issue for go itself that linker RAM usage increased with Go 1.11 - this probably explains why I hadn't seen this a while ago - the last build I did on a Pi with little RAM was with Go 1.10 on an older Singularity 3.x

golang/go#26186

<edit>I'm not so sure on this actually - we are in gcc as an external linker here... I need to look at this a bit more when I have an hour</edit>

[tri-adam]

Just wanted to comment that I'm seeing this on a Raspberry Pi 3 with 1GB RAM as well.

[dtrudg]

Also can hit this now on a Pi3. I'm still okay on a Pi4 2GB arm and LibreComputer Potato 2GB arm64. Will look into the cause in more detail when I have time.

[dtrudg]

This is something we aren't going to be able to address ourselves. The Go linker RAM usage is high enough on simpler programs than Singularity that a build on a different machine with more RAM will be required.