cygwin CI failure with LTO?

[eli-schwartz]

For the past couple of days, cygwin CI runners fail:

______________________ AllPlatformTests.test_lto_threads _______________________
[gw3] cygwin -- Python 3.9.16 /usr/bin/python3

self = <unittests.allplatformstests.AllPlatformTests testMethod=test_lto_threads>

    @skip_if_not_base_option('b_lto_threads')
    def test_lto_threads(self):
        testdir = os.path.join(self.common_test_dir, '6 linkshared')
    
        env = get_fake_env(testdir, self.builddir, self.prefix)
        cc = detect_c_compiler(env, MachineChoice.HOST)
        extra_args: T.List[str] = []
        if cc.get_id() == 'clang':
            if is_windows():
                raise SkipTest('LTO not (yet) supported by windows clang')
    
        self.init(testdir, extra_args=['-Db_lto=true', '-Db_lto_threads=8'] + extra_args)
>       self.build()
E       subprocess.CalledProcessError: Command '['/usr/bin/ninja', '-d', 'explain']' returned non-zero exit status 1.

unittests/allplatformstests.py:1368: CalledProcessError
----------------------------- Captured stdout call -----------------------------
$ /usr/bin/python3 /cygdrive/d/a/meson/meson/meson.py setup --backend=ninja --prefix /usr --libdir lib -Db_lto=true -Db_lto_threads=8 /cygdrive/d/a/meson/meson/tmpr23_u39j '/cygdrive/d/a/meson/meson/test cases/common/6 linkshared'
stdout:
The Meson build system
Version: 1.5.99
Source dir: /cygdrive/d/a/meson/meson/test cases/common/6 linkshared
Build dir: /cygdrive/d/a/meson/meson/tmpr23_u39j
Build type: native build
Project name: shared library linking test
Project version: undefined
C compiler for the host machine: cc (gcc 12.4.0 "cc (GCC) 12.4.0")
C linker for the host machine: cc ld.bfd 2.42
C++ compiler for the host machine: c++ (gcc 12.4.0 "c++ (GCC) 12.4.0")
C++ linker for the host machine: c++ ld.bfd 2.42
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 4

shared library linking test undefined

  User defined options
    b_lto        : true
    b_lto_threads: 8
    backend      : ninja
    libdir       : lib
    prefix       : /usr

Found ninja-1.12.0 at /usr/bin/ninja

$ /usr/bin/ninja -d explain
stdout:
ninja explain: deps for 'prog.exe.p/main.c.o' are missing
ninja explain: prog.exe.p/main.c.o is dirty
ninja explain: deps for 'cygmylib.dll.p/libfile.c.o' are missing
ninja explain: cygmylib.dll.p/libfile.c.o is dirty
ninja explain: cygmylib.dll is dirty
ninja explain: cygmylib.dll.p/cygmylib.dll.symbols is dirty
ninja explain: prog.exe is dirty
ninja explain: deps for 'cppprog.exe.p/cppmain.cpp.o' are missing
ninja explain: cppprog.exe.p/cppmain.cpp.o is dirty
ninja explain: deps for 'cygmycpplib.dll.p/cpplib.cpp.o' are missing
ninja explain: cygmycpplib.dll.p/cpplib.cpp.o is dirty
ninja explain: cygmycpplib.dll is dirty
ninja explain: cygmycpplib.dll.p/cygmycpplib.dll.symbols is dirty
ninja explain: cppprog.exe is dirty
ninja explain: meson-test-prereq is dirty
ninja explain: output meson-benchmark-prereq of phony edge with no inputs doesn't exist
ninja explain: meson-benchmark-prereq is dirty
ninja explain: cygmylib.dll is dirty
ninja explain: prog.exe is dirty
ninja explain: cygmycpplib.dll is dirty
ninja explain: cppprog.exe is dirty
[1/10] Compiling C++ object cygmycpplib.dll.p/cpplib.cpp.o
[2/10] Compiling C object cygmylib.dll.p/libfile.c.o
[3/10] Compiling C object prog.exe.p/main.c.o
[4/10] Compiling C++ object cppprog.exe.p/cppmain.cpp.o
[5/10] Linking target cygmycpplib.dll
[6/10] Linking target cygmylib.dll
[7/10] Generating symbol file cygmycpplib.dll.p/cygmycpplib.dll.symbols
[8/10] Generating symbol file cygmylib.dll.p/cygmylib.dll.symbols
[9/10] Linking target cppprog.exe
FAILED: cppprog.exe 
c++  -o cppprog.exe cppprog.exe.p/cppmain.cpp.o -flto -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,--start-group libmycpplib.dll.a -Wl,--subsystem,console -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
`main' referenced in section `.text' of /usr/lib/gcc/x86_64-pc-cygwin/12/../../../../lib/crt0.o: defined in discarded section `.text' of cppprog.exe.p/cppmain.cpp.o (symbol from plugin)
collect2: error: ld returned 1 exit status
[10/10] Linking target prog.exe
FAILED: prog.exe 
cc  -o prog.exe prog.exe.p/main.c.o -flto -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,--start-group libmylib.dll.a -Wl,--subsystem,console -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
`main' referenced in section `.text' of /usr/lib/gcc/x86_64-pc-cygwin/12/../../../../lib/crt0.o: defined in discarded section `.text' of prog.exe.p/main.c.o (symbol from plugin)
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

___________________ AllPlatformTests.test_static_library_lto ___________________
[gw0] cygwin -- Python 3.9.16 /usr/bin/python3

self = <unittests.allplatformstests.AllPlatformTests testMethod=test_static_library_lto>

    def test_static_library_lto(self):
        '''
        Test that static libraries can be built with LTO and linked to
        executables. On Linux, this requires the use of gcc-ar.
        https://github.com/mesonbuild/meson/issues/1646
        '''
        testdir = os.path.join(self.common_test_dir, '5 linkstatic')
    
        env = get_fake_env(testdir, self.builddir, self.prefix)
        if detect_c_compiler(env, MachineChoice.HOST).get_id() == 'clang' and is_windows():
            raise SkipTest('LTO not (yet) supported by windows clang')
    
        self.init(testdir, extra_args='-Db_lto=true')
>       self.build()
E       subprocess.CalledProcessError: Command '['/usr/bin/ninja', '-d', 'explain']' returned non-zero exit status 1.

unittests/allplatformstests.py:1353: CalledProcessError
----------------------------- Captured stdout call -----------------------------
$ /usr/bin/python3 /cygdrive/d/a/meson/meson/meson.py setup --backend=ninja --prefix /usr --libdir lib -Db_lto=true /cygdrive/d/a/meson/meson/tmp4awcuupw '/cygdrive/d/a/meson/meson/test cases/common/5 linkstatic'
stdout:
The Meson build system
Version: 1.5.99
Source dir: /cygdrive/d/a/meson/meson/test cases/common/5 linkstatic
Build dir: /cygdrive/d/a/meson/meson/tmp4awcuupw
Build type: native build
Project name: static library linking test
Project version: undefined
C compiler for the host machine: cc (gcc 12.4.0 "cc (GCC) 12.4.0")
C linker for the host machine: cc ld.bfd 2.42
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 2

static library linking test undefined

  User defined options
    b_lto  : true
    backend: ninja
    libdir : lib
    prefix : /usr

Found ninja-1.12.0 at /usr/bin/ninja

$ /usr/bin/ninja -d explain
stdout:
ninja explain: deps for 'prog.exe.p/main.c.o' are missing
ninja explain: prog.exe.p/main.c.o is dirty
ninja explain: deps for 'libmylib.a.p/libfile.c.o' are missing
ninja explain: libmylib.a.p/libfile.c.o is dirty
ninja explain: deps for 'libmylib.a.p/libfile2.c.o' are missing
ninja explain: libmylib.a.p/libfile2.c.o is dirty
ninja explain: deps for 'libmylib.a.p/libfile3.c.o' are missing
ninja explain: libmylib.a.p/libfile3.c.o is dirty
ninja explain: deps for 'libmylib.a.p/libfile4.c.o' are missing
ninja explain: libmylib.a.p/libfile4.c.o is dirty
ninja explain: libmylib.a is dirty
ninja explain: prog.exe is dirty
ninja explain: meson-test-prereq is dirty
ninja explain: output meson-benchmark-prereq of phony edge with no inputs doesn't exist
ninja explain: meson-benchmark-prereq is dirty
ninja explain: libmylib.a is dirty
ninja explain: prog.exe is dirty
[1/7] Compiling C object libmylib.a.p/libfile.c.o
[2/7] Compiling C object libmylib.a.p/libfile2.c.o
[3/7] Compiling C object libmylib.a.p/libfile3.c.o
[4/7] Compiling C object libmylib.a.p/libfile4.c.o
[5/7] Compiling C object prog.exe.p/main.c.o
[6/7] Linking static target libmylib.a
[7/7] Linking target prog.exe
FAILED: prog.exe 
cc  -o prog.exe prog.exe.p/main.c.o -flto -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,--start-group libmylib.a -Wl,--subsystem,console -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
`main' referenced in section `.text' of /usr/lib/gcc/x86_64-pc-cygwin/12/../../../../lib/crt0.o: defined in discarded section `.text' of prog.exe.p/main.c.o (symbol from plugin)
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

=========================== short test summary info ============================
FAILED unittests/allplatformstests.py::AllPlatformTests::test_lto_threads - subprocess.CalledProcessError: Command '['/usr/bin/ninja', '-d', 'explain']' returned non-zero exit status 1.
FAILED unittests/allplatformstests.py::AllPlatformTests::test_static_library_lto - subprocess.CalledProcessError: Command '['/usr/bin/ninja', '-d', 'explain']' returned non-zero exit status 1.
============ 2 failed, 399 passed, 128 skipped in 697.02s (0:11:37) ============

@jon-turney

[jon-turney]

Thanks for pinging me about this.

A gcc package update was just rolled out, so i suspect that may well be involved.

[jon-turney]

Unfortunately, this is unlikely to be resolved quickly, so I suggest you disable these tests for the moment.

[eli-schwartz]

Disabled for now. Keeping the issue open.

[jon-turney]

This is a weird one, and the error message is totally misleading. You fix this by installing make.

make is used internally by gcc (for scheduling multithreaded lto) when the '-flto' flag is used.

If it's missing, gcc is supposed to fall back to serial lto (e.g. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97524), but it seems that there's some regression in 12.x where it emits this confusing error message instead.

[eli-schwartz]

Nice...

(It looks like there was a fix but just never backported. Very logical...)

[jon-turney]

(It looks like there was a fix but just never backported. Very logical...)

Fixed in 11.0, and then it seems to have regressed again in 12.x (although that might be a cygwin specific thing, it's proved awkward for me to confirm that, one way or the other).

FTR, here's a one-line reproducer.

( export MAKE=nonexistent && echo "void main() { }" | gcc -x c -flto=auto - )