Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tutorial case backwardFacingStep2D hangs forever after compiling with Xcode CLT >= 14.3 #8

Closed
gerlero opened this issue Jun 21, 2023 · 57 comments

Comments

@gerlero
Copy link
Contributor

gerlero commented Jun 21, 2023

$FOAM_TUTORIALS/incompressible/simpleFoam/backwardFacingStep2D gets stuck before the completing the first timestep after OpenFOAM is compiled with Xcode Command Line Tools >= 14.3 (gerlero/openfoam-app#160). Apparently, it hangs between the processing these lines:

    #includeFunc "pressureCoefficient"
    #includeFunc "sample"

in the controlDict.

I tried with your repo to see if I it was me who was doing something wrong, but after following the build instructions here I still run into the same problem. Do you have an idea on what could be going on here?

EDIT: removing mention of "Clang 14" as it might be confusing (Xcode CLT < 14.3 also comes with Clang 14)

@gerlero
Copy link
Contributor Author

gerlero commented Jun 21, 2023

BTW for anyone else seeing this, a workaround is to use Xcode Command Line Tools 14.2 to compile OpenFOAM.

@BrushXue
Copy link
Owner

Is that a ARM-specific problem or universal? I tested it with Xcode 14.0 on a x86 Hackintosh and I don't have any problem.

@gerlero
Copy link
Contributor Author

gerlero commented Jun 21, 2023

Is that a ARM-specific problem or universal?

Good question. My Intel builds so far are using CLT 13.2 on Big Sur, so it's expected that those are not affected. I'll try a compile with Xcode CLT 14.3 (Clang 14) on Intel Ventura to see what happens.

Xcode 14.0

Just to note, Xcode CLT 14.3 and 14.3.1 seem to trigger the problem (on Ventura at least). 14.2 and before work fine.

@gerlero
Copy link
Contributor Author

gerlero commented Jun 22, 2023

I'm back with some results (macOS Ventura 13.4 on Intel, Xcode Command Line Tools 14.3.1.0.1.1683849156):

OpenFOAM v2212

  • build OK
  • backwardFacingStep2D runs OK

OpenFOAM v2206

wmake meshTools
    Ctoo: geompack.C
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: x86_64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
clang: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/2p/6fxzqc0d64xbxvt_6n0252k40000gn/T/geompack-20d5e5.cpp
clang: note: diagnostic msg: /var/folders/2p/6fxzqc0d64xbxvt_6n0252k40000gn/T/geompack-20d5e5.sh
clang: note: diagnostic msg: Crash backtrace is located in
clang: note: diagnostic msg: /Users/runner/Library/Logs/DiagnosticReports/clang_<YYYY-MM-DD-HHMMSS>_<hostname>.crash
clang: note: diagnostic msg: (choose the .crash file that corresponds to your crash)
clang: note: diagnostic msg: 

********************

OpenFOAM v2112

Similar errors to v2206

Conclusion

Seems only affect Apple silicon as of v2212 (and not Intel). IDK what's going in with v2206 and v2112 making the compiler crash, but that's a different (although possibly related) issue.

The Clang bundled in Xcode CLT 14.3 seems to have enabled floating point exception support on Apple silicon (as it no longer emits the "Overriding currently unsupported..." compile warnings), so this issue might have something to do with that (OpenFOAM issue about FPE on arm64)

EDIT: disregard these results!. This was actually built with Xcode 14.2, as can be clearly seen in the log output. My bad! This doesn't say anything on whether the problem also affects Intel or not.

@BrushXue
Copy link
Owner

I’ve seen this geompack.C patch before.
https://github.com/mrklein/openfoam-os-x/blob/274546a8aec2911e9078e3b070d2f028460b4f51/OpenFOAM-v2206.patch#L195
Maybe it’s related to this issue.

@gerlero
Copy link
Contributor Author

gerlero commented Jun 22, 2023

Good find. That patch looks like it's worth a shot.

@gerlero
Copy link
Contributor Author

gerlero commented Jun 23, 2023

Well, looks like that patch has been upstreamed (https://develop.openfoam.com/Development/openfoam/-/issues/2555), and is already part of v2212. This likely explains why earlier OpenFOAM versions fail to compile on Intel, but it won't do anything for this issue.

EDIT: https://develop.openfoam.com/Development/openfoam/-/issues/2555 directly mentions the compiler segfaults, so confirmed that that's the cause of the build failures on Intel

@gerlero
Copy link
Contributor Author

gerlero commented Jul 5, 2023

OpenFOAM v2306 seems to also be affected by this. @BrushXue what version of Xcode/CLT are you using to build? Are you able to confirm this issue?

@BrushXue
Copy link
Owner

BrushXue commented Jul 5, 2023

I'm staying in Monterey because Ventura drains my battery quickly. I didn't encounter your problem.

$ clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Xcode 14.3 dropped Monterey support.

@gerlero
Copy link
Contributor Author

gerlero commented Jul 5, 2023

Thanks. This is the offending version on Ventura:

Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

I assume yours is the latest clang available for Monterey? Then, this issue affects only macOS Ventura on Apple silicon.

I'm fairly confident in pointing the finger at FPE support (which didn't work at all with previous Clang versions) but I've tried every way I could think of disabling that and the problem always persists.

EDIT: saw your edit. Since CLT 14.3 is Ventura-only then this issue can only affect this macOS version.

@BrushXue
Copy link
Owner

BrushXue commented Jul 5, 2023

What if you patch these lines? (I removed these recently because I thought export FOAM_SIGFPE=false was enough to disable sigfpe)

diff --git a/src/OSspecific/POSIX/signals/sigFpe.C b/src/OSspecific/POSIX/signals/sigFpe.C
index 29a95852..bc284b9b 100644
--- a/src/OSspecific/POSIX/signals/sigFpe.C
+++ b/src/OSspecific/POSIX/signals/sigFpe.C
@@ -44,7 +44,7 @@ License
#include <malloc.h>
#endif
-#ifdef __APPLE__
+#if defined(__APPLE__) && defined(__x86_64__)
#include "feexceptErsatz.H"
#endif
@@ -117,7 +117,7 @@ void* Foam::sigFpe::mallocNan(size_t size)
void Foam::sigFpe::sigHandler(int)
{
- #if (defined(__linux__) && defined(__GNUC__)) || defined(__APPLE__)
+ #if (defined(__linux__) && defined(__GNUC__)) || (defined(__APPLE__) && defined(__x86_64__))
resetHandler("SIGFPE", SIGFPE);
@@ -184,7 +184,7 @@ void Foam::sigFpe::set(bool verbose)
{
if (!sigActive_ && requested())
{
- #if (defined(__linux__) && defined(__GNUC__)) || defined(__APPLE__)
+ #if (defined(__linux__) && defined(__GNUC__)) || (defined(__APPLE__) && defined(__x86_64__))
feenableexcept
(
@@ -240,7 +240,7 @@ void Foam::sigFpe::set(bool verbose)
void Foam::sigFpe::unset(bool verbose)
{
- #if (defined(__linux__) && defined(__GNUC__)) || defined(__APPLE__)
+ #if (defined(__linux__) && defined(__GNUC__)) || (defined(__APPLE__) && defined(__x86_64__))
if (sigActive_)
{
if (verbose)

@gerlero
Copy link
Contributor Author

gerlero commented Jul 5, 2023

Will try and report back.

I thought export FOAM_SIGFPE=false was enough to disable sigfpe

It's enough to disable it at runtime, but doing that doesn't seem to help for this particular issue. Setting sigFpe 0; in etc/controlDict should do the same (tried that too but still no luck).

The issue behaves like it's a compiler thing, so I've also tried disabling FPE support at the compiler level by removing the -ftrapping-math flag from the darwin64Clang (or replacing it with -fno-trapping-math, which should be the default) before building OpenFOAM. That didn't fix it either, even when running with FOAM_SIGFPE=false.

@BrushXue
Copy link
Owner

BrushXue commented Jul 5, 2023

Do you think if this is related to your issue?
python/cpython#97524 (comment)

@gerlero
Copy link
Contributor Author

gerlero commented Jul 5, 2023

Do you think if this is related to your issue?
python/cpython#97524 (comment)

Not sure. IIRC I've been seeing those ld warnings for a while (~ since Ventura came out), while this problem is more recent...

EDIT: typo

@BrushXue
Copy link
Owner

BrushXue commented Jul 5, 2023

Yes I've been using -Wl,-w to suppress these warnings for a while. But the compiler changed it behavior again since 14.3.

@gerlero
Copy link
Contributor Author

gerlero commented Jul 5, 2023

But the compiler changed it behavior again since 14.3.

Oh, thanks for this. I only skimmed the CPython issue and managed to miss that part. I'll try one of the workarounds mentioned in that thread if the sigFpe.C patch doesn't do it.

@gerlero
Copy link
Contributor Author

gerlero commented Jul 5, 2023

Thanks for the suggestions. Unfortunately, neither the sigFpe.C patch nor the addition of a -no_fixup_chains linker flag (nor a combination of the two) seem to help. I guess we'll have to keep looking...

BTW, we can disregard my previous results of builds on Intel Ventura, as those were actually using Xcode 14.2 (can't believe I missed that). So, I'm still not sure whether this also affects Intel or not. Not sure what hardware you have access too, but you're able to try a build with 14.3 on Intel Ventura, that should be helpful.

@BrushXue
Copy link
Owner

BrushXue commented Jul 5, 2023

Well my hackintosh is i7-6700K which simulates a 2015 iMac. Unfortunately Apple dropped Ventura support for that model. Maybe I can try to install another newer OS on my laptop.

@gerlero
Copy link
Contributor Author

gerlero commented Jul 6, 2023

I tried compiling for x86_64 on my M1 through Rosetta 2, and in that case OpenFOAM works just fine. So, this is still likely an Apple silicon–only issue

@BrushXue
Copy link
Owner

BrushXue commented Jul 6, 2023

Have you tried other compiler flags like -O1?

@gerlero
Copy link
Contributor Author

gerlero commented Jul 6, 2023

Same issue after compiling with -O0 😕

@BrushXue
Copy link
Owner

BrushXue commented Jul 8, 2023

This is a irrelevant story, but here’s my experience dealing with a very weird bug in the past two days:

I recently upgraded my workstation to Ubuntu 22.04. My viscoelastic fluid solver started to behave strangely and always crashes after a few time steps. It was working on all platforms including ARM. I tried to compile it with Clang 14 again the problem was solved. So it turned out that the gcc 11 came with Ubuntu 22.04 has floating point accuracy problem.

Conclusion: don’t use bleeding edge compilers.

@gerlero
Copy link
Contributor Author

gerlero commented Jul 8, 2023

Yeah, hitting a compiler/toolchain bug has to be one of the worst things. Good that you managed to find it before losing your head (I assume!).

Not sure I'd want to call the current non-beta release of Xcode CLT/Apple Clang "bleeding edge" though—although I'll have to partially concede, having hit a very annoying bug in the non-beta CLT that required a downgrade in the very recent past. Unfortunately, this time I'm having no luck at all tracing a cause, upstream ticket or any similar report. Thought CLT 14.3 would finally let us enable FPE, but I guess that'll have to wait.

@BrushXue
Copy link
Owner

BrushXue commented Jul 8, 2023

CLT 14.3 would finally let us enable FPE

Is there any source mentioning this new feature?

@gerlero
Copy link
Contributor Author

gerlero commented Jul 8, 2023

Not that I could find. Just that the “overriding unsupported […] on this target” warnings when passing -ftrapping-math (as OpenFOAM does) are gone since 14.3. I haven’t tested if the floating point exceptions actually trigger now

@BrushXue
Copy link
Owner

BrushXue commented Jul 8, 2023

This is kind of hopeless. I tried ARM HPC which also uses ARMv8 and Clang, and it doesn't support sigfpe. I doubt if ARM and Apple has hardware implementation.

@BrushXue
Copy link
Owner

BrushXue commented Aug 1, 2023

I just tested on my Intel MacBook(macOS 15 beta, Xcode 15 beta), and I couldn't reproduce this bug. It's 100% an ARM-only issue.

@gerlero
Copy link
Contributor Author

gerlero commented Aug 1, 2023

Thanks. Then it's 100% confirmed to be Apple silicon–only, and it might (or might not) be related to FPE.

FWIW, I saw yesterday that NumPy is enabling FPE on Apple silicon (numpy/numpy#24060)...

@BrushXue
Copy link
Owner

BrushXue commented Aug 2, 2023

OpenFOAM uses its own implementation
https://develop.openfoam.com/Development/openfoam/-/blob/master/src/OSspecific/POSIX/signals/feexceptErsatz.H
Obviously, the updated code from Tatsuya Shimizu never worked.

@gerlero
Copy link
Contributor Author

gerlero commented Aug 2, 2023

Thanks. Might be worth a try to disable the __arm64__ special casing and see if that changes anything...

@gerlero
Copy link
Contributor Author

gerlero commented Aug 2, 2023

Thanks. Might be worth a try to disable the __arm64__ special casing and see if that changes anything...

Well... that didn't work. OpenFOAM fails to compile on arm64 without the special cases in that file.

@BrushXue
Copy link
Owner

BrushXue commented Aug 2, 2023

According to https://xem.github.io/minix86/manual/intel-x86-and-64-manual-vol1/o_7281d5ea06a5b67a-197.html , https://help.totalview.io/current/PDFs/TotalView_Reference_Guide.pdf and fenv.h
For x86:

Flag Value __control __mxcsr
FE_INEXACT 0x0020 0x20 0x1000
FE_UNDERFLOW 0x0010 0x10 0x0800
FE_OVERFLOW 0x0008 0x08 0x0400
FE_DIVBYZERO 0x0004 0x04 0x0200
FE_DENORMALOPERAND 0x0002 0x02 0x0100
FE_INVALID 0x0001 0x01 0x0080

which corresponds to these lines:

    fenv.__control &= ~new_excepts;
    fenv.__mxcsr   &= ~(new_excepts << 7);

https://develop.openfoam.com/Development/openfoam/-/blob/master/src/OSspecific/POSIX/signals/feexceptErsatz.H#L52-53

For arm64:

Flag Value __fpsr __fpcr
FE_INEXACT 0x0010 0x10 0x1000
FE_UNDERFLOW 0x0008 0x8 0x800
FE_OVERFLOW 0x0004 0x4 0x400
FE_DIVBYZERO 0x0002 0x2 0x200
FE_FLUSHTOZERO 0x0080 0x80 0x8000
FE_INVALID 0x0001 0x1 0x100

Assuming that Apple is strictly following this, you need to shift 8 bits instead of 7. Therefore it should be (updated)
Line 42:

    old_excepts = fenv.__fpsr & FE_ALL_EXCEPT;

Line 49-50:

    fenv.__fpsr &= ~new_excepts;
    fenv.__fpcr &= ~(new_excepts << 8);

Line 72:

    old_excepts = fenv.__fpsr & FE_ALL_EXCEPT;

Line 79-80:

    fenv.__fpsr |= new_excepts;
    fenv.__fpcr |= new_excepts << 8;

Could you test it with the new Clang? (I don't have Ventura installed on my laptop)

@gerlero
Copy link
Contributor Author

gerlero commented Aug 2, 2023

Could you test it with the new Clang? (I don't have Ventura installed on my laptop)

Tested it; unfortunately it doesn't fix it (backwardFacingStep2D case gets stuck the same way).

@BrushXue
Copy link
Owner

BrushXue commented Aug 2, 2023

I wonder if it can successfully trigger SIGFPE on M1 because that has been on my to-do-list for 2 years.

@gerlero
Copy link
Contributor Author

gerlero commented Aug 2, 2023

IIRC I tried with the latest Clang about a month ago, but couldn't get a float division by zero to trap. NumPy does raise a warning in such a case, but IDK if it's relying on FPE to catch that.

@BrushXue
Copy link
Owner

BrushXue commented Aug 3, 2023

Could you try this test code (make the same changes)?
mrklein/openfoam-os-x#68 (comment)

@gerlero
Copy link
Contributor Author

gerlero commented Aug 3, 2023

Could you try this test code (make the same changes)?
mrklein/openfoam-os-x#68 (comment)

No errors whether I leave the 7s or I change them to 8s.

Output is:

1
-inf
-inf
4
nan
-inf

@BrushXue
Copy link
Owner

BrushXue commented Aug 3, 2023

I installed Sonoma beta on my M1 MacBook and did some testing:
For the FPE code, you can test the code in my latest comment. Now OpenFOAM will trigger SIGILL instead of SIGFPE. It's definitely a huge improvement, but still not there yet.
For this tutorial case, I can confirm the problem is in #includeFunc "sample" while simpleFoam uses 100% CPU. If you check other cases using sampling like tutorials/incompressible/pimpleFoam/LES/surfaceMountedCube, the same issue can be reproduced. So I suspect the problem lies in src/sampling which have triggered a bug in newer versions of Clang.

@BrushXue
Copy link
Owner

BrushXue commented Aug 3, 2023

FWIW, I saw yesterday that NumPy is enabling FPE on Apple silicon (numpy/numpy#24060)...

Do you have a minimum example to trigger SIGFPE on numpy? I have 1.25.2 from homebrew but it's not trapping signals.

>>> import numpy as np
>>> np.sqrt(-1)
<stdin>:1: RuntimeWarning: invalid value encountered in sqrt
nan

@gerlero
Copy link
Contributor Author

gerlero commented Aug 3, 2023

I actually meant that RuntimeWarning (which can be controlled with this https://numpy.org/doc/stable/reference/generated/numpy.errstate.html).

Maybe it's is just manually checking the inputs/outputs, but I can see it being implemented by catching floating-point signals from the hardware. Still, as I said before I have no idea how NumPy implements it...

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

After testing every items in the sample file, the problematic set is

    x_by_h_m04
    {
        type    face;
        axis    y;
        start   (-0.0508 -0.0127 0);
        end     (-0.0508 0.1016 0);
        nPoints 100;
    }

If you comment this block, it will run normally. So it's very likely a numerical bug.

I'm doing a full debug build, hopefully LLDB will tell me what happened in libsampling

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

Here's the lldb log

Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00000001000465c0 simpleFoam`Foam::min(s1=0.065987628869999995, s2=0.067035051550000002) at doubleFloat.H:55:1
   52  	}
   53
   54
-> 55  	MAXMINPOW(double, double, double)
   56  	MAXMINPOW(double, double, float)
   57  	MAXMINPOW(double, float, double)
   58  	MAXMINPOW(double, double, int)
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (double) $0 = 0.065987628869999995

    frame #0: 0x000000010304a84c libfiniteVolume.dylib`Foam::minOp<double>::operator()(this=0x000000016fdf836f, x=0x000000016fdf8650, y=0x00000001180e3480) const at ops.H:218:1
   215 	Op(cmptPow, cmptPow(x, y))
   216 	Op(cmptDivide, cmptDivide(x, y))
   217 	Op(stabilise, stabilise(x, y))
-> 218 	Op(min, min(x, y))
   219 	Op(max, max(x, y))
   220 	Op(minMagSqr, (magSqr(x) <= magSqr(y) ? x : y))
   221 	Op(maxMagSqr, (magSqr(x) >= magSqr(y) ? x : y))
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (double) $1 = 0.065987628869999995

    frame #0: 0x000000010304a7fc libfiniteVolume.dylib`void Foam::VectorSpaceOps<(unsigned char)3, (unsigned char)0>::op<Foam::Vector<double>, Foam::VectorSpace<Foam::Vector<double>, double, (unsigned char)3>, Foam::minOp<double>>(vs=0x000000016fdf83f8, vs1=0x000000016fdf8650, vs2=0x00000001180e3480, o=minOp<double> @ 0x000000016fdf836f) at VectorSpaceOps.H:87:9
   84  	    template<class V, class V1, class Op>
   85  	    static inline void op(V& vs, const V1& vs1, const V1& vs2, Op o)
   86  	    {
-> 87  	        vs.v_[I] = o(vs1.v_[I], vs2.v_[I]);
   88  	        VectorSpaceOps<N, I+1>::op(vs, vs1, vs2, o);
   89  	    }
   90  	};
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x000000010304a7c8 libfiniteVolume.dylib`Foam::Vector<double> Foam::min<Foam::Vector<double>, double, (unsigned char)3>(vs1=0x000000016fdf8650, vs2=0x00000001180e3480) at VectorSpaceI.H:666:5
   663 	{
   664 	    Form v;
   665 	    VectorSpaceOps<Ncmpts,0>::op(v, vs1, vs2, minOp<Cmpt>());
-> 666 	    return v;
   667 	}
   668
   669
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x00000001038609d8 libfiniteVolume.dylib`Foam::boundBox::add(this=0x000000016fdf8650, pt=0x00000001180e3480) at boundBoxI.H:339:12
   336
   337 	inline void Foam::boundBox::add(const point& pt)
   338 	{
-> 339 	    min_ = ::Foam::min(min_, pt);
   340 	    max_ = ::Foam::max(max_, pt);
   341 	}
   342
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000107441264 libOpenFOAM.dylib`void Foam::boundBox::add<Foam::UList<int>>(this=0x000000016fdf8650, points=0x000000012380acc8, indices=0x00000001281093f0) at boundBoxTemplates.C:113:9
   110 	        if (pointi >= 0 && pointi < len)
   111 	        {
   112 	            add(points[pointi]);
-> 113 	        }
   114 	    }
   115 	}
   116
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x000000010744116c libOpenFOAM.dylib`Foam::boundBox::boundBox(this=0x000000016fdf8650, points=0x000000012380acc8, indices=0x00000001281093f0, doReduce=false) at boundBox.C:120:9
   117 	{
   118 	    add(points, indices);
   119
-> 120 	    if (doReduce)
   121 	    {
   122 	        reduce();
   123 	    }
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000107443a10 libOpenFOAM.dylib`Foam::treeBoundBox::treeBoundBox(this=0x000000016fdf8650, points=0x000000012380acc8, indices=0x00000001281093f0) at treeBoundBox.C:84:9
   81  	:
   82  	    boundBox(points, indices, false)
   83  	{
-> 84  	    if (points.empty() || indices.empty())
   85  	    {
   86  	        WarningInFunction
   87  	            << "No bounding box for zero-sized pointField" << nl;
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000107443aa4 libOpenFOAM.dylib`Foam::treeBoundBox::treeBoundBox(this=0x000000016fdf8650, points=0x000000012380acc8, indices=0x00000001281093f0) at treeBoundBox.C:83:1
   80  	)
   81  	:
   82  	    boundBox(points, indices, false)
-> 83  	{
   84  	    if (points.empty() || indices.empty())
   85  	    {
   86  	        WarningInFunction
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101cd119c libmeshTools.dylib`Foam::treeDataFace::getBounds(this=0x0000600003be6200, index=16915) const at treeDataFace.C:161:5
   158 	Foam::treeDataFace::getBounds(const label index) const
   159 	{
   160 	    const label facei = objectIndex(index);
-> 161 	    return treeBoundBox(mesh_.points(), mesh_.faces()[facei]);
   162 	}
   163
   164
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101cd0e38 libmeshTools.dylib`Foam::treeDataFace::overlaps(this=0x0000600003be6200, index=16915, searchBox=0x000000016fdf8710) const at treeDataFace.C:545:29
   542 	    (
   543 	        cacheBb_
   544 	      ? !searchBox.overlaps(bbs_[index])
-> 545 	      : !searchBox.overlaps(getBounds(index))
   546 	    )
   547 	    {
   548 	        return false;
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (bool) $2 = false

    frame #0: 0x0000000101c0e2dc libmeshTools.dylib`Foam::indexedOctree<Foam::treeDataFace>::divide(this=0x0000600003be6200, indices=0x00000001183411b0, bb=0x000000016fdf8998, dividedIndices=0x000000016fdf8838) const at indexedOctree.C:56:25
   53
   54  	        for (const label index : indices)
   55  	        {
-> 56  	            if (shapes_.overlaps(index, subBbs))
   57  	            {
   58  	                contains.push_back(index);
   59  	            }
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101c0d920 libmeshTools.dylib`Foam::indexedOctree<Foam::treeDataFace>::divide(this=0x0000600003be6200, bb=0x000000016fdf8998, contents=0x000000016fdf8db0, contentIndex=282) const at indexedOctree.C:96:5
   93  	    const labelList& indices = contents[contentIndex];
   94
   95  	    FixedList<labelList, 8> dividedIndices;
-> 96  	    divide(indices, bb, dividedIndices);
   97
   98  	    node nod;
   99  	    nod.bb_ = bb;
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101c0dc14 libmeshTools.dylib`Foam::indexedOctree<Foam::treeDataFace>::splitNodes(this=0x0000600003be6200, minSize=10, nodes=0x000000016fdf8dc8, contents=0x000000016fdf8db0) const at indexedOctree.C:172:39
   169 	                    const treeBoundBox bb(nod.bb_.subBbox(octant));
   170
   171 	                    node subNode(divide(bb, contents, contentI));
-> 172 	                    subNode.parent_ = nodeI;
   173 	                    label sz = nodes.size();
   174 	                    nodes.push_back(subNode);
   175 	                    nodes[nodeI].subNodes_[octant] = nodePlusOctant(sz, octant);
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101c0d148 libmeshTools.dylib`Foam::indexedOctree<Foam::treeDataFace>::indexedOctree(this=0x0000600003be6200, shapes=0x000000016fdf8ef8, bb=0x0000600001b52be0, maxLevels=8, maxLeafRatio=10, maxDuplicity=3) at indexedOctree.C:2229:9
   2226
   2227	        // Split nodes with more than maxLeafRatio elements
   2228	        label nOldNodes = nodes.size();
-> 2229	        splitNodes
   2230	        (
   2231	            label(maxLeafRatio),
   2232	            nodes,
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101bfa3dc libmeshTools.dylib`Foam::indexedOctree<Foam::treeDataFace>::indexedOctree(this=0x0000600003be6200, shapes=0x000000016fdf8ef8, bb=0x0000600001b52be0, maxLevels=8, maxLeafRatio=10, maxDuplicity=3) at indexedOctree.C:2156:1
   2153	    nodes_(),
   2154	    contents_(),
   2155	    nodeTypes_()
-> 2156	{
   2157	    int oldMemSize = 0;
   2158	    if (debug)
   2159	    {
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101c370dc libmeshTools.dylib`Foam::meshSearch::boundaryTree(this=0x0000000123107680) const at meshSearch.C:629:17
   626
   627 	        boundaryTreePtr_.reset
   628 	        (
-> 629 	            new indexedOctree<treeDataFace>
   630 	            (
   631 	                treeDataFace(mesh_, bndFaces),  // Boundary faces
   632
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000101c37ef8 libmeshTools.dylib`Foam::meshSearch::intersections(this=0x0000000123107680, pStart=0x000000016fdf9268, pEnd=0x000000016fdf9240) const at meshSearch.C:874:31
   871 	) const
   872 	{
   873 	    DynamicList<pointIndexHit> hits;
-> 874 	    findUniqueIntersectOp iop(boundaryTree(), hits);
   875
   876 	    while (true)
   877 	    {
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 14628 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (Foam::List<Foam::PointIndexHit<Foam::Vector<double> > >) $3 = {
  Foam::UList<Foam::PointIndexHit<Foam::Vector<double> > > = {
    size_ = 1876922360
    v_ = 0x000000016fdf8ff8
  }
}

    frame #0: 0x00000001054182bc libsampling.dylib`Foam::faceOnlySet::calcSamples(this=0x00006000020e1380, samplingPts=0x000000016fdf9590, samplingCells=0x000000016fdf9578, samplingFaces=0x000000016fdf9560, samplingSegments=0x000000016fdf9548, samplingCurveDist=0x000000016fdf9530) const at faceOnlySet.C:128:48
   125 	    passiveParticleCloud particleCloud(mesh());
   126
   127 	    // Get all boundary intersections
-> 128 	    List<pointIndexHit> bHits = searchEngine().intersections
   129 	    (
   130 	        start_ - smallVec,
   131 	        end_ + smallVec
Target 0: (simpleFoam) stopped.

@gerlero
Copy link
Contributor Author

gerlero commented Aug 4, 2023

Nice. I guess one of those functions is in an infinite loop?

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

Maybe the new Clang failed to expand the macro defined in src/OpenFOAM/primitives/Scalar/doubleFloat.H?

@gerlero
Copy link
Contributor Author

gerlero commented Aug 4, 2023

Maybe the new Clang failed to expand the macro defined in src/OpenFOAM/primitives/Scalar/doubleFloat.H?

Not sure.. isn't this line showing the expansion?

    frame #0: 0x00000001000465c0 simpleFoam`Foam::min(s1=0.065987628869999995, s2=0.067035051550000002) at doubleFloat.H:55:1

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

Forget about it. I ran lldb again and this time it gives an another totally different result. Now I'm totally lost.

@gerlero
Copy link
Contributor Author

gerlero commented Aug 4, 2023

How different? If it's an infinite loop then the function that "contains" that loop must appear in the trace no matter when the snapshot is taken, right?

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

I feel it's in a huge infinite loop across multiple functions. If it's cause by a compiler bug, this method may not work at all.

Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x000000010001e040 simpleFoam`Foam::UList<int>::checkIndex(this=0x0000000120009810, i=2) const at UListI.H:173:1
   170 	            << size_ << "]\n"
   171 	            << abort(FatalError);
   172 	    }
-> 173 	}
   174
   175
   176 	template<class T>
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000100045cec simpleFoam`Foam::UList<int>::operator[](this=0x0000000120009810, i=2) const at UListI.H:311:5
   308 	inline const T& Foam::UList<T>::operator[](const label i) const
   309 	{
   310 	    #ifdef FULLDEBUG
-> 311 	    checkIndex(i);
   312 	    #endif
   313 	    return v_[i];
   314 	}
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000102d899dc libfiniteVolume.dylib`Foam::UList<int>::fcValue(this=0x0000000120009810, i=1) const at UListI.H:75:5
   72  	template<class T>
   73  	inline const T& Foam::UList<T>::fcValue(const label i) const
   74  	{
-> 75  	    return this->operator[](this->fcIndex(i));
   76  	}
   77
   78
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000103131cb0 libfiniteVolume.dylib`Foam::face::nextLabel(this=0x0000000120009810, i=1) const at faceI.H:192:23
   189
   190 	inline Foam::label Foam::face::nextLabel(const label i) const
   191 	{
-> 192 	    return labelList::fcValue(i);
   193 	}
   194
   195
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (Foam::label) $0 = 6355

    frame #0: 0x0000000103130c88 libfiniteVolume.dylib`Foam::face::edge(this=0x0000000120009810, edgei=1) const at faceI.H:142:41
   139
   140 	inline Foam::edge Foam::face::edge(const label edgei) const
   141 	{
-> 142 	    return Foam::edge(thisLabel(edgei), nextLabel(edgei));
   143 	}
   144
   145
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
Return value: (Foam::edge) $1 = {
  Foam::labelPair = {
    Foam::FixedList<int, 2> = {
      v_ = ([0] = 1876921420, [1] = 1)
    }
  }
}

    frame #0: 0x0000000100846940 liblagrangian.dylib`Foam::particle::changeFace(this=0x000000016fdf9160, tetTriI=3) at particle.C:299:47
   296 	        (
   297 	            ;
   298 	            edgeI < newFace.size()
-> 299 	         && edge::compare(sharedEdge, newFace.edge(edgeI)) != edgeComp;
   300 	            ++ edgeI
   301 	        );
   302
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000100846458 liblagrangian.dylib`Foam::particle::changeTet(this=0x000000016fdf9160, tetTriI=3) at particle.C:217:13
   214 	            if (tetPti_ == firstTetPtI)
   215 	            {
   216 	                changeFace(tetTriI);
-> 217 	            }
   218 	            else
   219 	            {
   220 	                reflect();
Target 0: (simpleFoam) stopped.
(lldb) finish
Process 20383 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step out
    frame #0: 0x0000000100848b60 liblagrangian.dylib`Foam::particle::trackToFace(this=0x000000016fdf9160, displacement=0x000000016fdf8f70, fraction=0) at particle.C:686:13
   683 	        else
   684 	        {
   685 	            // Move to the next tet and continue the track
-> 686 	            changeTet(tetTriI);
   687 	        }
   688 	    }
   689
Target 0: (simpleFoam) stopped.

@gerlero
Copy link
Contributor Author

gerlero commented Aug 4, 2023

If it's cause by a compiler bug, this method may not work at all.

That's an interesting point.

I feel it's in a huge infinite loop across multiple functions.

Yeah, glancing at both stack traces (and assuming any compiler bug that may exist is not that bad) I see no functions that appear in both. Is there a way to print it with maximum depth (i.e. down to main())?

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

I tested llvm@16 from homebrew and it failed at the same place. So it's not Apple's fault, but an upstream problem (the bug could be introduced between 14.0.0 and 14.0.3).

llvm@13 is working and could fit your current workflow: it's the last working formula that is still available. Since the bug is still in llvm@16, I would expect the same thing happens in at least macOS 15. So llvm@13 might be a better long-term solution, until the bug is fixed by llvm upstream.

@BrushXue
Copy link
Owner

BrushXue commented Aug 4, 2023

You'll need to do

brew install llvm@13 flex

and

export PATH="/opt/homebrew/opt/llvm@13/bin:/opt/homebrew/opt/flex/bin:$PATH"
export FOAM_EXTRA_CXXFLAGS="-I/opt/homebrew/opt/flex/include"
export FOAM_EXTRA_LDFLAGS="-L/opt/homebrew/opt/flex/lib -L/opt/homebrew/opt/llvm@13/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm@13/lib/c++"

plus you may want to suppress compiler warnings with -fno-trapping-math

@BrushXue
Copy link
Owner

BrushXue commented Aug 5, 2023

Problem solved! I tested it on Xcode 15 beta and didn't trigger the infinite loop.
Clang changed its default floating point number behavior: https://releases.llvm.org/14.0.0/tools/clang/docs/ReleaseNotes.html#floating-point-support-in-clang. FMA is a very dangerous thing in scientific computing(unless you really know what you are doing) so I don't understand why they changed the default behavior. For some reason, Apple didn't pull this upstream change until 14.0.3 (my guess, because Apple clang 14.0.0 is OK.)
So please add this flag:-ffp-contract=off and test again. If it works we can close this issue.

Additional reading: https://www.kdab.com/fma-woes/

@gerlero
Copy link
Contributor Author

gerlero commented Aug 6, 2023

Yes, that definitely does it! Excellent find; thanks!

If you can open a PR at https://github.com/gerlero/openfoam-app, I'd prefer it so that you get the full credit for the fix. FWIW, adding these lines:

sed -i '' 's|-ftrapping-math|-ftrapping-math -ffp-contract=off|' wmake/rules/darwin64Clang/c
sed -i '' 's|-ftrapping-math|-ftrapping-math -ffp-contract=off|' wmake/rules/darwin64Clang/c++

to configure.sh did the trick for me. Let me know if you're able to do it.

I guess this workaround should be upstreamed too; what do you think? If you agree, I'd also prefer that you were the one to open the issue there so that you get credited for that as well.

@BrushXue
Copy link
Owner

BrushXue commented Aug 6, 2023

I’ve already opened an issue there.
https://develop.openfoam.com/Development/openfoam/-/issues/2958

@BrushXue
Copy link
Owner

BrushXue commented Aug 6, 2023

Looks like it’s a well-know issue in computational geometry.
WebAssembly/relaxed-simd#5 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants