Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests sporadically timing out on 1.11 #329

Closed
christiangnrd opened this issue Apr 11, 2024 · 10 comments
Closed

Tests sporadically timing out on 1.11 #329

christiangnrd opened this issue Apr 11, 2024 · 10 comments

Comments

@christiangnrd
Copy link
Contributor

christiangnrd commented Apr 11, 2024 •
edited
Loading

I was able to reproduce on main. For a CI run that fails in this way, see https://buildkite.com/julialang/metal-dot-jl/builds/935#018ec983-4e76-42ab-bc96-fb795c07e721.
@christiangnrd christiangnrd mentioned this issue Apr 11, 2024
Merged
@maleadt maleadt mentioned this issue Apr 11, 2024
Merged
@maleadt
Copy link
Member

maleadt commented Apr 11, 2024

I can't reproduce this locally. Let's see if #330 reveals anything.

@christiangnrd
Copy link
Contributor Author

Local stacktraces from main (not #321).

Stacktrace 1 
┌ Info: System information:
│ macOS 14.4.1, Darwin 23.4.0
│ 
│ Toolchain:
│ - Julia: 1.11.0-beta1
│ - LLVM: 16.0.6
│ 
│ Julia packages: 
│ - Metal.jl: 1.1.0
│ - LLVMDowngrader_jll: 0.1.0+1
│ 
│ 1 device:
└ - Apple M2 Max (64.000 KiB allocated)
[ Info: Running 8 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the JULIA_CPU_THREADS environment variable.
                                                  |          | ---------------- CPU ---------------- |
Test                                     (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
metallib                                      (8) |     1.90 |   0.03 |  1.6 |     372.75 |   587.86 |
pool                                          (8) |     1.04 |   0.03 |  2.6 |     295.86 |   637.12 |
      From worker 8:    Starting recording with the Blank template and GPU, Time Profiler, Metal Application, Metal GPU Counters, Metal Resource Events, os_signpost Instruments. Attaching to: julia (65430). 
      From worker 8:    Ctrl-C to stop the recording
      From worker 8:    Stopping recording...
      From worker 10:   2024-04-11 08:39:47.998 julia[65436:466723] Metal GPU Frame Capture Enabled
      From worker 8:    Recording completed. Saving output file...
      From worker 8:    Output file saved as: julia_1.trace
      From worker 8:    [ Info: System trace saved to /private/var/folders/4g/lnkpkf3s4rxd_wbl8vwnqs4r0000gn/T/jl_w0cddG/julia_1.trace; open the resulting trace in Instruments
profiling                                     (8) |     9.08 |   0.00 |  0.0 |      88.05 |   639.23 |
      From worker 10:   [ Info: GPU frame capture saved to /private/var/folders/4g/lnkpkf3s4rxd_wbl8vwnqs4r0000gn/T/jl_wnbFKJ/julia_1.gputrace; open the resulting trace in Xcode
      From worker 10:   [ Info: GPU frame capture saved to /private/var/folders/4g/lnkpkf3s4rxd_wbl8vwnqs4r0000gn/T/jl_wnbFKJ/julia_2.gputrace; open the resulting trace in Xcode
capturing                                    (10) |    16.27 |   0.33 |  2.0 |    1868.73 |   778.45 |
execution                                     (5) |    24.84 |   0.44 |  1.8 |    3091.01 |   865.25 |
random                                        (8) |    18.19 |   0.27 |  1.5 |    2212.92 |   845.95 |
mps                                           (9) |    32.21 |   0.64 |  2.0 |    5104.27 |   888.62 |
array                                         (2) |    36.68 |   0.58 |  1.6 |    4336.47 |   906.94 |
kernelabstractions                            (6) |    38.52 |   0.81 |  2.1 |    5793.94 |  1105.39 |
gpuarrays/indexing scalar                    (11) |    22.14 |   0.40 |  1.8 |    2964.13 |   865.27 |
gpuarrays/math/power                          (2) |    19.72 |   0.40 |  2.0 |    4588.44 |  1166.78 |
examples                                      (4) |    63.68 |   0.02 |  0.0 |     116.91 |   550.05 |
gpuarrays/indexing find                      (11) |    15.39 |   0.51 |  3.3 |    5869.89 |  1110.92 |
gpuarrays/linalg/mul!/vector-matrix           (6) |    27.95 |   0.39 |  1.4 |    5132.56 |  1433.64 |
gpuarrays/reductions/any all count           (11) |     8.14 |   0.13 |  1.6 |    1972.99 |  1254.25 |
gpuarrays/interface                           (4) |    15.49 |   0.30 |  2.0 |    1914.47 |   784.20 |
gpuarrays/uniformscaling                     (11) |     4.99 |   0.02 |  0.5 |     604.77 |  1356.72 |
gpuarrays/indexing multidimensional           (2) |    32.46 |   0.37 |  1.1 |    4751.40 |  1398.31 |
gpuarrays/reductions/reducedim!               (8) |    58.70 |   1.22 |  2.1 |   12908.88 |  1462.27 |
gpuarrays/math/intrinsics                     (2) |     0.85 |   0.00 |  0.0 |     104.68 |  1398.62 |
gpuarrays/linalg                              (9) |    62.26 |   1.24 |  2.0 |   11241.87 |  1587.69 |
gpuarrays/reductions/mapreducedim!_large      (4) |    33.38 |   1.15 |  3.4 |    9702.13 |  1502.05 |
gpuarrays/linalg/norm                         (8) |    25.10 |   0.59 |  2.4 |    6982.03 |  1724.77 |
gpuarrays/statistics                          (2) |    24.79 |   0.37 |  1.5 |    4487.90 |  1533.23 |
gpuarrays/random                              (8) |     6.23 |   0.03 |  0.5 |     767.07 |  1792.05 |
gpuarrays/linalg/mul!/matrix-matrix          (11) |    45.78 |   0.60 |  1.3 |    7299.27 |  1534.56 |
gpuarrays/base                                (2) |    15.04 |   0.47 |  3.2 |    3328.70 |  1814.16 |
gpuarrays/constructors                        (4) |    20.25 |   0.18 |  0.9 |    2402.72 |  1587.59 |
gpuarrays/reductions/== isequal               (8) |    26.24 |   0.64 |  2.4 |    7572.20 |  2060.88 |
gpuarrays/reductions/minimum maximum extrema  (6) |    98.36 |   2.51 |  2.6 |   24604.65 |  2190.00 |
gpuarrays/reductions/mapreduce                (9) |    79.39 |   1.95 |  2.5 |   19352.78 |  2148.30 |
┌ Warning: Test timed out: metal
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:247
      From worker 7:
      From worker 7:    ======================================================================================
      From worker 7:    Information request received. A stacktrace will print followed by a 1.0 second profile
      From worker 7:    ======================================================================================
      From worker 7:
      From worker 7:    signal (29): Information request: 29
      From worker 7:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 7:    unknown function (ip: 0x0)
      From worker 7:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 7:    unknown function (ip: 0x0)
      From worker 7:
      From worker 7:    ==============================================================
      From worker 7:    Profile collected. A report will print at the next yield point
      From worker 7:    ==============================================================
      From worker 7:
gpuarrays/reductions/reduce                   (4) |    52.68 |   1.36 |  2.6 |   14353.29 |  1835.11 |
┌ Warning: Couldn't kill worker 7, killing process 65429 forcefully
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:256
      From worker 7:
      From worker 7:    [65429] signal 15: Terminated: 15
      From worker 7:    in expression starting at /Users/christian/.julia/dev/Metal/test/metal.jl:1
      From worker 7:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 7:    unknown function (ip: 0x0)
      From worker 7:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 7:    unknown function (ip: 0x0)
      From worker 7:    Allocations: 39690602 (Pool: 39689453; Big: 1149); GC: 23
metal                                         (7) |         failed at 2024-04-11T08:42:53.985
┌ Warning: Test timed out: device/intrinsics
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:247
      From worker 5:
      From worker 5:    ======================================================================================
      From worker 5:    Information request received. A stacktrace will print followed by a 1.0 second profile
      From worker 5:    ======================================================================================
      From worker 5:
      From worker 5:    signal (29): Information request: 29
      From worker 5:    jl_gc_state_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:337 [inlined]
      From worker 5:    jl_gc_state_save_and_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:343 [inlined]
      From worker 5:    jl_delete_thread at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/threading.c:446
      From worker 5:    _pthread_tsd_cleanup at /usr/lib/system/libsystem_pthread.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
gpuarrays/reductions/mapreducedim!            (2) |    73.66 |   1.68 |  2.3 |   16823.66 |  2413.50 |
      From worker 5:
      From worker 5:    ==============================================================
      From worker 5:    Profile collected. A report will print at the next yield point
      From worker 5:    ==============================================================
      From worker 5:
gpuarrays/reductions/sum prod                 (8) |    68.08 |   1.66 |  2.4 |   16160.46 |  2900.47 |
┌ Warning: Couldn't kill worker 5, killing process 65427 forcefully
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:256
      From worker 5:
      From worker 5:    [65427] signal 15: Terminated: 15
      From worker 5:    in expression starting at /Users/christian/.julia/dev/Metal/test/device/intrinsics.jl:412
      From worker 5:    jl_gc_state_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:337 [inlined]
      From worker 5:    jl_gc_state_save_and_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:343 [inlined]
      From worker 5:    jl_delete_thread at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/threading.c:446
      From worker 5:    _pthread_tsd_cleanup at /usr/lib/system/libsystem_pthread.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    Allocations: 115583005 (Pool: 115579189; Big: 3816); GC: 52
device/intrinsics                             (5) |         failed at 2024-04-11T08:43:20.314
gpuarrays/broadcasting                       (11) |    97.77 |   1.66 |  1.7 |   17158.54 |  2184.98 |
Testing finished in 3 minutes, 47 seconds, 58 milliseconds
metal: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(7)
device/intrinsics: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)

Test Summary:                                  | Pass  Error  Broken  Total  Time
  Overall                                      | 8284      2       9   8295      
    metallib                                   |   20                    20      
    pool                                       |    5                     5      
    profiling                                  |    1                     1      
    capturing                                  |   24                    24      
    execution                                  |   35                    35      
    random                                     |   57                    57      
    mps                                        |   39                    39      
    array                                      |  303                   303      
    kernelabstractions                         | 2165              8   2173      
    gpuarrays/indexing scalar                  |  399                   399      
    gpuarrays/math/power                       |   60                    60      
    examples                                   |    3                     3      
    gpuarrays/indexing find                    |   45                    45      
    gpuarrays/linalg/mul!/vector-matrix        |  140                   140      
    gpuarrays/reductions/any all count         |  101                   101      
    gpuarrays/interface                        |    7                     7      
    gpuarrays/uniformscaling                   |   56                    56      
    gpuarrays/indexing multidimensional        |   88              1     89      
    gpuarrays/reductions/reducedim!            |  160                   160      
    gpuarrays/math/intrinsics                  |   10                    10      
    gpuarrays/linalg                           |  325                   325      
    gpuarrays/reductions/mapreducedim!_large   |   40                    40      
    gpuarrays/linalg/norm                      |  264                   264      
    gpuarrays/statistics                       |   52                    52      
    gpuarrays/random                           |   52                    52      
    gpuarrays/linalg/mul!/matrix-matrix        |  360                   360      
    gpuarrays/base                             |   93                    93      
    gpuarrays/constructors                     |  832                   832      
    gpuarrays/reductions/== isequal            |  248                   248      
    gpuarrays/reductions/minimum maximum extrema |  555                   555      
    gpuarrays/reductions/mapreduce             |  330                   330      
    gpuarrays/reductions/reduce                |  220                   220      
    metal                                      |           1              1      
    gpuarrays/reductions/mapreducedim!         |  260                   260      
    gpuarrays/reductions/sum prod              |  636                   636      
    device/intrinsics                          |           1              1      
    gpuarrays/broadcasting                     |  299                   299      
    FAILURE

Error in testset metal:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(7)
Error in testset device/intrinsics:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)
ERROR: LoadError: Test run finished with errors
in expression starting at /Users/christian/.julia/dev/Metal/test/runtests.jl:410
ERROR: Package Metal errored during testing
Stacktrace 2 
     Testing Running tests...
┌ Info: System information:
│ macOS 14.4.1, Darwin 23.4.0
│ 
│ Toolchain:
│ - Julia: 1.11.0-beta1
│ - LLVM: 16.0.6
│ 
│ Julia packages: 
│ - Metal.jl: 1.1.0
│ - LLVMDowngrader_jll: 0.1.0+1
│ 
│ 1 device:
└ - Apple M2 Max (64.000 KiB allocated)
[ Info: Running 8 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the JULIA_CPU_THREADS environment variable.
                                                  |          | ---------------- CPU ---------------- |
Test                                     (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
metallib                                      (8) |     1.88 |   0.03 |  1.5 |     372.75 |   593.38 |
pool                                          (8) |     1.02 |   0.03 |  2.5 |     295.86 |   643.22 |
      From worker 8:    Starting recording with the Blank template and GPU, Time Profiler, Metal Application, Metal GPU Counters, Metal Resource Events, os_signpost Instruments. Attaching to: julia (70273). 
      From worker 8:    Ctrl-C to stop the recording
metal                                         (7) |     5.65 |   0.32 |  5.6 |    1616.65 |   870.56 |
      From worker 8:    Stopping recording...
      From worker 10:   2024-04-11 09:01:45.794 julia[70277:481336] Metal GPU Frame Capture Enabled
      From worker 8:    Recording completed. Saving output file...
      From worker 8:    Output file saved as: julia_1.trace
      From worker 8:    [ Info: System trace saved to /private/var/folders/4g/lnkpkf3s4rxd_wbl8vwnqs4r0000gn/T/jl_iPyeJV/julia_1.trace; open the resulting trace in Instruments
profiling                                     (8) |     8.91 |   0.00 |  0.0 |      88.05 |   653.14 |
      From worker 10:   [ Info: GPU frame capture saved to /private/var/folders/4g/lnkpkf3s4rxd_wbl8vwnqs4r0000gn/T/jl_7589eV/julia_1.gputrace; open the resulting trace in Xcode
      From worker 10:   [ Info: GPU frame capture saved to /private/var/folders/4g/lnkpkf3s4rxd_wbl8vwnqs4r0000gn/T/jl_7589eV/julia_2.gputrace; open the resulting trace in Xcode
capturing                                    (10) |    16.46 |   0.34 |  2.1 |    1868.73 |   816.19 |
random                                        (7) |    16.53 |   0.08 |  0.5 |    1142.14 |  1022.84 |
execution                                     (5) |    25.15 |   0.45 |  1.8 |    3093.07 |   917.36 |
gpuarrays/indexing scalar                     (7) |     7.03 |   0.08 |  1.1 |    1106.37 |  1073.12 |
mps                                           (9) |    32.49 |   0.65 |  2.0 |    5103.96 |   973.47 |
array                                         (2) |    36.70 |   0.54 |  1.5 |    4336.11 |  1015.17 |
kernelabstractions                            (6) |    38.65 |   0.79 |  2.0 |    5793.87 |  1203.16 |
device/intrinsics                             (8) |    36.20 |   0.57 |  1.6 |    4662.11 |  1071.52 |
gpuarrays/interface                           (8) |     0.93 |   0.00 |  0.0 |     239.40 |  1106.36 |
gpuarrays/math/power                          (7) |    19.93 |   0.39 |  2.0 |    4579.54 |  1231.45 |
gpuarrays/indexing find                       (2) |    15.33 |   0.56 |  3.6 |    5746.46 |  1183.09 |
gpuarrays/uniformscaling                      (2) |     5.90 |   0.04 |  0.7 |     670.73 |  1320.20 |
gpuarrays/reductions/any all count            (8) |     9.06 |   0.15 |  1.6 |    2154.03 |  1234.17 |
gpuarrays/linalg/mul!/vector-matrix           (9) |    30.10 |   0.43 |  1.4 |    5138.28 |  1259.28 |
gpuarrays/math/intrinsics                     (9) |     0.90 |   0.00 |  0.0 |     107.14 |  1271.69 |
examples                                      (4) |    64.76 |   0.01 |  0.0 |     114.85 |   583.16 |
gpuarrays/indexing multidimensional           (6) |    34.70 |   0.38 |  1.1 |    4985.34 |  1461.42 |
gpuarrays/reductions/reducedim!               (5) |    61.51 |   1.30 |  2.1 |   13015.03 |  1537.92 |
gpuarrays/reductions/mapreducedim!_large      (2) |    30.99 |   0.99 |  3.2 |    9118.87 |  2032.45 |
gpuarrays/linalg/norm                         (9) |    26.42 |   0.55 |  2.1 |    6655.84 |  1639.03 |
gpuarrays/random                              (2) |     8.14 |   0.08 |  1.0 |    1088.25 |  2083.78 |
gpuarrays/constructors                        (5) |    17.26 |   0.17 |  1.0 |    1925.15 |  1779.45 |
gpuarrays/linalg/mul!/matrix-matrix           (8) |    47.86 |   0.60 |  1.3 |    7159.48 |  1645.39 |
gpuarrays/base                                (9) |    17.25 |   0.50 |  2.9 |    3594.25 |  1978.89 |
gpuarrays/statistics                          (4) |    45.91 |   0.90 |  2.0 |    7192.61 |  1043.03 |
gpuarrays/linalg                             (11) |    82.65 |   1.55 |  1.9 |   13528.54 |  1527.69 |
gpuarrays/reductions/== isequal               (2) |    27.08 |   0.65 |  2.4 |    7586.81 |  2187.42 |
gpuarrays/reductions/minimum maximum extrema  (7) |   101.73 |   2.54 |  2.5 |   24664.97 |  2223.12 |
gpuarrays/reductions/mapreduce                (6) |    88.42 |   1.99 |  2.2 |   20831.73 |  2164.03 |
gpuarrays/reductions/reduce                   (9) |    56.70 |   1.47 |  2.6 |   14746.59 |  2422.55 |
gpuarrays/reductions/mapreducedim!            (8) |    76.37 |   1.68 |  2.2 |   17222.73 |  2450.88 |
gpuarrays/reductions/sum prod                 (4) |    78.09 |   1.82 |  2.3 |   18164.57 |  1960.77 |
┌ Warning: Test timed out: gpuarrays/broadcasting
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:247
      From worker 5:
      From worker 5:    ======================================================================================
      From worker 5:    Information request received. A stacktrace will print followed by a 1.0 second profile
      From worker 5:    ======================================================================================
      From worker 5:
      From worker 5:    signal (29): Information request: 29
      From worker 5:    jl_gc_state_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:337 [inlined]
      From worker 5:    jl_gc_state_save_and_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:343 [inlined]
      From worker 5:    jl_delete_thread at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/threading.c:446
      From worker 5:    _pthread_tsd_cleanup at /usr/lib/system/libsystem_pthread.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:
      From worker 5:    ==============================================================
      From worker 5:    Profile collected. A report will print at the next yield point
      From worker 5:    ==============================================================
      From worker 5:
┌ Warning: Couldn't kill worker 5, killing process 70270 forcefully
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:256
      From worker 5:
      From worker 5:    [70270] signal 15: Terminated: 15
      From worker 5:    in expression starting at none:1
      From worker 5:    jl_gc_state_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:337 [inlined]
      From worker 5:    jl_gc_state_save_and_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia_threads.h:343 [inlined]
      From worker 5:    jl_delete_thread at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/threading.c:446
      From worker 5:    _pthread_tsd_cleanup at /usr/lib/system/libsystem_pthread.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    __psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 5:    unknown function (ip: 0x0)
      From worker 5:    Allocations: 505743173 (Pool: 505727459; Big: 15714); GC: 143
gpuarrays/broadcasting                        (5) |         failed at 2024-04-11T09:11:37.975
Testing finished in 10 minutes, 2 seconds, 141 milliseconds
gpuarrays/broadcasting: Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)

Test Summary:                                  | Pass  Error  Broken  Total  Time
  Overall                                      | 8242      1       9   8252      
    metallib                                   |   20                    20      
    pool                                       |    5                     5      
    metal                                      |  131                   131      
    profiling                                  |    1                     1      
    capturing                                  |   24                    24      
    random                                     |   57                    57      
    execution                                  |   35                    35      
    gpuarrays/indexing scalar                  |  399                   399      
    mps                                        |   39                    39      
    array                                      |  303                   303      
    kernelabstractions                         | 2165              8   2173      
    device/intrinsics                          |  126                   126      
    gpuarrays/interface                        |    7                     7      
    gpuarrays/math/power                       |   60                    60      
    gpuarrays/indexing find                    |   45                    45      
    gpuarrays/uniformscaling                   |   56                    56      
    gpuarrays/reductions/any all count         |  101                   101      
    gpuarrays/linalg/mul!/vector-matrix        |  140                   140      
    gpuarrays/math/intrinsics                  |   10                    10      
    examples                                   |    3                     3      
    gpuarrays/indexing multidimensional        |   88              1     89      
    gpuarrays/reductions/reducedim!            |  160                   160      
    gpuarrays/reductions/mapreducedim!_large   |   40                    40      
    gpuarrays/linalg/norm                      |  264                   264      
    gpuarrays/random                           |   52                    52      
    gpuarrays/constructors                     |  832                   832      
    gpuarrays/linalg/mul!/matrix-matrix        |  360                   360      
    gpuarrays/base                             |   93                    93      
    gpuarrays/statistics                       |   52                    52      
    gpuarrays/linalg                           |  325                   325      
    gpuarrays/reductions/== isequal            |  248                   248      
    gpuarrays/reductions/minimum maximum extrema |  555                   555      
    gpuarrays/reductions/mapreduce             |  330                   330      
    gpuarrays/reductions/reduce                |  220                   220      
    gpuarrays/reductions/mapreducedim!         |  260                   260      
    gpuarrays/reductions/sum prod              |  636                   636      
    gpuarrays/broadcasting                     |           1              1      
    FAILURE

Error in testset gpuarrays/broadcasting:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)
ERROR: LoadError: Test run finished with errors
in expression starting at /Users/christian/.julia/dev/Metal/test/runtests.jl:410
ERROR: Package Metal errored during testing

@maleadt
Copy link
Member

maleadt commented Apr 11, 2024

Hmm, that's not terribly useful. It looks like its stuck without hitting a yield, so we don't get a backtrace for the active tasks. Could you maybe do so manually from LLDB by calling jl_print_task_backtraces(0)? For that, maybe get rid of the kill and replace it with a @show pid and sleep, or something like that.

@christiangnrd
Copy link
Contributor Author

I'm not sure I did exactly what you were looking for but I got something.

Backtrace 
┌ Warning: Test timed out: examples
└ @ Main ~/.julia/dev/Metal/test/runtests.jl:247
pid = 47702
==== Thread 1 created 42 live tasks
     ---- Root task (0x107f28010)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
_wait at ./task.jl:321
sync_end at ./task.jl:418
jfptr_sync_end_64548.3 at /Users/christian/.julia/juliaup/julia-1.11.0-beta1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_stmt_value at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:174
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:659
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:539
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:539
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:817
macro expansion at task.jl:492 [inlined]
top-level scope at /Users/christian/.julia/dev/Metal/test/runtests.jl:220
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:428 [inlined]
include_string at ./loading.jl:2543
_include at ./loading.jl:2603
include at ./sysimg.jl:38
unknown function (ip: 0x111a44087)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_stmt_value at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:174
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:659
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:817
top-level scope at none:6
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
jlplt_ijl_toplevel_eval_in_28146.3 at /Users/christian/.julia/juliaup/julia-1.11.0-beta1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
eval at ./boot.jl:428 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:526
jfptr__start_71182.3 at /Users/christian/.julia/juliaup/julia-1.11.0-beta1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
true_main at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/jlapi.c:1059
     ---- End root task
     ---- Task 3 (0x14c51cc90)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_swap_fiber at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1310
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:570
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
put_buffered at ./channels.jl:380
put! at ./channels.jl:358
unknown function (ip: 0x12d078103)
put! at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:703
#54 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:688
unknown function (ip: 0x12d0ac077)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 3
     ---- Task 4 (0x109f5c330)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_swap_fiber at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1310
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:570
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
take_buffered at ./channels.jl:493
take! at ./channels.jl:487
unknown function (ip: 0x12d3fc05f)
take! at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:726
#remotecall_fetch#159 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:461
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/builtins.c:831
remotecall_fetch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:454
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/builtins.c:831
#remotecall_fetch#162 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:492
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/builtins.c:831
remotecall_fetch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:492
#34 at /Users/christian/.julia/dev/Metal/test/runtests.jl:274
unknown function (ip: 0x12d3b11a7)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 4
     ---- Task 20 (0x10bd22720)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_swap_fiber at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1310
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:570
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
put_buffered at ./channels.jl:380
put! at ./channels.jl:358
unknown function (ip: 0x12d078103)
put! at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:703
#54 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:688
unknown function (ip: 0x12d0ac077)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 20
     ---- Task 33 (0x107f2a590)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
wait_readnb at ./stream.jl:416
eof at ./stream.jl:106
#37 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:282
unknown function (ip: 0x1129b41ab)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 33
     ---- Task 34 (0x107f2aa40)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
wait_readnb at ./stream.jl:416
wait_locked at ./stream.jl:972
unsafe_read at ./stream.jl:978
unsafe_read at ./io.jl:891 [inlined]
unsafe_read at ./io.jl:890
read! at ./io.jl:895 [inlined]
deserialize_hdr_raw at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/messages.jl:167 [inlined]
message_handler_loop at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:172
process_tcp_streams at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:133
#103 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:121
unknown function (ip: 0x12d014053)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 34
     ---- Task 36 (0x107f2b6c0)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
put_buffered at ./channels.jl:380
put! at ./channels.jl:358
unknown function (ip: 0x12d078103)
put! at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:703
#54 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:688
unknown function (ip: 0x12d0ac077)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 36
     ---- Task 41 (0x107f2b9e0)
          (sticky: 1, started: 1, state: 0, tid: 1)
jl_rec_backtrace at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/stackwalk.c:865 [inlined]
jlbacktracet at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/stackwalk.c:1133
jl_print_task_backtraces at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/stackwalk.c:1199
#35 at /Users/christian/.julia/dev/Metal/test/runtests.jl:251
#810 at ./asyncevent.jl:306
unknown function (ip: 0x12d7dc223)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 41
==== End thread 1
==== Thread 2 created 13 live tasks
     ---- Task 1 (0x107f28330)
          (sticky: 0, started: 1, state: 0, tid: 0)
jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
_trywait at ./asyncevent.jl:142
profile_printing_listener at ./Base.jl:578
#1171 at ./Base.jl:615
jfptr_YY.1171_73702.3 at /Users/christian/.julia/juliaup/julia-1.11.0-beta1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 1
     ---- Task 2 (0x10bd21140)
          (sticky: 0, started: 1, state: 0, tid: 0)
jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
try_yieldto at ./task.jl:934
wait at ./task.jl:1008
#wait#725 at ./condition.jl:130
wait at ./condition.jl:125 [inlined]
#138 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:281 [inlined]
lock at ./lock.jl:232
lock at ./condition.jl:78 [inlined]
#137 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:279
unknown function (ip: 0x111a60077)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
     ---- End task 2
==== End thread 2
==== Thread 3 created 4 live tasks
==== End thread 3
==== Thread 4 created 1 live tasks
==== End thread 4
==== Thread 5 created 0 live tasks
==== End thread 5
==== Thread 6 created 0 live tasks
==== End thread 6
==== Thread 7 created 0 live tasks
==== End thread 7
==== Thread 8 created 0 live tasks
==== End thread 8
==== Done
examples                                      (4) |         failed at 2024-04-11T14:30:39.282

@maleadt
Copy link
Member

maleadt commented Apr 11, 2024

That is the correct output, however seeing the remotecalls in there I think you attached to the parent process, and not the process that actually timed out (we spawn multiple processes during testing).

You can find out the PID of the child process in htop, or by logging it here:

Metal.jl/test/runtests.jl

Lines 245 to 247 in 7042281

pid = remotecall_fetch(getpid, wrkr)
timer = Timer(480) do _
@warn "Test timed out: $test"
. However, for that logging to be useful (i.e., the child process staying alive after the log message for you to attach) you'd need to disable the kills below.

I'd look into this myself but so far haven't been able to reproduce...

@christiangnrd
Copy link
Contributor Author

I think this is the right thing. I've never used lldb before but I found JuliaLang/julia#47928 which has a usage example.

Backtrace 
      From worker 7:	==== Thread 1 created 6 live tasks
      From worker 7:	     ---- Root task (0x108904010)
      From worker 7:	          (sticky: 1, started: 1, state: 0, tid: 1)
      From worker 7:	jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
      From worker 7:	ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
      From worker 7:	ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
      From worker 7:	try_yieldto at ./task.jl:934
      From worker 7:	wait at ./task.jl:1008
      From worker 7:	#start_worker#34 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:271
      From worker 7:	start_worker at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:232
      From worker 7:	unknown function (ip: 0x10de5c057)
      From worker 7:	#start_worker#33 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:231
      From worker 7:	start_worker at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:231
      From worker 7:	start_worker at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:231
      From worker 7:	process_opts at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:1351
      From worker 7:	unknown function (ip: 0x10dd80377)
      From worker 7:	jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
      From worker 7:	jl_f__call_latest at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/builtins.c:875
      From worker 7:	#invokelatest#2 at ./essentials.jl:1030 [inlined]
      From worker 7:	invokelatest at ./essentials.jl:1027 [inlined]
      From worker 7:	exec_options at ./client.jl:268
      From worker 7:	_start at ./client.jl:526
      From worker 7:	jfptr__start_71182.3 at /Users/christian/.julia/juliaup/julia-1.11.0-beta1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
      From worker 7:	jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
      From worker 7:	true_main at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/jlapi.c:900
      From worker 7:	jl_repl_entrypoint at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/jlapi.c:1059
      From worker 7:	     ---- End root task
      From worker 7:	     ---- Task 1 (0x108904330)
      From worker 7:	          (sticky: 0, started: 1, state: 0, tid: 0)
      From worker 7:	jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
      From worker 7:	ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
      From worker 7:	ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
      From worker 7:	try_yieldto at ./task.jl:934
      From worker 7:	wait at ./task.jl:1008
      From worker 7:	#wait#725 at ./condition.jl:130
      From worker 7:	wait at ./condition.jl:125 [inlined]
      From worker 7:	_trywait at ./asyncevent.jl:142
      From worker 7:	profile_printing_listener at ./Base.jl:578
      From worker 7:	#1171 at ./Base.jl:615
      From worker 7:	jfptr_YY.1171_73702.3 at /Users/christian/.julia/juliaup/julia-1.11.0-beta1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
      From worker 7:	jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
      From worker 7:	start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
      From worker 7:	     ---- End task 1
      From worker 7:	     ---- Task 2 (0x10c790010)
      From worker 7:	          (sticky: 0, started: 1, state: 0, tid: 0)
      From worker 7:	jl_swap_fiber at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1310
      From worker 7:	ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:570
      From worker 7:	ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
      From worker 7:	try_yieldto at ./task.jl:934
      From worker 7:	wait at ./task.jl:1008
      From worker 7:	#wait#725 at ./condition.jl:130
      From worker 7:	wait at ./condition.jl:125 [inlined]
      From worker 7:	#138 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:281 [inlined]
      From worker 7:	lock at ./lock.jl:232
      From worker 7:	lock at ./condition.jl:78 [inlined]
      From worker 7:	#137 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:279
      From worker 7:	unknown function (ip: 0x10dda4077)
      From worker 7:	jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
      From worker 7:	start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
      From worker 7:	     ---- End task 2
      From worker 7:	     ---- Task 3 (0x10a5ec970)
      From worker 7:	          (sticky: 1, started: 1, state: 0, tid: 1)
      From worker 7:	jl_start_fiber_swap at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1397
      From worker 7:	ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:616
      From worker 7:	ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
      From worker 7:	try_yieldto at ./task.jl:934
      From worker 7:	wait at ./task.jl:1008
      From worker 7:	#wait#725 at ./condition.jl:130
      From worker 7:	wait at ./condition.jl:125 [inlined]
      From worker 7:	accept at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Sockets/src/Sockets.jl:700
      From worker 7:	accept at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Sockets/src/Sockets.jl:149 [inlined]
      From worker 7:	#35 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:251
      From worker 7:	unknown function (ip: 0x10decc583)
      From worker 7:	jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
      From worker 7:	start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
      From worker 7:	     ---- End task 3
      From worker 7:	     ---- Task 4 (0x10ad08650)
      From worker 7:	          (sticky: 1, started: 1, state: 0, tid: 1)
      From worker 7:	jl_swap_fiber at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1310
      From worker 7:	ctx_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:570
      From worker 7:	ijl_switch at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:649
      From worker 7:	try_yieldto at ./task.jl:934
      From worker 7:	wait at ./task.jl:1008
      From worker 7:	#wait#725 at ./condition.jl:130
      From worker 7:	wait at ./condition.jl:125 [inlined]
      From worker 7:	wait_readnb at ./stream.jl:416
      From worker 7:	wait_locked at ./stream.jl:972
      From worker 7:	unsafe_read at ./stream.jl:978
      From worker 7:	unsafe_read at ./io.jl:891 [inlined]
      From worker 7:	unsafe_read at ./io.jl:890
      From worker 7:	read! at ./io.jl:895 [inlined]
      From worker 7:	deserialize_hdr_raw at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/messages.jl:167 [inlined]
      From worker 7:	message_handler_loop at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:172
      From worker 7:	process_tcp_streams at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:133
      From worker 7:	#103 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:121
      From worker 7:	unknown function (ip: 0x10e298053)
      From worker 7:	jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2154 [inlined]
      From worker 7:	start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/task.c:1202
      From worker 7:	     ---- End task 4
      From worker 7:	     ---- Task 5 (0x31b2652d0)
      From worker 7:	          (sticky: 1, started: 1, state: 0, tid: 1)
      From worker 7:	jl_rec_backtrace at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/stackwalk.c:865 [inlined]
      From worker 7:	jlbacktracet at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/stackwalk.c:1133
      From worker 7:	jl_print_task_backtraces at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/stackwalk.c:1199
      From worker 7:	     ---- End task 5
      From worker 7:	==== End thread 1
      From worker 7:	==== Thread 2 created 1 live tasks
      From worker 7:	     ---- Root task (0x322e20010)
      From worker 7:	          (sticky: 1, started: 1, state: 0, tid: 2)
      From worker 7:	__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 7:	     ---- End root task
      From worker 7:	==== End thread 2
      From worker 7:	==== Done

@christiangnrd christiangnrd mentioned this issue Apr 14, 2024
Merged
@maleadt
Copy link
Member

maleadt commented May 4, 2024

I haven't seen this recently anymore, maybe this was a bug in Julia?

@christiangnrd
Copy link
Contributor Author

Maybe? It happened with 1.11 beta1 though so I don’t know what would have changed for it to stop happening.

@christiangnrd
Copy link
Contributor Author

Somewhat related but do you know why testing time in 1.11 is ~1.5x the 1.10 time?

@christiangnrd
Copy link
Contributor Author

Closing as the recent sporadic failures seem different.

@christiangnrd christiangnrd mentioned this issue Aug 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maleadt @christiangnrd