creating temporary files takes a long time on Java 11

[wkozaczuk]

As I was running java unit tests on Java 11 I noticed that one of the test cases that are part of modules/java-tests/tests/src/main/java/io/osv/LoggingWithoutIsolationTest.java takes a long time (around 10 seconds) when run on Java 11. The same test runs very quickly on Java 8.

After adding printouts I have isolated this Java code taking a long time:

File file = File.createTempFile("test", null);

See https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/File.html#createTempFile(java.lang.String,java.lang.String) for what this does.

Eventually the program finishes and one can see this (I added the line above to the Java hello worlds example):

./scripts/run.py -e '/usr/bin/java -cp /java-example Hello !' -c 1
OSv v0.55.0-298-g18555601
eth0: 192.168.122.15
Booted up in 174.74 ms
Cmdline: /usr/bin/java -cp /java-example Hello !
Hello, World! //->> This is the last line after which it waits for 10 seconds or so
fpathconf() stubbed
-> After File.createTempFile()

After connecting with gdb I see this:

(gdb) bt
#0  fill_ts<long, std::ratio<1, 1000000000> > (ts=0x2000197019a0, d=...) at /usr/include/c++/10/chrono:422
#1  clock_gettime (clk_id=<optimized out>, ts=0x2000197019a0) at libc/time.cc:81
#2  0x0000100000c7e580 in os::javaTimeNanos() ()
#3  0x0000200010cd0ce8 in ?? ()
#4  0x0000000800356780 in ?? ()
#5  0x0000000000000000 in ?? ()

(gdb) osv info threads
  id  address             name            cpu  status         vruntime  total_cpu_time location
   1 (0xffff800000016040) reclaimer       cpu0 waiting     2.97268e-25             562 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
   2 (0xffff800000053040) kvm_wall_clock_ cpu0 waiting     3.95907e-15            8305 std::_Function_handler<void(), kvmclock::kvmclock()::<lambda()> >::_M_invoke(const std::_Any_data &) at /usr/include/c++/10/bits/std_function.h:291
   3 (0xffff800000071040) page-access-sca cpu0 waiting     1.48546e-11         8301839 pagecache::access_scanner::run() at core/pagecache.cc:695
   4 (0xffff800000089040) page_pool_l2    cpu0 waiting     2.02963e-16        11508850 memory::page_pool::l2::fill_thread() at core/mempool.cc:1440
   5 (0xffff800000096040) itimer-real     cpu? unstarted             0               0 ?? at arch/x64/entry.S:112
   6 (0xffff800000122040) itimer-virt     cpu? unstarted             0               0 ?? at arch/x64/entry.S:112
   7 (0xffff800000159040) balancer0       cpu0 waiting     1.07872e-13           64790 ?? at arch/x64/entry.S:113
   8 (0xffff800000171040) rcu0            cpu0 waiting     1.55096e-24            2804 osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178
   9 (0xffff80000018b040) page_pool_l1_0  cpu0 waiting     1.08365e-13          437879 memory::page_pool::l1::fill_thread() at core/mempool.cc:1352
  10 (0xffff8000001a2040) percpu0         cpu0 waiting      4.3944e-25             786 workman::call_of_duty() at core/percpu-worker.cc:92
  11 (0xffff8000001ba040) async_worker0   cpu0 waiting     1.01945e-21            3058 async::async_worker::run() at core/async.cc:158
  12 (0xffff8000001d2040) idle0           cpu0 queued              inf       536943227 std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /usr/include/c++/10/bits/std_function.h:291
  13 (0xffff800000b22040) >init           cpu0 waiting     5.84196e-24             484 osv::acpi_interrupt::process_interrupts() at drivers/acpi.cc:247
  14 (0xffff800000b39040) >init           cpu0 waiting     9.14373e-14          172890 std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) at /usr/include/c++/10/bits/std_function.h:291
  15 (0xffff800000b50040) thread taskq    cpu0 waiting     6.15216e-24            1014 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  16 (0xffff800000b5b040) callout         cpu0 waiting     9.43065e-14          117580 _callout_thread() at bsd/porting/callout.cc:138
  17 (0xffff800000b74040) system_taskq_0  cpu0 waiting     5.77734e-24             321 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  18 (0xffff800000b7f040) system_taskq_1  cpu0 waiting     5.76441e-24             328 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  19 (0xffff800000b8a040) system_taskq_2  cpu0 waiting     5.73857e-24             257 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  20 (0xffff800000b95040) system_taskq_3  cpu0 waiting     5.76441e-24             310 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  21 (0xffff800000ba0040) system_taskq_4  cpu0 waiting     5.73857e-24             252 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  22 (0xffff800000bab040) system_taskq_5  cpu0 waiting     5.76441e-24             308 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  23 (0xffff800000bb6040) system_taskq_6  cpu0 waiting     5.75149e-24             290 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  24 (0xffff800000bc1040) system_taskq_7  cpu0 waiting     5.75149e-24             294 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  25 (0xffff80007fe41040) solthread-0x402 cpu0 waiting     3.34802e-16           26930 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  26 (0xffff80007fe4c040) solthread-0x402 cpu0 waiting     3.73453e-15            5615 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  27 (0xffff80007fe73040) netisr          cpu0 waiting     1.78878e-23            3796 netisr_osv_thread_wrapper(void (*)(void*), void*) at bsd/sys/net/netisr1.cc:26
  28 (0xffff800000bce040) >init           cpu0 waiting     6.07803e-20        86062398 sched::thread::join() at core/sched.cc:1370
	joining on <optimized out>
  29 (0xffff800000be8040) virtio-net-rx   cpu0 waiting     3.32942e-28           27785 virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:158
  30 (0xffff800000c10040) virtio-tx-0     cpu0 waiting      1.0469e-28            2239 osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:399
  31 (0xffff800000d28040) virtio-blk      cpu0 waiting     6.49072e-20         5057796 virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:158
  32 (0xffff800000d3f040) virtio-rng      cpu0 waiting     3.37945e-13          798065 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  33 (0xffff800000d59040) isa-serial-inpu cpu0 waiting     1.28213e-23            4559 console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86
  34 (0xffff800000d70040) kbd-input       cpu0 waiting     1.28988e-23            4650 console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86
  35 (0xffff800000d92040) rand_harvestq   cpu0 waiting     1.17297e-12         3278500 synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101
  36 (0xffff800000dac040) dhcp            cpu0 waiting     6.64756e-22          734044 dhcp::dhcp_worker::dhcp_worker_fn() at core/dhcp.cc:828
  37 (0xffff800000f01040) /usr/bin/java   cpu0 waiting      1.8707e-20        13123899 sched::thread::join() at core/sched.cc:1370
	joining on <optimized out>
  38 (0xffff800001855040) >/usr/bin/java  cpu0 waiting     2.00517e-13       131008482 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  39 (0xffff8000025ab040) VM Thread       cpu0 waiting     3.94684e-13          574237 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  40 (0xffff8000027e3040) Reference Handl cpu0 waiting     2.10104e-21          193697 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  41 (0xffff8000028fb040) Finalizer       cpu0 waiting     3.95082e-21          348549 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  42 (0xffff800002bf4040) Signal Dispatch cpu0 waiting     7.74241e-22           47401 semaphore::wait(unsigned int, sched::timer*) at core/semaphore.cc:46
  43 (0xffff800002cfa040) Service Thread  cpu0 waiting     2.41123e-21          168287 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  44 (0xffff800002e12040) C2 CompilerThre cpu0 waiting     2.21803e-18        30482642 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  45 (0xffff800002f3c040) C1 CompilerThre cpu0 waiting     8.66538e-19        31026992 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  46 (0xffff800003079040) Sweeper thread  cpu0 waiting     8.82395e-22           52394 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  47 (0xffff800003774040) VM Periodic Tas cpu0 waiting     2.82989e-13          137938 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  48 (0xffff800003972040) Common-Cleaner  cpu0 waiting     1.61176e-21           44635 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
  49 (0xffff800003e3a040) SeedGenerator T cpu0 running     1.32268e-09      3735622772 clock_gettime(clockid_t, timespec*) at libc/time.cc:81
  65 (0xffff8000009f5040) SeedGenerator T cpu0 waiting     2.62568e-13           12648 condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43

osv thread apply all bt
thread 0xffff800000016040 reclaimer

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...)
    at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000016040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...)
    at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x0, this=0xffff90000001dec0) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0x4090d850 <memory::reclaimer_thread+16>, user_mutex=user_mutex@entry=0x4090d900 <memory::free_page_ranges_lock>, 
    tmr=tmr@entry=0x0) at core/condvar.cc:43
#8  0x00000000403e617d in condvar::wait (tmr=0x0, user_mutex=..., this=0x4090d850 <memory::reclaimer_thread+16>) at include/osv/condvar.h:226
#9  memory::reclaimer::_do_reclaim (this=0x4090d840 <memory::reclaimer_thread>) at core/mempool.cc:1071
#10 0x00000000403e635c in operator() (__closure=<optimized out>, __closure=<optimized out>) at core/mempool.cc:1026
#11 std::__invoke_impl<void, memory::reclaimer::reclaimer()::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#12 std::__invoke_r<void, memory::reclaimer::reclaimer()::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#13 std::_Function_handler<void(), memory::reclaimer::reclaimer()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...)
    at /usr/include/c++/10/bits/std_function.h:291
#14 0x00000000403fbffa in sched::thread::main (this=0xffff800000016040) at core/sched.cc:1267
#15 sched::thread_main_c (t=0xffff800000016040) at arch/x64/arch-switch.hh:325
#16 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000053040 kvm_wall_clock_

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...)
    at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=<optimized out>) at core/sched.cc:1273
#4  0x00000000403fce56 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::sleep_impl(sched::timer&)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<sched::thread::sleep_impl(sched::timer&)::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  sched::thread::sleep_impl (t=...) at core/sched.cc:1429
#7  0x000000004038057d in sched::thread::sleep<long, std::ratio<1l, 1l> > (duration=...) at include/osv/sched.hh:1314
#8  operator() (__closure=0xffff800000053070) at drivers/kvmclock.cc:69
#9  std::__invoke_impl<void, kvmclock::kvmclock()::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#10 std::__invoke_r<void, kvmclock::kvmclock()::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
--Type <RET> for more, q to quit, c to continue without paging--c
#11 std::_Function_handler<void(), kvmclock::kvmclock()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#12 0x00000000403fbffa in sched::thread::main (this=0xffff800000053040) at core/sched.cc:1267
#13 sched::thread_main_c (t=0xffff800000053040) at arch/x64/arch-switch.hh:325
#14 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000071040 page-access-sca

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=<optimized out>) at core/sched.cc:1273
#4  0x00000000403fce56 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::sleep_impl(sched::timer&)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<sched::thread::sleep_impl(sched::timer&)::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  sched::thread::sleep_impl (t=...) at core/sched.cc:1429
#7  0x000000004038aee6 in sched::thread::sleep<long, std::ratio<1l, 1000000000l> > (duration=...) at include/osv/sched.hh:1314
#8  0x00000000403e30e6 in pagecache::access_scanner::run (this=0x4090c940 <pagecache::s_access_scanner>) at core/pagecache.cc:695
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000071040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000071040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000089040 page_pool_l2

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000089040) at core/sched.cc:1273
#4  0x00000000403e8a5c in sched::thread::do_wait_for<sched::thread::dummy_lock, sched::wait_object<memory::page_pool::l2::fill_thread()::<lambda()> > > (mtx=<synthetic pointer>...) at include/osv/sched.hh:1252
#5  sched::thread::wait_for<memory::page_pool::l2::fill_thread()::<lambda()> > () at include/osv/sched.hh:1276
#6  memory::page_pool::l2::fill_thread (this=0x4090d640 <memory::page_pool::global_l2>) at core/mempool.cc:1440
#7  0x00000000403eb80c in memory::page_pool::l2::l2()::{lambda()#1}::operator()() const (__closure=<optimized out>, __closure=<optimized out>) at core/mempool.cc:1247
#8  std::__invoke_impl<void, memory::page_pool::l2::l2()::{lambda()#1}&>(std::__invoke_other, memory::page_pool::l2::l2()::{lambda()#1}&) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, memory::page_pool::l2::l2()::{lambda()#1}&>(memory::page_pool::l2::l2()::{lambda()#1}&) (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void (), memory::page_pool::l2::l2()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff800000089040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff800000089040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000096040 itimer-real

#0  thread_main () at arch/x64/entry.S:112

thread 0xffff800000122040 itimer-virt

#0  thread_main () at arch/x64/entry.S:112

thread 0xffff800000159040 balancer0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=<optimized out>) at core/sched.cc:1273
#4  0x00000000403fc76d in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::cpu::load_balance()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<sched::cpu::load_balance()::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  sched::cpu::load_balance (this=0xffff80000001e040) at core/sched.cc:730
#7  0x00000000403fbffa in sched::thread::main (this=0xffff800000159040) at core/sched.cc:1267
#8  sched::thread_main_c (t=0xffff800000159040) at arch/x64/arch-switch.hh:325
#9  0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000171040 rcu0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff80000018b040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000171040) at core/sched.cc:1273
#4  0x00000000403de6fe in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, osv::rcu::cpu_quiescent_state_thread::do_work()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<osv::rcu::cpu_quiescent_state_thread::do_work()::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  osv::rcu::cpu_quiescent_state_thread::do_work (this=0xffffa0000015f7a0) at core/rcu.cc:178
#7  0x00000000403dea49 in osv::rcu::cpu_quiescent_state_thread::work (this=<optimized out>) at core/rcu.cc:113
#8  0x00000000403dea6c in operator() (__closure=<optimized out>, __closure=<optimized out>) at core/rcu.cc:67
#9  std::__invoke_impl<void, osv::rcu::cpu_quiescent_state_thread::cpu_quiescent_state_thread(sched::cpu*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#10 std::__invoke_r<void, osv::rcu::cpu_quiescent_state_thread::cpu_quiescent_state_thread(sched::cpu*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#11 std::_Function_handler<void(), osv::rcu::cpu_quiescent_state_thread::cpu_quiescent_state_thread(sched::cpu*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#12 0x00000000403fbffa in sched::thread::main (this=0xffff800000171040) at core/sched.cc:1267
#13 sched::thread_main_c (t=0xffff800000171040) at arch/x64/arch-switch.hh:325
#14 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff80000018b040 page_pool_l1_0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff80000018b040) at core/sched.cc:1273
#4  0x00000000403e88a1 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, memory::page_pool::l1::fill_thread()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<memory::page_pool::l1::fill_thread()::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  memory::page_pool::l1::fill_thread () at core/mempool.cc:1352
#7  0x00000000403eb7f9 in memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}::operator()() const (__closure=<optimized out>) at core/mempool.cc:1164
#8  std::__invoke_impl<void, memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}&>(std::__invoke_other, memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}&) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}&>(memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}&) (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void (), memory::page_pool::l1::l1(sched::cpu*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff80000018b040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff80000018b040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff8000001a2040 percpu0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff8000001ba040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff8000001a2040) at core/sched.cc:1273
#4  0x000000004042084f in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, workman::call_of_duty()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<workman::call_of_duty()::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  workman::call_of_duty () at core/percpu-worker.cc:92
#7  0x0000000040420a19 in operator() (__closure=<optimized out>) at core/percpu-worker.cc:124
#8  std::__invoke_impl<void, workman::pcpu_init()::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, workman::pcpu_init()::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void(), workman::pcpu_init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff8000001a2040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff8000001a2040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff8000001ba040 async_worker0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800001855040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff8000001ba040) at core/sched.cc:1273
#4  0x0000000040429564 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, async::async_worker::run()::{lambda()#1}>(sched::thread::dummy_lock&, async::async_worker::run()::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<async::async_worker::run()::{lambda()#1}>(async::async_worker::run()::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  async::async_worker::run (this=0xffffa000001a87c0) at core/async.cc:158
#7  0x00000000403fbffa in sched::thread::main (this=0xffff8000001ba040) at core/sched.cc:1267
#8  sched::thread_main_c (t=0xffff8000001ba040) at arch/x64/arch-switch.hh:325
#9  0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff8000001d2040 idle0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000d28040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fc2b4 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::cpu::idle (this=0xffff80000001e040) at core/sched.cc:474
#4  0x00000000403fc2dc in operator() (__closure=<optimized out>, __closure=<optimized out>) at core/sched.cc:165
#5  std::__invoke_impl<void, sched::cpu::init_idle_thread()::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#6  std::__invoke_r<void, sched::cpu::init_idle_thread()::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#7  std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#8  0x00000000403fbffa in sched::thread::main (this=0xffff8000001d2040) at core/sched.cc:1267
#9  sched::thread_main_c (t=0xffff8000001d2040) at arch/x64/arch-switch.hh:325
#10 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b22040 >init

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b22040) at core/sched.cc:1273
#4  0x0000000040381de0 in sched::thread::do_wait_for<sched::thread::dummy_lock, sched::wait_object<osv::acpi_interrupt::process_interrupts()::{lambda()#1}>, sched::wait_object<osv::acpi_interrupt::process_interrupts()::{lambda()#2}> >(sched::thread::dummy_lock&, sched::wait_object<osv::acpi_interrupt::process_interrupts()::{lambda()#1}>&&, sched::wait_object<osv::acpi_interrupt::process_interrupts()::{lambda()#2}>&&) (mtx=<synthetic pointer>...) at include/osv/sched.hh:1252
#5  sched::thread::wait_for<osv::acpi_interrupt::process_interrupts()::{lambda()#1}, osv::acpi_interrupt::process_interrupts()::{lambda()#2}>(osv::acpi_interrupt::process_interrupts()::{lambda()#1}&&, osv::acpi_interrupt::process_interrupts()::{lambda()#2}&&) () at include/osv/sched.hh:1276
#6  osv::acpi_interrupt::process_interrupts (this=0xffffa0000097f380) at drivers/acpi.cc:247
#7  0x00000000403fbffa in sched::thread::main (this=0xffff800000b22040) at core/sched.cc:1267
#8  sched::thread_main_c (t=0xffff800000b22040) at arch/x64/arch-switch.hh:325
#9  0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b39040 >init

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff80000018b040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=<optimized out>) at core/sched.cc:1273
#4  0x00000000403fd180 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, sched::thread::reaper::reap()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<sched::thread::reaper::reap()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1132
#6  sched::thread::reaper::reap (this=0xffffa00000978400) at core/sched.cc:1612
#7  0x00000000403fd29c in operator() (__closure=<optimized out>, __closure=<optimized out>) at core/sched.cc:1603
#8  std::__invoke_impl<void, sched::thread::reaper::reaper()::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, sched::thread::reaper::reaper()::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff800000b39040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff800000b39040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b50040 thread taskq

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b50040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa00000978530, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa00000978530, p=0xffffa00000978500, tq=0xffffa00000978500) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000b50040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000b50040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b5b040 callout

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b5b040) at core/sched.cc:1273
#4  0x0000000040258eef in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, _callout_thread()::<lambda()> > (mtx=..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<_callout_thread()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1132
#6  _callout_thread () at bsd/porting/callout.cc:138
#7  0x00000000403fbffa in sched::thread::main (this=0xffff800000b5b040) at core/sched.cc:1267
#8  sched::thread_main_c (t=0xffff800000b5b040) at arch/x64/arch-switch.hh:325
#9  0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b74040 system_taskq_0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b74040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000b74040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000b74040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b7f040 system_taskq_1

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b7f040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000b7f040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000b7f040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b8a040 system_taskq_2

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b8a040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000b8a040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000b8a040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000b95040 system_taskq_3

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000b95040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000b95040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000b95040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000ba0040 system_taskq_4

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000ba0040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000ba0040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000ba0040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000bab040 system_taskq_5

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000bab040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000bab040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000bab040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000bb6040 system_taskq_6

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000bb6040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000bb6040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000bb6040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000bc1040 system_taskq_7

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0x408b1df0) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000bc1040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0xffffa000009786b0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=0) at bsd/porting/synch.cc:101
#7  0x00000000402542c5 in TQ_SLEEP (t=0, wm=0x4068a6ea "-", pri=0, m=0xffffa000009786b0, p=0xffffa00000978680, tq=0xffffa00000978680) at bsd/sys/kern/subr_taskqueue.c:92
#8  taskqueue_thread_loop (arg=<optimized out>) at bsd/sys/kern/subr_taskqueue.c:474
#9  0x00000000403fbffa in sched::thread::main (this=0xffff800000bc1040) at core/sched.cc:1267
#10 sched::thread_main_c (t=0xffff800000bc1040) at arch/x64/arch-switch.hh:325
#11 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff80007fe41040 solthread-0x402

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff80007fe41040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0xffff80007fe4aea0, this=0xffff80007fe4ae00) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0x408f0520 <arc_reclaim_thr_cv>, user_mutex=user_mutex@entry=0x408f0560 <arc_reclaim_thr_lock>, tmr=tmr@entry=0xffff80007fe4aea0) at core/condvar.cc:43
#8  0x0000000040256cad in condvar::wait<long, std::ratio<1l, 1000000000l> > (duration=..., user_mutex=0x408f0560 <arc_reclaim_thr_lock>, this=0x408f0520 <arc_reclaim_thr_cv>) at include/osv/condvar.h:151
#9  cv_timedwait (cv=cv@entry=0x408f0520 <arc_reclaim_thr_cv>, mutex=mutex@entry=0x408f0560 <arc_reclaim_thr_lock>, tmo=tmo@entry=1000) at bsd/porting/netport1.cc:66
#10 0x00000000402c427e in arc_reclaim_thread (dummy=<optimized out>) at bsd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2539
#11 0x00000000403fbffa in sched::thread::main (this=0xffff80007fe41040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff80007fe41040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff80007fe4c040 solthread-0x402

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff80007fe41040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff80007fe4c040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0xffff80007fe55e30, this=0xffff80007fe55d80) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0x408e05c0 <l2arc_feed_thr_cv>, user_mutex=user_mutex@entry=0x408e0600 <l2arc_feed_thr_lock>, tmr=tmr@entry=0xffff80007fe55e30) at core/condvar.cc:43
#8  0x0000000040256cad in condvar::wait<long, std::ratio<1l, 1000000000l> > (duration=..., user_mutex=0x408e0600 <l2arc_feed_thr_lock>, this=0x408e05c0 <l2arc_feed_thr_cv>) at include/osv/condvar.h:151
#9  cv_timedwait (cv=cv@entry=0x408e05c0 <l2arc_feed_thr_cv>, mutex=mutex@entry=0x408e0600 <l2arc_feed_thr_lock>, tmo=tmo@entry=1000) at bsd/porting/netport1.cc:66
#10 0x00000000402c0e88 in l2arc_feed_thread (dummy=<optimized out>) at bsd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4983
#11 0x00000000403fbffa in sched::thread::main (this=0xffff80007fe4c040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff80007fe4c040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff80007fe73040 netisr

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000dac040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff80007fe73040) at core/sched.cc:1273
#4  0x0000000040268c7a in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, netisr_osv_thread_wrapper(netisr_osv_handler_t, void*)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<netisr_osv_thread_wrapper(netisr_osv_handler_t, void*)::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  netisr_osv_thread_wrapper (handler=0x402678f0 <swi_net(void*)>, arg=0x408b92e0 <main_nws>) at bsd/sys/net/netisr1.cc:26
#7  0x0000000040268d10 in operator() (__closure=<optimized out>, __closure=<optimized out>) at bsd/sys/net/netisr1.cc:39
#8  std::__invoke_impl<void, netisr_osv_start_thread(netisr_osv_handler_t, void*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, netisr_osv_start_thread(netisr_osv_handler_t, void*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void(), netisr_osv_start_thread(netisr_osv_handler_t, void*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff80007fe73040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff80007fe73040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000bce040 >init

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff8000001d2040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=<optimized out>) at core/sched.cc:1273
#4  0x00000000403fd09f in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::join()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<sched::thread::join()::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  sched::thread::join (this=<optimized out>) at core/sched.cc:1370
#7  0x0000000040460416 in pthread_private::pthread::join (retval=0x0, this=0xffffa000009a45c0) at /usr/include/c++/10/bits/unique_ptr.h:173
#8  pthread_join (thread=18446638520603395520, retval=0x0) at libc/pthread.cc:246
#9  0x000000004042b198 in osv::application::join (this=0xffffa0007ffb9e10) at core/app.cc:259
#10 0x000000004022bd7f in do_main_thread (_main_args=<optimized out>) at /usr/include/c++/10/bits/shared_ptr_base.h:1324
#11 0x000000004045f6a6 in operator() (__closure=0xffffa0007ff77100) at libc/pthread.cc:115
#12 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#13 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#14 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#15 0x00000000403fbffa in sched::thread::main (this=0xffff800000bce040) at core/sched.cc:1267
#16 sched::thread_main_c (t=0xffff800000bce040) at arch/x64/arch-switch.hh:325
#17 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000be8040 virtio-net-rx

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000dac040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000be8040) at core/sched.cc:1273
#4  0x000000004036c591 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  virtio::virtio_driver::wait_for_queue (this=this@entry=0xffff90007ff82000, queue=queue@entry=0xffffa0007ff77300, pred=<optimized out>) at drivers/virtio.cc:158
#7  0x00000000403715d2 in virtio::net::receiver (this=0xffff90007ff82000) at drivers/virtio-net.cc:468
#8  0x0000000040371b5c in operator() (__closure=<optimized out>, __closure=<optimized out>) at drivers/virtio-net.cc:242
#9  std::__invoke_impl<void, virtio::net::net(virtio::virtio_device&)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#10 std::__invoke_r<void, virtio::net::net(virtio::virtio_device&)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#11 std::_Function_handler<void(), virtio::net::net(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#12 0x00000000403fbffa in sched::thread::main (this=0xffff800000be8040) at core/sched.cc:1267
#13 sched::thread_main_c (t=0xffff800000be8040) at arch/x64/arch-switch.hh:325
#14 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000c10040 virtio-tx-0

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000bce040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000c10040) at core/sched.cc:1273
#4  0x00000000403745cb in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until()::{lambda()#1}>(sched::thread::dummy_lock&, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until()::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until()::{lambda()#1}>(osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until()::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() (this=0xffff90007ff823c0) at include/osv/percpu_xmit.hh:399
#7  0x000000004037471c in osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}::operator()() const (this=<optimized out>, this=<optimized out>) at include/osv/percpu_xmit.hh:222
#8  std::__invoke_impl<void, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}&>(std::__invoke_other, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}&) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}&>(osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}&) (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void (), osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::xmitter(virtio::net::txq*, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff800000c10040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff800000c10040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000d28040 virtio-blk

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800002e12040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000d28040) at core/sched.cc:1273
#4  0x000000004036c591 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const)::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  virtio::virtio_driver::wait_for_queue (this=this@entry=0xffffa00000cff400, queue=queue@entry=0xffffa0007ff77700, pred=<optimized out>) at drivers/virtio.cc:158
#7  0x000000004037b00a in virtio::blk::req_done (this=0xffffa00000cff400) at drivers/virtio-blk.cc:235
#8  0x000000004037b29c in operator() (__closure=<optimized out>, __closure=<optimized out>) at drivers/virtio-blk.cc:130
#9  std::__invoke_impl<void, virtio::blk::blk(virtio::virtio_device&)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#10 std::__invoke_r<void, virtio::blk::blk(virtio::virtio_device&)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#11 std::_Function_handler<void(), virtio::blk::blk(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#12 0x00000000403fbffa in sched::thread::main (this=0xffff800000d28040) at core/sched.cc:1267
#13 sched::thread_main_c (t=0xffff800000d28040) at arch/x64/arch-switch.hh:325
#14 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000d3f040 virtio-rng

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000d3f040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x0, this=0xffff800000d54f40) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa00000cffa48, user_mutex=user_mutex@entry=0xffffa00000cffad0, tmr=tmr@entry=0x0) at core/condvar.cc:43
#8  0x000000004037ed85 in condvar::wait_until<virtio::rng::worker()::<lambda()> > (pred=..., mtx=..., this=0xffffa00000cffa48) at include/osv/condvar.h:237
#9  virtio::rng::worker (this=0xffffa00000cff800) at drivers/virtio-rng.cc:104
#10 0x000000004037edcc in operator() (__closure=<optimized out>, __closure=<optimized out>) at drivers/virtio-rng.cc:41
#11 std::__invoke_impl<void, virtio::rng::rng(virtio::virtio_device&)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#12 std::__invoke_r<void, virtio::rng::rng(virtio::virtio_device&)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#13 std::_Function_handler<void(), virtio::rng::rng(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#14 0x00000000403fbffa in sched::thread::main (this=0xffff800000d3f040) at core/sched.cc:1267
#15 sched::thread_main_c (t=0xffff800000d3f040) at arch/x64/arch-switch.hh:325
#16 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000d59040 isa-serial-inpu

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000bce040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000d59040) at core/sched.cc:1273
#4  0x0000000040350de0 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1132
#6  console::LineDiscipline::read_poll (this=0xffffa0007ff77900, driver=0x408f5740 <console::arch_early_console>) at drivers/line-discipline.cc:86
#7  0x00000000403fbffa in sched::thread::main (this=0xffff800000d59040) at core/sched.cc:1267
#8  sched::thread_main_c (t=0xffff800000d59040) at arch/x64/arch-switch.hh:325
#9  0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000d70040 kbd-input

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000bce040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000d70040) at core/sched.cc:1273
#4  0x0000000040350de0 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<console::LineDiscipline::read_poll(console::console_driver*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1132
#6  console::LineDiscipline::read_poll (this=0xffffa0007ff77900, driver=0xffff800000bd4040) at drivers/line-discipline.cc:86
#7  0x00000000403fbffa in sched::thread::main (this=0xffff800000d70040) at core/sched.cc:1267
#8  sched::thread_main_c (t=0xffff800000d70040) at arch/x64/arch-switch.hh:325
#9  0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000d92040 rand_harvestq

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000d3f040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000d92040) at core/sched.cc:1273
#4  0x0000000040259e94 in sched::thread::do_wait_until<sched::interruptible, sched::thread::dummy_lock, synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until_interruptible<synch_port::_msleep(void*, mtx*, int, char const*, int)::<lambda()> > (pred=...) at include/osv/sched.hh:1147
#6  synch_port::_msleep (this=<optimized out>, chan=<optimized out>, mtx=0x0, priority=<optimized out>, wmesg=<optimized out>, timo_hz=100) at bsd/porting/synch.cc:101
#7  0x00000000402a9c25 in random_kthread (arg=0x402a9250 <random_process_event(harvest*)>) at bsd/sys/dev/random/random_harvestq.cc:112
#8  0x00000000403fbffa in sched::thread::main (this=0xffff800000d92040) at core/sched.cc:1267
#9  sched::thread_main_c (t=0xffff800000d92040) at arch/x64/arch-switch.hh:325
#10 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000dac040 dhcp

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800000071040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800000dac040) at core/sched.cc:1273
#4  0x0000000040423f90 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, dhcp::dhcp_worker::dhcp_worker_fn()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<dhcp::dhcp_worker::dhcp_worker_fn()::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1132
#6  dhcp::dhcp_worker::dhcp_worker_fn (this=0x40911de0 <net_dhcp_worker>) at core/dhcp.cc:828
#7  0x00000000404240dc in operator() (__closure=<optimized out>, __closure=<optimized out>) at core/dhcp.cc:773
#8  std::__invoke_impl<void, dhcp::dhcp_worker::init()::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#9  std::__invoke_r<void, dhcp::dhcp_worker::init()::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#10 std::_Function_handler<void(), dhcp::dhcp_worker::init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#11 0x00000000403fbffa in sched::thread::main (this=0xffff800000dac040) at core/sched.cc:1267
#12 sched::thread_main_c (t=0xffff800000dac040) at arch/x64/arch-switch.hh:325
#13 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800000f01040 /usr/bin/java

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff8000001d2040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=<optimized out>) at core/sched.cc:1273
#4  0x00000000403fd09f in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, sched::thread::join()::<lambda()> > (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<sched::thread::join()::<lambda()> > (pred=...) at include/osv/sched.hh:1125
#6  sched::thread::join (this=<optimized out>) at core/sched.cc:1370
#7  0x0000000040460416 in pthread_private::pthread::join (retval=0x2000006fcbd8, this=0xffffa000009a46f0) at /usr/include/c++/10/bits/unique_ptr.h:173
#8  pthread_join (thread=18446638520603395824, retval=0x2000006fcbd8) at libc/pthread.cc:246
#9  0x0000100000014d3d in ContinueInNewThread0 ()
#10 0x000010000001105e in ContinueInNewThread ()
#11 0x00001000000129fa in JLI_Launch ()
#12 0x00001000000062c3 in main ()
#13 0x000000004042d70d in osv::application::run_main (this=0xffffa0007ffb9e10) at core/app.cc:413
#14 0x000000004042d851 in __libc_start_main (main=0x100000006190 <main>) at core/app.cc:37
#15 0x000010000000636e in _start ()

thread 0xffff800001855040 >/usr/bin/java

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800001855040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x0, this=0x2000007ffb00) at include/osv/wait_record.hh:46
#7  condvar::wait (this=0xffffa000014a9d80, user_mutex=0xffffa00003908d00, tmr=0x0) at core/condvar.cc:43
#8  0x0000100000c9036b in os::PlatformEvent::park() ()
#9  0x0000100000c5de25 in ObjectMonitor::wait(long, bool, Thread*) ()
#10 0x0000100000e676e8 in ObjectSynchronizer::wait(Handle, long, Thread*) ()
#11 0x00001000009a182f in JVM_MonitorWait ()
#12 0x000020000920fe17 in ?? ()
#13 0x0000000000000000 in ?? ()

thread 0xffff8000025ab040 VM Thread

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff8000025ab040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x200018d00a90, this=0x200018d00a00) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa0000179b300, user_mutex=0xffffa000017d4dc0, tmr=tmr@entry=0x200018d00a90) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa00001755d50, mutex=0xffffa00001755d28, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c38529 in Monitor::IWait(Thread*, long) ()
#11 0x0000100000c38f41 in Monitor::wait(bool, long, bool) ()
#12 0x0000100000f206a2 in VMThread::loop() ()
#13 0x0000100000f20e35 in VMThread::run() ()
#14 0x0000100000ea75bf in Thread::call_run() ()
#15 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#16 0x000000004045f6a6 in operator() (__closure=0xffffa0000179c800) at libc/pthread.cc:115
#17 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#18 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#19 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#20 0x00000000403fbffa in sched::thread::main (this=0xffff8000025ab040) at core/sched.cc:1267
#21 sched::thread_main_c (t=0xffff8000025ab040) at arch/x64/arch-switch.hh:325
#22 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff8000027e3040 Reference Handl

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800001855040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff8000027e3040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x0, this=0x200018e00800) at include/osv/wait_record.hh:46
#7  condvar::wait (this=0xffffa0000179b780, user_mutex=0xffffa00002567300, tmr=0x0) at core/condvar.cc:43
#8  0x0000100000c9036b in os::PlatformEvent::park() ()
#9  0x0000100000c38468 in Monitor::IWait(Thread*, long) ()
#10 0x0000100000c38fbe in Monitor::wait(bool, long, bool) ()
#11 0x00001000009aa886 in JVM_WaitForReferencePendingList ()
#12 0x000020000920fe17 in ?? ()
#13 0x000020000920fad6 in ?? ()
#14 0x0000200018e00988 in ?? ()
#15 0x0000000800514038 in ?? ()
#16 0x0000200018e009e8 in ?? ()
#17 0x0000000800009cd8 in ?? ()
#18 0x0000000000000000 in ?? ()

thread 0xffff8000028fb040 Finalizer

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800001855040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff8000028fb040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x0, this=0x200018f00640) at include/osv/wait_record.hh:46
#7  condvar::wait (this=0xffffa0000179bc00, user_mutex=0xffffa00002567cc0, tmr=0x0) at core/condvar.cc:43
#8  0x0000100000c9036b in os::PlatformEvent::park() ()
#9  0x0000100000c5de25 in ObjectMonitor::wait(long, bool, Thread*) ()
#10 0x0000100000e676e8 in ObjectSynchronizer::wait(Handle, long, Thread*) ()
#11 0x00001000009a182f in JVM_MonitorWait ()
#12 0x000020000920fe17 in ?? ()
#13 0x0000200018f00920 in ?? ()
#14 0x00000000000000b6 in ?? ()
#15 0xffff900002561000 in ?? ()
#16 0x0000200018f008c8 in ?? ()
#17 0x0000000000000000 in ?? ()

thread 0xffff800002bf4040 Signal Dispatch

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800001855040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800002bf4040) at core/sched.cc:1273
#4  0x00000000403db737 in sched::thread::do_wait_until<sched::noninterruptible, lockfree::mutex, semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<semaphore::wait(unsigned int, sched::timer*)::<lambda()> > (pred=..., mtx=...) at include/osv/sched.hh:1132
#6  semaphore::wait (this=0xffffa00001473180, units=units@entry=1, tmr=tmr@entry=0x0) at core/semaphore.cc:46
#7  0x0000000040466463 in sem_wait (s=<optimized out>) at libc/sem.cc:47
#8  0x0000100000d4a2da in PosixSemaphore::wait() ()
#9  0x0000100000c807c0 in os::signal_wait() ()
#10 0x0000100000c73165 in signal_thread_entry(JavaThread*, Thread*) ()
#11 0x0000100000eac308 in JavaThread::thread_main_inner() ()
#12 0x0000100000ea75bf in Thread::call_run() ()
#13 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#14 0x000000004045f6a6 in operator() (__closure=0xffffa00002798b00) at libc/pthread.cc:115
#15 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#16 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#17 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#18 0x00000000403fbffa in sched::thread::main (this=0xffff800002bf4040) at core/sched.cc:1267
#19 sched::thread_main_c (t=0xffff800002bf4040) at arch/x64/arch-switch.hh:325
#20 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800002cfa040 Service Thread

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800001855040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800002cfa040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x0, this=0x200019100c40) at include/osv/wait_record.hh:46
#7  condvar::wait (this=0xffffa000028ac880, user_mutex=0xffffa000028107c0, tmr=0x0) at core/condvar.cc:43
#8  0x0000100000c9036b in os::PlatformEvent::park() ()
#9  0x0000100000c38468 in Monitor::IWait(Thread*, long) ()
#10 0x0000100000c38f41 in Monitor::wait(bool, long, bool) ()
#11 0x0000100000d4ab42 in ServiceThread::service_thread_entry(JavaThread*, Thread*) ()
#12 0x0000100000eac308 in JavaThread::thread_main_inner() ()
#13 0x0000100000ea75bf in Thread::call_run() ()
#14 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#15 0x000000004045f6a6 in operator() (__closure=0xffffa00002b03100) at libc/pthread.cc:115
#16 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#17 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#18 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#19 0x00000000403fbffa in sched::thread::main (this=0xffff800002cfa040) at core/sched.cc:1267
#20 sched::thread_main_c (t=0xffff800002cfa040) at arch/x64/arch-switch.hh:325
#21 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800002e12040 C2 CompilerThre

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800002e12040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x200019200b90, this=0x200019200b00) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa000028acf00, user_mutex=0xffffa00002810c40, tmr=tmr@entry=0x200019200b90) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa000029f7750, mutex=0xffffa000029f7728, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c38529 in Monitor::IWait(Thread*, long) ()
#11 0x0000100000c38fbe in Monitor::wait(bool, long, bool) ()
#12 0x00001000006a0341 in CompileQueue::get() ()
#13 0x00001000006a375f in CompileBroker::compiler_thread_loop() ()
#14 0x0000100000eac308 in JavaThread::thread_main_inner() ()
#15 0x0000100000ea75bf in Thread::call_run() ()
#16 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#17 0x000000004045f6a6 in operator() (__closure=0xffffa00002b03700) at libc/pthread.cc:115
#18 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#19 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#20 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#21 0x00000000403fbffa in sched::thread::main (this=0xffff800002e12040) at core/sched.cc:1267
#22 sched::thread_main_c (t=0xffff800002e12040) at arch/x64/arch-switch.hh:325
#23 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800002f3c040 C1 CompilerThre

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800002f3c040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x200019300b90, this=0x200019300b00) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa000029f8500, user_mutex=0xffffa00002d1d380, tmr=tmr@entry=0x200019300b90) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa00002b00550, mutex=0xffffa00002b00528, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c38529 in Monitor::IWait(Thread*, long) ()
#11 0x0000100000c38fbe in Monitor::wait(bool, long, bool) ()
#12 0x00001000006a0341 in CompileQueue::get() ()
#13 0x00001000006a375f in CompileBroker::compiler_thread_loop() ()
#14 0x0000100000eac308 in JavaThread::thread_main_inner() ()
#15 0x0000100000ea75bf in Thread::call_run() ()
#16 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#17 0x000000004045f6a6 in operator() (__closure=0xffffa00002b03e00) at libc/pthread.cc:115
#18 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#19 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#20 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#21 0x00000000403fbffa in sched::thread::main (this=0xffff800002f3c040) at core/sched.cc:1267
#22 sched::thread_main_c (t=0xffff800002f3c040) at arch/x64/arch-switch.hh:325
#23 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800003079040 Sweeper thread

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800002f3c040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800003079040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x200019400cb0, this=0x200019400c00) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa000029f8a00, user_mutex=0xffffa00002e28140, tmr=tmr@entry=0x200019400cb0) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa00002b00f50, mutex=0xffffa00002b00f28, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c38529 in Monitor::IWait(Thread*, long) ()
#11 0x0000100000c38f41 in Monitor::wait(bool, long, bool) ()
#12 0x0000100000e5f450 in NMethodSweeper::sweeper_loop() ()
#13 0x0000100000eac308 in JavaThread::thread_main_inner() ()
#14 0x0000100000ea75bf in Thread::call_run() ()
#15 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#16 0x000000004045f6a6 in operator() (__closure=0xffffa00002d18400) at libc/pthread.cc:115
#17 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#18 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#19 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#20 0x00000000403fbffa in sched::thread::main (this=0xffff800003079040) at core/sched.cc:1267
#21 sched::thread_main_c (t=0xffff800003079040) at arch/x64/arch-switch.hh:325
#22 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800003774040 VM Periodic Tas

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800003e3a040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800003774040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x200019501cd0, this=0x200019501c40) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa00002f53200, user_mutex=0xffffa00002f56880, tmr=tmr@entry=0x200019501cd0) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa00003031d50, mutex=0xffffa00003031d28, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c38529 in Monitor::IWait(Thread*, long) ()
#11 0x0000100000c38f41 in Monitor::wait(bool, long, bool) ()
#12 0x0000100000ea10f9 in WatcherThread::sleep() const ()
#13 0x0000100000ea11af in WatcherThread::run() ()
#14 0x0000100000ea75bf in Thread::call_run() ()
#15 0x0000100000c863b6 in thread_native_entry(Thread*) ()
#16 0x000000004045f6a6 in operator() (__closure=0xffffa00002ff8c00) at libc/pthread.cc:115
#17 std::__invoke_impl<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#18 std::__invoke_r<void, pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()>&> (__fn=...) at /usr/include/c++/10/bits/invoke.h:153
#19 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#20 0x00000000403fbffa in sched::thread::main (this=0xffff800003774040) at core/sched.cc:1267
#21 sched::thread_main_c (t=0xffff800003774040) at arch/x64/arch-switch.hh:325
#22 0x0000000040390ec3 in thread_main () at arch/x64/entry.S:113

thread 0xffff800003972040 Common-Cleaner

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff800001855040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff800003972040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x2000196015c0, this=0x200019601500) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa00002f53700, user_mutex=0xffffa00003526880, tmr=tmr@entry=0x2000196015c0) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa000034fb750, mutex=0xffffa000034fb728, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c5d6d8 in ObjectMonitor::wait(long, bool, Thread*) ()
#11 0x0000100000e676e8 in ObjectSynchronizer::wait(Handle, long, Thread*) ()
#12 0x00001000009a182f in JVM_MonitorWait ()
#13 0x000020000920fe17 in ?? ()
#14 0x0000000000000000 in ?? ()

thread 0xffff800003e3a040 SeedGenerator T

#0  fill_ts<long, std::ratio<1, 1000000000> > (ts=0x2000197019a0, d=...) at /usr/include/c++/10/chrono:422
#1  clock_gettime (clk_id=<optimized out>, ts=0x2000197019a0) at libc/time.cc:81
#2  0x0000100000c7e580 in os::javaTimeNanos() ()
#3  0x0000200010cd0ce8 in ?? ()
#4  0x0000000800356780 in ?? ()
#5  0x0000000000000000 in ?? ()

thread 0xffff8000009f5040 SeedGenerator T

#0  sched::thread::switch_to (this=0x11225f0bd, this@entry=0xffff8000009ef040) at arch/x64/arch-switch.hh:108
#1  0x00000000403fb397 in sched::cpu::reschedule_from_interrupt (this=0xffff80000001e040, called_from_yield=<optimized out>, preempt_after=...) at core/sched.cc:374
#2  0x00000000403fb945 in sched::cpu::schedule () at include/osv/sched.hh:1358
#3  sched::thread::wait (this=this@entry=0xffff8000009f5040) at core/sched.cc:1273
#4  0x00000000403dbb01 in sched::thread::do_wait_until<sched::noninterruptible, sched::thread::dummy_lock, waiter::wait(sched::timer*) const::{lambda()#1}>(sched::thread::dummy_lock&, waiter::wait(sched::timer*) const::{lambda()#1}) (mtx=<synthetic pointer>..., pred=...) at include/osv/sched.hh:1114
#5  sched::thread::wait_until<waiter::wait(sched::timer*) const::{lambda()#1}>(waiter::wait(sched::timer*) const::{lambda()#1}) (pred=...) at include/osv/sched.hh:1125
#6  waiter::wait (tmr=0x200019901780, this=0x2000199016c0) at include/osv/wait_record.hh:46
#7  condvar::wait (this=this@entry=0xffffa00003fbe080, user_mutex=0xffffa00003eafb40, tmr=tmr@entry=0x200019901780) at core/condvar.cc:43
#8  0x000000004046110d in pthread_cond_timedwait(pthread_cond_t * __restrict__, pthread_mutex_t * __restrict__, const timespec * __restrict__) (cond=0xffffa00003ee5f50, mutex=0xffffa00003ee5f28, ts=<optimized out>) at libc/pthread.cc:623
#9  0x0000100000c8d85e in os::PlatformEvent::park(long) [clone .part.0] ()
#10 0x0000100000c8e966 in os::sleep(Thread*, long, bool) ()
#11 0x00001000009b61c7 in JVM_Sleep ()
#12 0x000020000920fe17 in ?? ()
#13 0x0000000800108530 in ?? ()
#14 0xffff9000041e0000 in ?? ()
#15 0x0000200019901990 in ?? ()
#16 0x0000000000000000 in ?? ()

The last two threads or actually the 2nd last (49 - SeedGenerator) is running and seems to be stuck in fill_ts<long, std::ratio<1, 1000000000> > (ts=0x2000197019a0, d=...) at /usr/include/c++/10/chrono:422. The fill_ts is an inline function implemented in libc/time.cc and I do not see how it can be stuck like this?

Is this some kind of a temporary deadlock situation?

[nyh]

fill_ts itself cannot really hang... It does write into user-supplied memory, and if that memory is mmaped we could get a page fault - but I'm not seeing any of that in the backtrace.
Is it possible that Java simply calls os::javaTimeNanos() in a loop, so when you interrupt it you see it hang int the time function? Can you try the same run again (or "c"ontinue it) and see if always hangs in the same place?

Here is my wild guess - it will need to be backed up by looking at actual Java 11 source code (which I didn't):

1. createTempFile() needs a random number generator to invent a random file name.
2. When the random number generator is first used, it waits for the "seed generator" to generate a seed - and until it does, createTempFile() just waits.
3. For some reason, the seed generator does a long loop inolving clock_gettime(), I have no idea why. Perhaps osv is missing some random-seed-related code (although we do have /dev/random...) which causes Java to fall back to some slow fallback mechanism to generate random entropy?

[nyh]

Some random sources on the internet suggests that the Java parameter -Djava.security.egd=file:/dev/./urandom will cause it to use /dev/urandom instead of a slow Java mechanism.
See for example https://www.baeldung.com/java-security-egd

[wkozaczuk]

It turns out this is simply caused by the missing conf subdirectory of JDK which contains java.security conf file. After adding it createTempFile() runs very quickly. I will be sending relevant patches to the modules directory.