Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arc: Allow IO on AUX reg handling (fixes icount on ARC) #11

Closed
wants to merge 19 commits into from

Conversation

abrodkin
Copy link
Member

@abrodkin abrodkin commented Feb 6, 2021

AUX registers are used in ARC processors to deal with settings or internal states of different internals like built-in timers, interrupt controller(s), caches etc.

Though for us here interrupts and timers are of the main interest as those are very good examples of IO operations and we do need explicitly allow it to make icount subsystem happy, as otherwise on the first attempt to set ARC built-in timer LIMIT register we see icount_get_raw_locked() barking: qemu-system-arc: Bad icount read, and that's because:

gdb --args ./build/qemu-system-arc ... -icount auto

...

(gdb) b icount.c:116
Breakpoint 1 at 0x54178a: file ../softmmu/icount.c, line 116.

(gdb) r

...

Thread 3 "qemu-system-arc" hit Breakpoint 1, icount_get_raw_locked () at ../softmmu/icount.c:116
116                 error_report("Bad icount read");
(gdb) bt
#0  icount_get_raw_locked () at ../softmmu/icount.c:116
#1  0x0000555555a957d5 in icount_get_locked () at ../softmmu/icount.c:128
#2  0x0000555555a9586d in icount_get () at ../softmmu/icount.c:154
#3  0x0000555555a901fd in tcg_get_virtual_clock () at ../accel/tcg/tcg-cpus.c:524
#4  0x0000555555a0fe05 in cpus_get_virtual_clock () at ../softmmu/cpus.c:211
#5  0x0000555555c88fd4 in qemu_clock_get_ns (type=QEMU_CLOCK_VIRTUAL) at ../util/qemu-timer.c:638
#6  0x00005555559aa8eb in cpu_arc_timer_update (env=0x5555565280a0, timer=0) at ../target/arc/timer.c:42
#7  0x00005555559ab272 in cpu_arc_store_limit (env=0x5555565280a0, timer=0, value=500000) at ../target/arc/timer.c:246
 #8  0x00005555559aba04 in aux_timer_set (aux_reg_detail=0x5555562eedb0 <arc_aux_regs_detail+2352>, val=500000, data=0x5555565280a0) at ../target/arc/timer.c:436
#9  0x00005555559a4cb2 in helper_sr (env=0x5555565280a0, val=500000, aux=35) at ../target/arc/op_helper.c:209
#10 0x00007fffb041dc6a in code_gen_buffer ()
#11 0x0000555555a2f6c8 in cpu_tb_exec (cpu=0x55555651f960, itb=0x7fffb041db00 <code_gen_buffer+4315859>) at ./accel/tcg/cpu-exec.c:178
#12 0x0000555555a304ae in cpu_loop_exec_tb (cpu=0x55555651f960, tb=0x7fffb041db00 <code_gen_buffer+4315859>, ast_tb=0x7ffff6013928, tb_exit=0x7ffff6013920)
     at ../accel/tcg/cpu-exec.c:658
#13 0x0000555555a307a6 in cpu_exec (cpu=0x55555651f960) at ../accel/tcg/cpu-exec.c:771
#14 0x0000555555a8f911 in tcg_cpu_exec (cpu=0x55555651f960) at ../accel/tcg/tcg-cpus.c:243
#15 0x0000555555a8fc12 in tcg_rr_cpu_thread_fn (arg=0x55555651f960) at ../accel/tcg/tcg-cpus.c:346
#16 0x0000555555c9562e in qemu_thread_start (args=0x5555565339e0) at ../util/qemu-thread-posix.c:521
#17 0x00007ffff7899609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007ffff77c0293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Fix that, hinting QEMU about possible IO on access of AUX regs, which are only possible via LR & SR instructions.

Cupertino Miranda and others added 18 commits November 11, 2020 12:18
Signed-off-by: Cupertino Miranda <[email protected]>
The decoder and the disassembler inspired by ARC GNU binutils.

Signed-off-by: Claudiu Zissulescu <[email protected]>
Signed-off-by: Claudiu Zissulescu <[email protected]>
Add the most generic parts of TCG constructions. It contains the
basic infrastructure for fundamental ARC features, such as
ZOL (zero overhead loops) and delay-slots.
Also includes hand crafted TCG for more intricate instructions, such
as vector instructions.

Signed-off-by: Shahab Vahedi <[email protected]>
TCG definitions as defined by our domain-specific-language (DSL) ISA
infrastructure.

Signed-off-by: Cupertino Miranda <[email protected]>
Add the infrastructure to define build configuration (BCR) and auxiliary
registers allowing independent modules (MMU, MPU, etc.) to use and extend
them.

Signed-off-by: Cupertino Miranda <[email protected]>
Signed-off-by: Claudiu Zissulescu <[email protected]>
Add Synopsys ARC MMU version 4 support. The implementation is
restricted to 8K page size support.

Signed-off-by: Cupertino Miranda <[email protected]>
Add memory implementation for Synopsys MPU unit version 3.
Synopsys MPU allows to create memory regions against unauthorized
execution/read/writes accesses.

Signed-off-by: Shahab Vahedi <[email protected]>
Register layout for the target and the mechanisms to read and set them.

Signed-off-by: Shahab Vahedi <[email protected]>
Add the Synopsys ARC boards, arc_sim for testing, sim-hs main emulation
board using standard UART and nsim which includes a Synopsys ARC specific
UART implementation.

Signed-off-by: Claudiu Zissulescu <[email protected]>
Add remaining bits of the Synopsys ARCv2 (EM/HS) support into QEMU,
configure bits, arch_init and configuration files for softmmu (hardware
emulation).

Signed-off-by: Shahab Vahedi <[email protected]>
Signed-off-by: Cupertino Miranda <[email protected]>
The added tests verify basic instructions execution as well
as more advanced features such as zero overhead loops interrupt
system, memory management unit and memory protection unit.

Signed-off-by: Claudiu Zissulescu <[email protected]>
Signed-off-by: Cupertino Miranda <[email protected]>
Just an acceptance test with ARC Linux booting.

Signed-off-by: Cupertino Miranda <[email protected]>
Nuke the qemu internal translation cache on a TLBWriteNI cmd, otherwise
we can end up with stale TLB entries.

Linux does this on an ASID rollover (when 8-bit ASID overflows)

The problem shows up on a kernel where TLB flush interfaces were dumbed
down to actually flush the TLB (with the TLBWriteNi cmd) as opposed to more
optimal ASID increment for large flush ranges as that is quicker on real
hardware (vs. iterating through the TLB and nuking entries one page worth
at a time).

Signed-off-by: Vineet Gupta <[email protected]>
Moves in_delar_slot variable to DisasCtxt structure that is thread safe.
This issue fixes assert happening in glibc testsuite execution.
Signed-off-by: Alexey Brodkin <[email protected]>
@abrodkin abrodkin changed the title Abrodkin icount arc: Allow IO on AUX reg handling (fixes icount on ARC) Feb 6, 2021
AUX registers are used in ARC processors to deal with settings or
internal states of different internals like built-in timers, interrupt
controller(s), caches etc.

Though for us here interrupts and timers are of the main interest as
those are very good examples of IO operations and we do need explicitly
allow it to make "icount" subsystem happy, as otherwise on the first
attempt to set ARC built-in timer LIMIT register we see icount_get_raw_locked()
barking: "qemu-system-arc: Bad icount read", and that's because:
------------------>8-----------------
| gdb --args ./build/qemu-system-arc ... -icount auto
|
| ...
|
| (gdb) b icount.c:116
| Breakpoint 1 at 0x54178a: file ../softmmu/icount.c, line 116.
|
| (gdb) r
|
| Thread 3 "qemu-system-arc" hit Breakpoint 1, icount_get_raw_locked () at ../softmmu/icount.c:116
| 116                 error_report("Bad icount read");
| (gdb) bt
| #0  icount_get_raw_locked () at ../softmmu/icount.c:116
| #1  0x0000555555a957d5 in icount_get_locked () at ../softmmu/icount.c:128
| #2  0x0000555555a9586d in icount_get () at ../softmmu/icount.c:154
| #3  0x0000555555a901fd in tcg_get_virtual_clock () at ../accel/tcg/tcg-cpus.c:524
| #4  0x0000555555a0fe05 in cpus_get_virtual_clock () at ../softmmu/cpus.c:211
| #5  0x0000555555c88fd4 in qemu_clock_get_ns (type=QEMU_CLOCK_VIRTUAL) at ../util/qemu-timer.c:638
| #6  0x00005555559aa8eb in cpu_arc_timer_update (env=0x5555565280a0, timer=0) at ../target/arc/timer.c:42
| #7  0x00005555559ab272 in cpu_arc_store_limit (env=0x5555565280a0, timer=0, value=500000) at ../target/arc/timer.c:246
| #8  0x00005555559aba04 in aux_timer_set (aux_reg_detail=0x5555562eedb0 <arc_aux_regs_detail+2352>, val=500000, data=0x5555565280a0) at ../target/arc/timer.c:436
| #9  0x00005555559a4cb2 in helper_sr (env=0x5555565280a0, val=500000, aux=35) at ../target/arc/op_helper.c:209
| #10 0x00007fffb041dc6a in code_gen_buffer ()
| #11 0x0000555555a2f6c8 in cpu_tb_exec (cpu=0x55555651f960, itb=0x7fffb041db00 <code_gen_buffer+4315859>) at ../accel/tcg/cpu-exec.c:178
| #12 0x0000555555a304ae in cpu_loop_exec_tb (cpu=0x55555651f960, tb=0x7fffb041db00 <code_gen_buffer+4315859>, last_tb=0x7ffff6013928, tb_exit=0x7ffff6013920)
|     at ../accel/tcg/cpu-exec.c:658
| #13 0x0000555555a307a6 in cpu_exec (cpu=0x55555651f960) at ../accel/tcg/cpu-exec.c:771
| #14 0x0000555555a8f911 in tcg_cpu_exec (cpu=0x55555651f960) at ../accel/tcg/tcg-cpus.c:243
| #15 0x0000555555a8fc12 in tcg_rr_cpu_thread_fn (arg=0x55555651f960) at ../accel/tcg/tcg-cpus.c:346
| #16 0x0000555555c9562e in qemu_thread_start (args=0x5555565339e0) at ../util/qemu-thread-posix.c:521
| #17 0x00007ffff7899609 in start_thread (arg=<optimized out>) at pthread_create.c:477
| #18 0x00007ffff77c0293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
------------------>8-----------------

Fix that, hinting QEMU about possible IO on access of AUX regs,
which are only possible via LR & SR instructions.

Signed-off-by: Alexey Brodkin <[email protected]>
@abrodkin
Copy link
Member Author

abrodkin commented Feb 8, 2021

FWIW here's a nice high-level explanation of icount implementation in QEMU: https://qemu.readthedocs.io/en/latest/devel/tcg-icount.html

@claziss
Copy link
Contributor

claziss commented Feb 8, 2021

Based on the link which you shared, your patch is not OK.

@claziss
Copy link
Contributor

claziss commented Feb 8, 2021

Accordingly to ur link:
image

@abrodkin
Copy link
Member Author

abrodkin commented Feb 8, 2021

@claziss well, the first question is how that action of "ending of the TB" should be done?
Also if I look at how others use that gen_io_start() (https://elixir.bootlin.com/qemu/latest/A/ident/gen_io_start), I cannot figure-out how it differs from what I implemented here - we just process a very specific instruction. Moreover given we do exist from the target code executing LR/SR handlers I'd assume TB ends there anyway. So what's wrong there then?

@abrodkin
Copy link
Member Author

@claziss what about existing usage of that same gen_io_start() in https://github.com/foss-for-synopsys-dwc-arc-processors/qemu/blob/master/target/arc/translate.c#L320?

static void arc_tr_insn_start(DisasContextBase *dcbase, CPUState *cpu)
{
    DisasContext *dc = container_of(dcbase, DisasContext, base);


    tcg_gen_insn_start(dc->base.pc_next);
    dc->cpc = dc->base.pc_next;

    if (dc->base.num_insns == dc->base.max_insns &&
        (dc->base.tb->cflags & CF_LAST_IO)) {
        gen_io_start();
    }
}

Does TB immediately ends there?

@cupertinomiranda cupertinomiranda force-pushed the master branch 2 times, most recently from 3bd198c to aed9f67 Compare February 24, 2021 11:52
cupertinomiranda pushed a commit that referenced this pull request Apr 5, 2021
Incoming enabled bitmaps are busy, because we do
bdrv_dirty_bitmap_create_successor() for them. But disabled bitmaps
being migrated are not marked busy, and user can remove them during the
incoming migration. Then we may crash in cancel_incoming_locked() when
try to remove the bitmap that was already removed by user, like this:

 #0  qemu_mutex_lock_impl (mutex=0x5593d88c50d1, file=0x559680554b20
   "../block/dirty-bitmap.c", line=64) at ../util/qemu-thread-posix.c:77
 #1  bdrv_dirty_bitmaps_lock (bs=0x5593d88c0ee9)
   at ../block/dirty-bitmap.c:64
 #2  bdrv_release_dirty_bitmap (bitmap=0x5596810e9570)
   at ../block/dirty-bitmap.c:362
 #3  cancel_incoming_locked (s=0x559680be8208 <dbm_state+40>)
   at ../migration/block-dirty-bitmap.c:918
 #4  dirty_bitmap_load (f=0x559681d02b10, opaque=0x559680be81e0
   <dbm_state>, version_id=1) at ../migration/block-dirty-bitmap.c:1194
 #5  vmstate_load (f=0x559681d02b10, se=0x559680fb5810)
   at ../migration/savevm.c:908
 #6  qemu_loadvm_section_part_end (f=0x559681d02b10,
   mis=0x559680fb4a30) at ../migration/savevm.c:2473
 #7  qemu_loadvm_state_main (f=0x559681d02b10, mis=0x559680fb4a30)
   at ../migration/savevm.c:2626
 #8  postcopy_ram_listen_thread (opaque=0x0)
   at ../migration/savevm.c:1871
 #9  qemu_thread_start (args=0x5596817ccd10)
   at ../util/qemu-thread-posix.c:521
 #10 start_thread () at /lib64/libpthread.so.0
 #11 clone () at /lib64/libc.so.6

Note bs pointer taken from bitmap: it's definitely bad aligned. That's
because we are in use after free, bitmap is already freed.

So, let's make disabled bitmaps (being migrated) busy during incoming
migration.

Signed-off-by: Vladimir Sementsov-Ogievskiy <[email protected]>
Signed-off-by: Stefan Hajnoczi <[email protected]>
Message-Id: <[email protected]>
cupertinomiranda pushed a commit that referenced this pull request Apr 5, 2021
When building with --enable-sanitizers we get:

  Direct leak of 32 byte(s) in 2 object(s) allocated from:
      #0 0x5618479ec7cf in malloc (qemu-system-aarch64+0x233b7cf)
      #1 0x7f675745f958 in g_malloc (/lib64/libglib-2.0.so.0+0x58958)
      #2 0x561847f02ca2 in usb_packet_init hw/usb/core.c:531:5
      #3 0x561848df4df4 in usb_ehci_init hw/usb/hcd-ehci.c:2575:5
      #4 0x561847c119ac in ehci_sysbus_init hw/usb/hcd-ehci-sysbus.c:73:5
      #5 0x56184a5bdab8 in object_init_with_type qom/object.c:375:9
      #6 0x56184a5bd955 in object_init_with_type qom/object.c:371:9
      #7 0x56184a5a2bda in object_initialize_with_type qom/object.c:517:5
      #8 0x56184a5a24d5 in object_initialize qom/object.c:536:5
      #9 0x56184a5a2f6c in object_initialize_child_with_propsv qom/object.c:566:5
      #10 0x56184a5a2e60 in object_initialize_child_with_props qom/object.c:549:10
      #11 0x56184a5a3a1e in object_initialize_child_internal qom/object.c:603:5
      #12 0x561849542d18 in npcm7xx_init hw/arm/npcm7xx.c:427:5

Similarly to commit d710e1e ("usb: ehci: fix memory leak in
ehci"), fix by calling usb_ehci_finalize() to free the USBPacket.

Fixes: 7341ea0
Signed-off-by: Philippe Mathieu-Daudé <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Gerd Hoffmann <[email protected]>
cupertinomiranda pushed a commit that referenced this pull request Jul 21, 2021
Incoming enabled bitmaps are busy, because we do
bdrv_dirty_bitmap_create_successor() for them. But disabled bitmaps
being migrated are not marked busy, and user can remove them during the
incoming migration. Then we may crash in cancel_incoming_locked() when
try to remove the bitmap that was already removed by user, like this:

 #0  qemu_mutex_lock_impl (mutex=0x5593d88c50d1, file=0x559680554b20
   "../block/dirty-bitmap.c", line=64) at ../util/qemu-thread-posix.c:77
 #1  bdrv_dirty_bitmaps_lock (bs=0x5593d88c0ee9)
   at ../block/dirty-bitmap.c:64
 #2  bdrv_release_dirty_bitmap (bitmap=0x5596810e9570)
   at ../block/dirty-bitmap.c:362
 #3  cancel_incoming_locked (s=0x559680be8208 <dbm_state+40>)
   at ../migration/block-dirty-bitmap.c:918
 #4  dirty_bitmap_load (f=0x559681d02b10, opaque=0x559680be81e0
   <dbm_state>, version_id=1) at ../migration/block-dirty-bitmap.c:1194
 #5  vmstate_load (f=0x559681d02b10, se=0x559680fb5810)
   at ../migration/savevm.c:908
 #6  qemu_loadvm_section_part_end (f=0x559681d02b10,
   mis=0x559680fb4a30) at ../migration/savevm.c:2473
 #7  qemu_loadvm_state_main (f=0x559681d02b10, mis=0x559680fb4a30)
   at ../migration/savevm.c:2626
 #8  postcopy_ram_listen_thread (opaque=0x0)
   at ../migration/savevm.c:1871
 #9  qemu_thread_start (args=0x5596817ccd10)
   at ../util/qemu-thread-posix.c:521
 #10 start_thread () at /lib64/libpthread.so.0
 #11 clone () at /lib64/libc.so.6

Note bs pointer taken from bitmap: it's definitely bad aligned. That's
because we are in use after free, bitmap is already freed.

So, let's make disabled bitmaps (being migrated) busy during incoming
migration.

Signed-off-by: Vladimir Sementsov-Ogievskiy <[email protected]>
Signed-off-by: Stefan Hajnoczi <[email protected]>
Message-Id: <[email protected]>
cupertinomiranda pushed a commit that referenced this pull request Jul 21, 2021
When building with --enable-sanitizers we get:

  Direct leak of 32 byte(s) in 2 object(s) allocated from:
      #0 0x5618479ec7cf in malloc (qemu-system-aarch64+0x233b7cf)
      #1 0x7f675745f958 in g_malloc (/lib64/libglib-2.0.so.0+0x58958)
      #2 0x561847f02ca2 in usb_packet_init hw/usb/core.c:531:5
      #3 0x561848df4df4 in usb_ehci_init hw/usb/hcd-ehci.c:2575:5
      #4 0x561847c119ac in ehci_sysbus_init hw/usb/hcd-ehci-sysbus.c:73:5
      #5 0x56184a5bdab8 in object_init_with_type qom/object.c:375:9
      #6 0x56184a5bd955 in object_init_with_type qom/object.c:371:9
      #7 0x56184a5a2bda in object_initialize_with_type qom/object.c:517:5
      #8 0x56184a5a24d5 in object_initialize qom/object.c:536:5
      #9 0x56184a5a2f6c in object_initialize_child_with_propsv qom/object.c:566:5
      #10 0x56184a5a2e60 in object_initialize_child_with_props qom/object.c:549:10
      #11 0x56184a5a3a1e in object_initialize_child_internal qom/object.c:603:5
      #12 0x561849542d18 in npcm7xx_init hw/arm/npcm7xx.c:427:5

Similarly to commit d710e1e ("usb: ehci: fix memory leak in
ehci"), fix by calling usb_ehci_finalize() to free the USBPacket.

Fixes: 7341ea0
Signed-off-by: Philippe Mathieu-Daudé <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Gerd Hoffmann <[email protected]>
@abrodkin
Copy link
Member Author

abrodkin commented Aug 4, 2021

@evgeniy-paltsev that's my dirty hack to enable icount on today's merged https://github.com/foss-for-synopsys-dwc-arc-processors/qemu/tree/arc64 tree.

Note it should be usable for both ARCv2 & ARCv3 64-bit QEMU binaries.

diff --git a/target/arc/semfunc-v3.c b/target/arc/semfunc-v3.c
index 20f7d76955..5755f90456 100644
--- a/target/arc/semfunc-v3.c
+++ b/target/arc/semfunc-v3.c
@@ -21,6 +21,7 @@
 #include "qemu/osdep.h"
 #include "translate.h"
 #include "semfunc-v3.h"
+#include "exec/gen-icount.h"



@@ -4229,6 +4230,8 @@ int
 arc_gen_LR (DisasCtxt *ctx, TCGv dest, TCGv src)
 {
   int ret = DISAS_NEXT;
+  if (tb_cflags(ctx->base.tb) & CF_USE_ICOUNT)
+      gen_io_start();
   TCGv temp_1 = tcg_temp_local_new();
   readAuxReg(temp_1, src);
   tcg_gen_andi_tl(temp_1, temp_1, 0xffffffff);
@@ -12016,6 +12019,8 @@ int
 arc_gen_LRL (DisasCtxt *ctx, TCGv dest, TCGv src)
 {
   int ret = DISAS_NEXT;
+  if (tb_cflags(ctx->base.tb) & CF_USE_ICOUNT)
+      gen_io_start();
   TCGv temp_1 = tcg_temp_local_new();
   readAuxReg(temp_1, src);
   tcg_gen_mov_tl(dest, temp_1);
diff --git a/target/arc/semfunc.c b/target/arc/semfunc.c
index 2ee63398c4..755a90967d 100644
--- a/target/arc/semfunc.c
+++ b/target/arc/semfunc.c
@@ -22,6 +22,7 @@
 #include "qemu/osdep.h"
 #include "translate.h"
 #include "target/arc/semfunc.h"
+#include "exec/gen-icount.h"

 /*
  * FLAG
@@ -3830,6 +3831,10 @@ int
 arc_gen_LR(DisasCtxt *ctx, TCGv dest, TCGv src)
 {
     int ret = DISAS_NEXT;
+
+    if (tb_cflags(ctx->base.tb) & CF_USE_ICOUNT)
+        gen_io_start();
+
     TCGv temp_1 = tcg_temp_local_new();
     readAuxReg(temp_1, src);
     tcg_gen_mov_tl(dest, temp_1);
diff --git a/target/arc/translate.c b/target/arc/translate.c
index 7c2e45dede..301b446ad4 100644
--- a/target/arc/translate.c
+++ b/target/arc/translate.c
@@ -24,6 +24,7 @@
 #include "tcg/tcg-op-gvec.h"
 #include "target/arc/semfunc.h"
 #include "target/arc/arc-common.h"
+#include "exec/gen-icount.h"

 /* Globals */
 TCGv    cpu_S1f;
@@ -745,6 +746,8 @@ arc_gen_SR(DisasCtxt *ctx, TCGv src2, TCGv src1)
 {
     int ret = DISAS_NEXT;

+    if (tb_cflags(ctx->base.tb) & CF_USE_ICOUNT)
+        gen_io_start();
 #if defined(TARGET_ARCV2)
     writeAuxReg(src2, src1);
 #elif defined(TARGET_ARCV3)
@@ -760,6 +763,8 @@ arc_gen_SRL(DisasCtxt *ctx, TCGv src2, TCGv src1)
 {
     int ret = DISAS_NEXT;

+    if (tb_cflags(ctx->base.tb) & CF_USE_ICOUNT)
+        gen_io_start();
     writeAuxReg(src2, src1);
     return ret;
 }

@cupertinomiranda
Copy link

cupertinomiranda commented Jan 25, 2022

QEMU support for icount has been done. This pull request is outdated based on current state.

@abrodkin abrodkin deleted the abrodkin-icount branch February 17, 2022 08:37
kolerov pushed a commit that referenced this pull request Jul 13, 2024
virtio_load() as a whole should run in coroutine context because it
reads from the migration stream and we don't want this to block.

However, it calls virtio_set_features_nocheck() and devices don't
expect their .set_features callback to run in a coroutine and therefore
call functions that may not be called in coroutine context. To fix this,
drop out of coroutine context for calling virtio_set_features_nocheck().

Without this fix, the following crash was reported:

  #0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
  #1  0x00007efc738c05d3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
  #2  0x00007efc73873d26 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
  #3  0x00007efc738477f3 in __GI_abort () at abort.c:79
  #4  0x00007efc7384771b in __assert_fail_base (fmt=0x7efc739dbcb8 "", assertion=assertion@entry=0x560aebfbf5cf "!qemu_in_coroutine()",
     file=file@entry=0x560aebfcd2d4 "../block/graph-lock.c", line=line@entry=275, function=function@entry=0x560aebfcd34d "void bdrv_graph_rdlock_main_loop(void)") at assert.c:92
  #5  0x00007efc7386ccc6 in __assert_fail (assertion=0x560aebfbf5cf "!qemu_in_coroutine()", file=0x560aebfcd2d4 "../block/graph-lock.c", line=275,
     function=0x560aebfcd34d "void bdrv_graph_rdlock_main_loop(void)") at assert.c:101
  #6  0x0000560aebcd8dd6 in bdrv_register_buf ()
  #7  0x0000560aeb97ed97 in ram_block_added.llvm ()
  #8  0x0000560aebb8303f in ram_block_add.llvm ()
  #9  0x0000560aebb834fa in qemu_ram_alloc_internal.llvm ()
  #10 0x0000560aebb2ac98 in vfio_region_mmap ()
  #11 0x0000560aebb3ea0f in vfio_bars_register ()
  #12 0x0000560aebb3c628 in vfio_realize ()
  #13 0x0000560aeb90f0c2 in pci_qdev_realize ()
  #14 0x0000560aebc40305 in device_set_realized ()
  #15 0x0000560aebc48e07 in property_set_bool.llvm ()
  #16 0x0000560aebc46582 in object_property_set ()
  #17 0x0000560aebc4cd58 in object_property_set_qobject ()
  #18 0x0000560aebc46ba7 in object_property_set_bool ()
  #19 0x0000560aeb98b3ca in qdev_device_add_from_qdict ()
  #20 0x0000560aebb1fbaf in virtio_net_set_features ()
  #21 0x0000560aebb46b51 in virtio_set_features_nocheck ()
  #22 0x0000560aebb47107 in virtio_load ()
  #23 0x0000560aeb9ae7ce in vmstate_load_state ()
  #24 0x0000560aeb9d2ee9 in qemu_loadvm_state_main ()
  #25 0x0000560aeb9d45e1 in qemu_loadvm_state ()
  #26 0x0000560aeb9bc32c in process_incoming_migration_co.llvm ()
  #27 0x0000560aebeace56 in coroutine_trampoline.llvm ()

Cc: [email protected]
Buglink: https://issues.redhat.com/browse/RHEL-832
Signed-off-by: Kevin Wolf <[email protected]>
Message-ID: <[email protected]>
Reviewed-by: Stefan Hajnoczi <[email protected]>
Signed-off-by: Kevin Wolf <[email protected]>
(cherry picked from commit 92e2e6a)
Signed-off-by: Michael Tokarev <[email protected]>
kolerov pushed a commit that referenced this pull request Jul 13, 2024
There is no architectural requirement that SME implies SVE, but
our implementation currently assumes it. (FEAT_SME_FA64 does
imply SVE.) So if you try to run a CPU with eg "-cpu max,sve=off"
you quickly run into an assert when the guest tries to write to
SMCR_EL1:

#6  0x00007ffff4b38e96 in __GI___assert_fail
    (assertion=0x5555566e69cb "sm", file=0x5555566e5b24 "../../target/arm/helper.c", line=6865, function=0x5555566e82f0 <__PRETTY_FUNCTION__.31> "sve_vqm1_for_el_sm") at ./assert/assert.c:101
#7  0x0000555555ee33aa in sve_vqm1_for_el_sm (env=0x555557d291f0, el=2, sm=false) at ../../target/arm/helper.c:6865
#8  0x0000555555ee3407 in sve_vqm1_for_el (env=0x555557d291f0, el=2) at ../../target/arm/helper.c:6871
#9  0x0000555555ee3724 in smcr_write (env=0x555557d291f0, ri=0x555557da23b0, value=2147483663) at ../../target/arm/helper.c:6995
#10 0x0000555555fd1dba in helper_set_cp_reg64 (env=0x555557d291f0, rip=0x555557da23b0, value=2147483663) at ../../target/arm/tcg/op_helper.c:839
#11 0x00007fff60056781 in code_gen_buffer ()

Avoid this unsupported and slightly odd combination by
disabling SME when SVE is not present.

Cc: [email protected]
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2005
Signed-off-by: Peter Maydell <[email protected]>
Reviewed-by: Richard Henderson <[email protected]>
Message-id: [email protected]
(cherry picked from commit f7767ca)
Signed-off-by: Michael Tokarev <[email protected]>
kolerov pushed a commit that referenced this pull request Jul 13, 2024
…ock_status

Using fleecing backup like in [0] on a qcow2 image (with metadata
preallocation) can lead to the following assertion failure:

> bdrv_co_do_block_status: Assertion `!(ret & BDRV_BLOCK_ZERO)' failed.

In the reproducer [0], it happens because the BDRV_BLOCK_RECURSE flag
will be set by the qcow2 driver, so the caller will recursively check
the file child. Then the BDRV_BLOCK_ZERO set too. Later up the call
chain, in bdrv_co_do_block_status() for the snapshot-access driver,
the assertion failure will happen, because both flags are set.

To fix it, clear the recurse flag after the recursive check was done.

In detail:

> #0  qcow2_co_block_status

Returns 0x45 = BDRV_BLOCK_RECURSE | BDRV_BLOCK_DATA |
BDRV_BLOCK_OFFSET_VALID.

> #1  bdrv_co_do_block_status

Because of the data flag, bdrv_co_do_block_status() will now also set
BDRV_BLOCK_ALLOCATED. Because of the recurse flag,
bdrv_co_do_block_status() for the bdrv_file child will be called,
which returns 0x16 = BDRV_BLOCK_ALLOCATED | BDRV_BLOCK_OFFSET_VALID |
BDRV_BLOCK_ZERO. Now the return value inherits the zero flag.

Returns 0x57 = BDRV_BLOCK_RECURSE | BDRV_BLOCK_DATA |
BDRV_BLOCK_OFFSET_VALID | BDRV_BLOCK_ALLOCATED | BDRV_BLOCK_ZERO.

> #2  bdrv_co_common_block_status_above
> #3  bdrv_co_block_status_above
> #4  bdrv_co_block_status
> #5  cbw_co_snapshot_block_status
> #6  bdrv_co_snapshot_block_status
> #7  snapshot_access_co_block_status
> #8  bdrv_co_do_block_status

Return value is propagated all the way up to here, where the assertion
failure happens, because BDRV_BLOCK_RECURSE and BDRV_BLOCK_ZERO are
both set.

> #9  bdrv_co_common_block_status_above
> #10 bdrv_co_block_status_above
> #11 block_copy_block_status
> #12 block_copy_dirty_clusters
> #13 block_copy_common
> #14 block_copy_async_co_entry
> #15 coroutine_trampoline

[0]:

> #!/bin/bash
> rm /tmp/disk.qcow2
> ./qemu-img create /tmp/disk.qcow2 -o preallocation=metadata -f qcow2 1G
> ./qemu-img create /tmp/fleecing.qcow2 -f qcow2 1G
> ./qemu-img create /tmp/backup.qcow2 -f qcow2 1G
> ./qemu-system-x86_64 --qmp stdio \
> --blockdev qcow2,node-name=node0,file.driver=file,file.filename=/tmp/disk.qcow2 \
> --blockdev qcow2,node-name=node1,file.driver=file,file.filename=/tmp/fleecing.qcow2 \
> --blockdev qcow2,node-name=node2,file.driver=file,file.filename=/tmp/backup.qcow2 \
> <<EOF
> {"execute": "qmp_capabilities"}
> {"execute": "blockdev-add", "arguments": { "driver": "copy-before-write", "file": "node0", "target": "node1", "node-name": "node3" } }
> {"execute": "blockdev-add", "arguments": { "driver": "snapshot-access", "file": "node3", "node-name": "snap0" } }
> {"execute": "blockdev-backup", "arguments": { "device": "snap0", "target": "node1", "sync": "full", "job-id": "backup0" } }
> EOF

Signed-off-by: Fiona Ebner <[email protected]>
Reviewed-by: Vladimir Sementsov-Ogievskiy <[email protected]>
Message-id: [email protected]
Signed-off-by: Stefan Hajnoczi <[email protected]>
(cherry picked from commit 8a9be79)
Signed-off-by: Michael Tokarev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants