sel4vm, guest_ram: fix compare function

The compare function is used with qsort to sort the guest ram regions. However, the compare function was returning an int, when the region checked is a uintptr_t. Subtracting these values and returning an int can cause undefined behavior. This commit fixes the compare function such that the return is compatible with qsort. Signed-off-by: Chris Guikema <[email protected]>
seL4 · Oct 26, 2023 · b86f908 · b86f908
1 parent deab837
commit b86f908
Showing 1 changed file with 9 additions and 1 deletion.
diff --git a/libsel4vm/src/guest_ram.c b/libsel4vm/src/guest_ram.c
@@ -48,7 +48,15 @@ static int ram_region_cmp(const void *a, const void *b)
 {
     const vm_ram_region_t *aa = a;
     const vm_ram_region_t *bb = b;
-    return aa->start - bb->start;
+
+    /* qsort expected return values */
+    if (aa->start == bb->start) {
+        return 0;
+    } else if (aa->start < bb->start) {
+        return -1;
+    } else {
+        return 1;
+    }
 }
 
 static void sort_guest_ram_regions(vm_mem_t *guest_memory)