Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Linux 5.17 drivers #236

Merged
merged 778 commits into from
Nov 27, 2023
Merged

Update to Linux 5.17 drivers #236

merged 778 commits into from
Nov 27, 2023

Conversation

dumbbell
Copy link
Member

@dumbbell dumbbell commented Feb 16, 2023

This is the backport of the DRM drivers from Linux 5.17.

Progress:

Changes in Linux 5.17

You can read this Phoronix article to learn about the changes in the DRM drivers in Linux 5.17:
https://www.phoronix.com/news/Linux-5.17-DRM-Submitted

Patches to linuxkpi

This update depends on the following patches to linuxkpi in FreeBSD:

These patches are maintained in the following repository and branch:
https://github.com/dumbbell/freebsd-src/tree/linuxkpi-updates-for-drm

How to test

You need to run a recent FreeBSD 14-CURRENT to test it.

Here are some instructions:

  1. You need to checkout the FreeBSD src branch I mentionned, linuxkpi-updates-for-drm, and compile a kernel from that branch:

    git clone -b linuxkpi-updates-for-drm https://github.com/dumbbell/freebsd-src.git
    cd freebsd-src
    make -j8 buildkernel DEBUG_FLAGS=-g
    
    # This installs the kernel under another name, `kernel.drm`. Thus, you keep the default kernel
    # in case of trouble.
    sudo make installkernel DEBUG_FLAGS=-g INSTKERNNAME=kernel.drm
  2. You need to checkout the branch referenced in this pull request and compile it:

    git clone -b update-to-v5.17 https://github.com/dumbbell/drm-kmod.git
    cd drm-kmod
    make -j8 DEBUG_FLAGS=-g
    sudo make install DEBUG_FLAGS=-g KMODDIR=/boot/kernel.drm
    

    This will need access to the FreeBSD src tree cloned above. I don't remember the name of the variable to point the build to it. You can link /usr/src to your clone and it will be enough.

  3. You will need GPU firmwares in the kernel.drm directory as well. To compile and install them:

    git clone https://github.com/freebsd/drm-kmod-firmware.git
    cd drm-kmod-firmware
    make -j8 DEBUG_FLAGS=-g OSVERSION=1400000
    sudo make install DEBUG_FLAGS=-g KMODDIR=/boot/kernel.drm OSVERSION=1400000
    
  4. Load the relevant driver(s) as you usually do.

@mekanix
Copy link

mekanix commented Feb 17, 2023

Tried to compile and run it and I get

# kldload drm
iic0: <I2C generic I/O> on iicbus0
link_elf_obj: symbol drm_sysfs_connector_status_event undefined
iic0: detached
iicbus0: <unknown card> at addr 0
Warning: memory type debugfsint leaked memory on destroy (2 allocations, 80 bytes leaked).
linker_load_file: /boot/kernel.drm/drm.ko - unsupported file type
kldload: an error occurred while loading module drm. Please check dmesg(8) for more details.

If it's too early for testing, just ignore me :o)

@dumbbell
Copy link
Member Author

The backport is finished, but polishing it will take a significant amount of work: I need to work on/revisit the framebuffer/vt(4) integration code. The amdgpu driver dropped its specific framebuffer code to use the generic code in drm_fb_helper which is commentted out on FreeBSD.

@mekanix
Copy link

mekanix commented Feb 21, 2023

Driver loads and works fine with Xorg and resume/suspend is without glitches on amdgpu laptop that I have.

@orbitz
Copy link

orbitz commented Feb 21, 2023

@mekanix I'm getting the error you did. What did you do to address it?

@mekanix
Copy link

mekanix commented Feb 22, 2023

I had to check again, and I was on a wrong branch, so same error as before.

@rhelmot
Copy link

rhelmot commented Mar 4, 2023

I ran into the same issue as above, so I tried to fix it. I added the following shot-in-the-dark patch to fix the linking error:

diff --git a/drivers/gpu/drm/drm_sysfs.c b/drivers/gpu/drm/drm_sysfs.c
index 036aaf7a28..312d538231 100644
--- a/drivers/gpu/drm/drm_sysfs.c
+++ b/drivers/gpu/drm/drm_sysfs.c
@@ -31,6 +31,7 @@
 #include <drm/drm_device.h>
 #include <drm/drm_print.h>
 #include <drm/drm_sysfs.h>
+#include <drm/drm_property.h>
 
 #include "drm_internal.h"
 
@@ -97,6 +98,23 @@ void drm_sysfs_connector_hotplug_event(struct drm_connector *connector)
        sbuf_delete(sb);
 }
 
+void drm_sysfs_connector_status_event(
+       struct drm_connector *connector, struct drm_property *property)
+{
+       struct drm_device *dev = connector->dev;
+       struct sbuf *sb = sbuf_new_auto();
+
+       DRM_DEBUG("generating status event\n");
+
+       sbuf_printf(sb, "cdev=dri/%s connector_id=%u connector_name=\"%s\""
+           "property_name=\"%s\"",
+           dev_name(dev->primary->kdev), connector->base.id, connector->name,
+           property->name);
+       sbuf_finish(sb);
+       devctl_notify("DRM", "CONNECTOR", "STATUS", sbuf_data(sb));
+       sbuf_delete(sb);
+}
+
 static void drm_sysfs_release(struct device *dev)
 {
        kfree(dev);

Then, drm loaded without issue but attempting to load i915kms seems to crash my entire system (alder lake-p). The screen goes black or freezes, nothing appears in the logs, remote sessions immediately hang, no crash dump is produced, and (to my eye) nothing shows up in the syslogs (booted verbose, logged over the network):

Mar  3 23:32:22 daisy kernel: iic0: <I2C generic I/O> on iicbus0
Mar  3 23:32:22 daisy kernel: lindebugfs registered
Mar  3 23:32:22 daisy kernel: drmn0: <drmn> on vgapci0
Mar  3 23:32:22 daisy kernel: vgapci0: child drmn0 requested pci_enable_io
Mar  3 23:32:22 daisy syslogd: last message repeated 1 times
Mar  3 23:32:22 daisy kernel: [drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19).
Mar  3 23:32:22 daisy kernel: [drm] Got stolen memory base 0x0, size 0x0
Mar  3 23:32:22 daisy kernel: vgapci0: attempting to allocate 1 MSI vectors (1 supported)
Mar  3 23:32:22 daisy kernel: msi: routing MSI IRQ 170 to local APIC 16 vector 50
Mar  3 23:32:22 daisy kernel: vgapci0: using IRQ 170 for MSI
Mar  3 23:32:22 daisy kernel: adlp_dmc_ver2_12.bin: could not load firmware image, error 2
Mar  3 23:32:22 daisy kernel: i915/adlp_dmc_ver2_12.bin: could not load firmware image, error 2
Mar  3 23:32:22 daisy kernel: i915_adlp_dmc_ver2_12.bin: could not load firmware image, error 2
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus dpa
Mar  3 23:32:22 daisy kernel: lkpi_iic0: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus1: <Philips I2C bus> on lkpi_iic0
Mar  3 23:32:22 daisy kernel: iic1: <I2C generic I/O>
Mar  3 23:32:22 daisy kernel:  on iicbus1
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus dpb
Mar  3 23:32:22 daisy kernel: lkpi_iic1: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus2: <Philips I2C bus> on lkpi_iic1
Mar  3 23:32:22 daisy kernel: iic2: <I2C generic I/O> on iicbus2
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus dpc
Mar  3 23:32:22 daisy kernel: lkpi_iic2: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus3: <Philips I2C bus> on lkpi_iic2
Mar  3 23:32:22 daisy kernel: iic3: <I2C generic I/O> on iicbus3
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus tc1
Mar  3 23:32:22 daisy kernel: lkpi_iic3: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus4: <Philips I2C bus> on lkpi_iic3
Mar  3 23:32:22 daisy kernel: iic4: <I2C generic I/O> on iicbus4
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus tc2
Mar  3 23:32:22 daisy kernel: lkpi_iic4: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus5: <Philips I2C bus> on lkpi_iic4
Mar  3 23:32:22 daisy kernel: iic5: <I2C generic I/O> on iicbus5
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus tc3
Mar  3 23:32:22 daisy kernel: lkpi_iic5: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus6: <Philips I2C bus> on lkpi_iic5
Mar  3 23:32:22 daisy kernel: iic6: <I2C generic I/O> on iicbus6
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus tc4
Mar  3 23:32:22 daisy kernel: lkpi_iic6: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus7: <Philips I2C bus> on lkpi_iic6
Mar  3 23:32:22 daisy kernel: iic7: <I2C generic I/O> on iicbus7
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus tc5
Mar  3 23:32:22 daisy kernel: lkpi_iic7: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus8: <Philips I2C bus> on lkpi_iic7
Mar  3 23:32:22 daisy kernel: iic8: <I2C generic I/O> on iicbus8
Mar  3 23:32:22 daisy kernel: drmn0: Adding i2c adapter i915 gmbus tc6
Mar  3 23:32:22 daisy kernel: lkpi_iic8: <LinuxKPI I2C> on drmn0
Mar  3 23:32:22 daisy kernel: iicbus9: <Philips I2C bus> on lkpi_iic8
Mar  3 23:32:22 daisy kernel: iic9: <I2C generic I/O> on iicbus9
Mar  3 23:32:22 daisy kernel: [drm] Connector eDP-1: get mode from tunables:
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.modes.eDP-1
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.default_mode
Mar  3 23:32:22 daisy kernel: firmware: 'i915_adlp_dmc_ver2_12_bin' version 0: 72104 bytes loaded at 0xffffffff85571000
Mar  3 23:32:22 daisy kernel: drmn0: successfully loaded firmware image 'i915/adlp_dmc_ver2_12.bin'
Mar  3 23:32:22 daisy kernel: drmn0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_12.bin (v2.12)
Mar  3 23:32:22 daisy kernel: [drm] Connector HDMI-A-1: get mode from tunables:
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.modes.HDMI-A-1
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.default_mode
Mar  3 23:32:22 daisy kernel: [drm] Connector DP-1: get mode from tunables:
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.modes.DP-1
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.default_mode
Mar  3 23:32:22 daisy kernel: [drm] Connector DP-2: get mode from tunables:
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.modes.DP-2
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.default_mode
Mar  3 23:32:22 daisy kernel: [drm] Connector DP-3: get mode from tunables:
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.modes.DP-3
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.default_mode
Mar  3 23:32:22 daisy kernel: [drm] Connector DP-4: get mode from tunables:
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.modes.DP-4
Mar  3 23:32:22 daisy kernel: [drm]  - kern.vt.fb.default_mode

I would like to be useful on this project! Please let me know how I can help or how I can begin debugging this issue.

@rhelmot
Copy link

rhelmot commented Mar 4, 2023

Immediately after I sent this, I figured out that after the system hangs, I can trigger the break-to-debugger keystroke and type in the dump command even though the display isn't working. I now have a core dump from after the driver is loaded. I see a bunch of i915 threads and I will look into what they're doing, but help would still be appreciated :)

@rhelmot
Copy link

rhelmot commented Mar 5, 2023

Discovered the DRM_DEBUG_LOG_ALL flag. Here's the logs from that:

/var/log/messages exerpt
Mar  5 00:59:02 daisy kernel: iic0: <I2C generic I/O> on iicbus0
Mar  5 00:59:02 daisy kernel: [drm] Initialized
Mar  5 00:59:02 daisy kernel: drmn0: <drmn> on vgapci0
Mar  5 00:59:02 daisy kernel: vgapci0: child drmn0 requested pci_enable_io
Mar  5 00:59:02 daisy syslogd: last message repeated 1 times
Mar  5 00:59:02 daisy kernel: drmn0: intel_wopcm_init_early: WOPCM: 2048K
Mar  5 00:59:02 daisy kernel: drmn0: __confirm_options: enable_guc=3 (guc:yes submission:yes huc:yes slpc:yes)
Mar  5 00:59:02 daisy kernel: drmn0: intel_pch_type: Found Alder Lake PCH
Mar  5 00:59:02 daisy kernel: drmn0: get_allowed_dc_mask: Allowed DC state mask 08
Mar  5 00:59:02 daisy kernel: drmn0: intel_uncore_init_mmio: unclaimed mmio detected on uncore init, clearing
Mar  5 00:59:02 daisy kernel: drmn0: intel_device_info_runtime_init: rawclk rate: 19200 kHz
Mar  5 00:59:02 daisy kernel: drmn0: init_engine_mask: vdbox enable: 0005, instances: 0005
Mar  5 00:59:02 daisy kernel: drmn0: init_engine_mask: vebox enable: 0001, instances: 0001
Mar  5 00:59:02 daisy kernel: drmn0: ggtt_probe_hw: GGTT size = 4096M
Mar  5 00:59:02 daisy kernel: drmn0: ggtt_probe_hw: GMADR size = 256M
Mar  5 00:59:02 daisy kernel: drmn0: ggtt_probe_hw: DSM size = 0M
Mar  5 00:59:02 daisy kernel: [drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19).
Mar  5 00:59:02 daisy kernel: [drm] Got stolen memory base 0x0, size 0x0
Mar  5 00:59:02 daisy kernel: drmn0: intel_opregion_setup: graphic opregion physical addr: 0x8d64e018
Mar  5 00:59:02 daisy kernel: drmn0: intel_opregion_setup: ACPI OpRegion version 2.1.0
Mar  5 00:59:02 daisy kernel: drmn0: intel_opregion_setup: Public ACPI methods supported
Mar  5 00:59:02 daisy kernel: drmn0: intel_opregion_setup: ASLE supported
Mar  5 00:59:02 daisy kernel: drmn0: intel_opregion_setup: ASLE extension supported
Mar  5 00:59:02 daisy kernel: drmn0: intel_opregion_setup: Found valid VBT in ACPI OpRegion (RVDA)
Mar  5 00:59:02 daisy kernel: drmn0: intel_dram_detect: DRAM channels: 4
Mar  5 00:59:02 daisy kernel: drmn0: intel_dram_detect: Watermark level 0 adjustment needed: no
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: QGV 0: DCLK=2401 tRP=24 tRDPRE=20 tRAS=52 tRCD=24 tRC=76
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: QGV 1: DCLK=4801 tRP=44 tRDPRE=28 tRAS=104 tRCD=44 tRC=148
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: QGV 2: DCLK=5201 tRP=48 tRDPRE=28 tRAS=112 tRCD=48 tRC=160
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: QGV 3: DCLK=5201 tRP=48 tRDPRE=28 tRAS=112 tRCD=48 tRC=160
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: PSF GV 0: CLK=32 
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: PSF GV 1: CLK=48 
Mar  5 00:59:02 daisy kernel: drmn0: icl_get_qgv_points: PSF GV 2: CLK=48 
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / QGV 0: num_planes=0 deratedbw=16956
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / QGV 1: num_planes=0 deratedbw=23981
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / QGV 2: num_planes=0 deratedbw=24772
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / QGV 3: num_planes=0 deratedbw=24772
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / PSF GV 0: num_planes=0 bw=34133
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / PSF GV 1: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW0 / PSF GV 2: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / QGV 0: num_planes=4 deratedbw=21855
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / QGV 1: num_planes=4 deratedbw=34500
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / QGV 2: num_planes=4 deratedbw=36108
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / QGV 3: num_planes=4 deratedbw=36108
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / PSF GV 0: num_planes=4 bw=34133
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / PSF GV 1: num_planes=4 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW1 / PSF GV 2: num_planes=4 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / QGV 0: num_planes=0 deratedbw=25544
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / QGV 1: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / QGV 2: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / QGV 3: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / PSF GV 0: num_planes=0 bw=34133
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / PSF GV 1: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW2 / PSF GV 2: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / QGV 0: num_planes=0 deratedbw=27900
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / QGV 1: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / QGV 2: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / QGV 3: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / PSF GV 0: num_planes=0 bw=34133
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / PSF GV 1: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW3 / PSF GV 2: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / QGV 0: num_planes=0 deratedbw=29248
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / QGV 1: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / QGV 2: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / QGV 3: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / PSF GV 0: num_planes=0 bw=34133
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / PSF GV 1: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW4 / PSF GV 2: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / QGV 0: num_planes=0 deratedbw=29972
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / QGV 1: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / QGV 2: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / QGV 3: num_planes=0 deratedbw=38000
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / PSF GV 0: num_planes=0 bw=34133
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / PSF GV 1: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: tgl_get_bw_info: BW5 / PSF GV 2: num_planes=0 bw=51200
Mar  5 00:59:02 daisy kernel: drmn0: init_vbt_defaults: Set default to SSC at 120000 kHz
Mar  5 00:59:02 daisy kernel: drmn0: intel_bios_init: VBT signature "$VBT ALDERLAKE-P    ", BDB version 249
Mar  5 00:59:02 daisy kernel: drmn0: parse_general_features: BDB_GENERAL_FEATURES int_tv_support 0 int_crt_support 0 lvds_use_ssc 0 lvds_ssc_freq 120000 display_clock_mode 1 fdi_rx_polarity_inverted 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_general_definitions: crt_ddc_bus_pin: 2
Mar  5 00:59:02 daisy kernel: drmn0: parse_general_definitions: Expected child device config size for VBT version 249 not known; assuming 39
Mar  5 00:59:02 daisy kernel: drmn0: parse_general_definitions: Found VBT child device with type 0x1806
Mar  5 00:59:02 daisy kernel: drmn0: parse_general_definitions: Found VBT child device with type 0x60d2
Mar  5 00:59:02 daisy kernel: drmn0: parse_general_definitions: Found VBT child device with type 0x68c6
Mar  5 00:59:02 daisy syslogd: last message repeated 3 times
Mar  5 00:59:02 daisy kernel: drmn0: parse_panel_options: Invalid VBT panel type 0xff
Mar  5 00:59:02 daisy kernel: drmn0: parse_lfp_backlight: VBT backlight PWM modulation frequency 990 Hz, active high, min brightness 3, level 21845, controller 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_sdvo_device_mapping: Skipping SDVO device mapping
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port A VBT info: CRT:0 DVI:0 HDMI:0 DP:1 eDP:1 LSPCON:0 USB-Type-C:0 TBT:0 DSC:0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port A VBT HDMI level shift: 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port B VBT info: CRT:0 DVI:1 HDMI:1 DP:0 eDP:0 LSPCON:0 USB-Type-C:0 TBT:0 DSC:0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port B VBT HDMI level shift: 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port D VBT info: CRT:0 DVI:0 HDMI:0 DP:1 eDP:0 LSPCON:0 USB-Type-C:1 TBT:1 DSC:0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port D VBT HDMI level shift: 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port E VBT info: CRT:0 DVI:0 HDMI:0 DP:1 eDP:0 LSPCON:0 USB-Type-C:1 TBT:1 DSC:0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port E VBT HDMI level shift: 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port F VBT info: CRT:0 DVI:0 HDMI:0 DP:1 eDP:0 LSPCON:0 USB-Type-C:1 TBT:1 DSC:0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port F VBT HDMI level shift: 0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port G VBT info: CRT:0 DVI:0 HDMI:0 DP:1 eDP:0 LSPCON:0 USB-Type-C:1 TBT:1 DSC:0
Mar  5 00:59:02 daisy kernel: drmn0: parse_ddi_port: Port G VBT HDMI level shift: 0
Mar  5 00:59:02 daisy kernel: drmn0: gen9_set_dc_state: Setting DC state from 00 to 00
Mar  5 00:59:02 daisy kernel: drmn0: check_phy_reg: Combo PHY A reg 001628a0 state mismatch: current 90033bbc mask e0000000 expected a0000000
Mar  5 00:59:02 daisy kernel: drmn0: check_phy_reg: Combo PHY A reg 00162804 state mismatch: current 1c000004 mask 00300000 expected 00300000
Mar  5 00:59:02 daisy kernel: drmn0: check_phy_reg: Combo PHY B reg 0006c8a0 state mismatch: current 9003501c mask e0000000 expected a0000000
Mar  5 00:59:02 daisy kernel: drmn0: check_phy_reg: Combo PHY B reg 0006c804 state mismatch: current 1c000004 mask 00300000 expected 00300000
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling power well 1
Mar  5 00:59:02 daisy kernel: [drm] Current CDCLK 307200 kHz, VCO 614400 kHz, ref 38400 kHz, bypass 19200 kHz, voltage level 0
Mar  5 00:59:02 daisy kernel: drmn0: gen9_dbuf_slices_update: Updating dbuf slices to 0xf
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling always-on
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling DC off
Mar  5 00:59:02 daisy kernel: drmn0: gen9_set_dc_state: Setting DC state from 00 to 00
Mar  5 00:59:02 daisy kernel: drmn0: icl_combo_phys_init: Combo PHY A already enabled, won't reprogram it.
Mar  5 00:59:02 daisy kernel: drmn0: icl_combo_phys_init: Combo PHY B already enabled, won't reprogram it.
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling power well 2
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling power well A
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling power well B
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling power well C
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling power well D
Mar  5 00:59:02 daisy kernel: drmn0: intel_dmc_ucode_init: Loading i915/adlp_dmc_ver2_12.bin
Mar  5 00:59:02 daisy kernel: drmn0: intel_fbc_init: Sanitized enable_fbc value: 0
Mar  5 00:59:02 daisy kernel: drmn0: intel_print_wm_latency: Gen9 Plane WM0 latency 3 (3.0 usec)
Mar  5 00:59:02 daisy kernel: drmn0: intel_print_wm_latency: Gen9 Plane WM1 latency 54 (54.0 usec)
Mar  5 00:59:02 daisy kernel: drmn0: intel_print_wm_latency: Gen9 Plane WM2 latency 83 (83.0 usec)
Mar  5 00:59:02 daisy kernel: drmn0: intel_print_wm_latency: Gen9 Plane WM3 latency 102 (102.0 usec)
Mar  5 00:59:02 daisy kernel: drmn0: intel_print_wm_latency: Gen9 Plane WM4 latency 147 (147.0 usec)
Mar  5 00:59:02 daisy kernel: drmn0: intel_print_wm_latency: Gen9 Plane WM5 latency 147 (147.0 usec)
Mar  5 00:59:02 daisy kernel: lkpi_iic0: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus1: <Philips I2C bus> on lkpi_iic0
Mar  5 00:59:02 daisy kernel: iic1: <I2C generic I/O> on iicbus1
Mar  5 00:59:02 daisy kernel: lkpi_iic1: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus2: <Philips I2C bus> on lkpi_iic1
Mar  5 00:59:02 daisy kernel: iic2: <I2C generic I/O> on iicbus2
Mar  5 00:59:02 daisy kernel: lkpi_iic2: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus3: <Philips I2C bus> on lkpi_iic2
Mar  5 00:59:02 daisy kernel: iic3: <I2C generic I/O> on iicbus3
Mar  5 00:59:02 daisy kernel: lkpi_iic3: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus4: <Philips I2C bus> on lkpi_iic3
Mar  5 00:59:02 daisy kernel: iic4: <I2C generic I/O> on iicbus4
Mar  5 00:59:02 daisy kernel: lkpi_iic4: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus5: <Philips I2C bus> on lkpi_iic4
Mar  5 00:59:02 daisy kernel: iic5: <I2C generic I/O> on iicbus5
Mar  5 00:59:02 daisy kernel: lkpi_iic5: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus6: <Philips I2C bus> on lkpi_iic5
Mar  5 00:59:02 daisy kernel: iic6: <I2C generic I/O> on iicbus6
Mar  5 00:59:02 daisy kernel: lkpi_iic6: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus7: <Philips I2C bus> on lkpi_iic6
Mar  5 00:59:02 daisy kernel: iic7: <I2C generic I/O> on iicbus7
Mar  5 00:59:02 daisy kernel: lkpi_iic7: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus8: <Philips I2C bus> on lkpi_iic7
Mar  5 00:59:02 daisy kernel: iic8: <I2C generic I/O> on iicbus8
Mar  5 00:59:02 daisy kernel: lkpi_iic8: <LinuxKPI I2C> on drmn0
Mar  5 00:59:02 daisy kernel: iicbus9: <Philips I2C bus> on lkpi_iic8
Mar  5 00:59:02 daisy kernel: iic9: <I2C generic I/O> on iicbus9
Mar  5 00:59:02 daisy kernel: drmn0: intel_modeset_init_nogem: 4 display pipes available.
Mar  5 00:59:02 daisy kernel: [drm] Current CDCLK 307200 kHz, VCO 614400 kHz, ref 38400 kHz, bypass 19200 kHz, voltage level 0
Mar  5 00:59:02 daisy kernel: drmn0: intel_update_max_cdclk: Max CD clock rate: 652800 kHz
Mar  5 00:59:02 daisy kernel: drmn0: intel_update_max_cdclk: Max dotclock rate: 1305600 kHz
Mar  5 00:59:02 daisy kernel: drmn0: intel_bios_port_aux_ch: using AUX A for port A (VBT)
Mar  5 00:59:02 daisy kernel: drmn0: intel_dp_init_connector: Adding eDP connector on [ENCODER:235:DDI A/PHY A]
Mar  5 00:59:02 daisy kernel: [drm] cur t1_t3 1 t8 1 t9 1 t10 500 t11_t12 6000
Mar  5 00:59:02 daisy kernel: [drm] vbt t1_t3 2000 t8 10 t9 2600 t10 500 t11_t12 6000
Mar  5 00:59:02 daisy kernel: drmn0: pps_init_delays: panel power up delay 200, power down delay 50, power cycle delay 600
Mar  5 00:59:02 daisy kernel: drmn0: pps_init_delays: backlight on delay 1, off delay 260
Mar  5 00:59:02 daisy kernel: drmn0: pps_init_registers: panel power sequencer register settings: PP_ON 0x7d00001, PP_OFF 0x1f40001, PP_DIV 0x60
Mar  5 00:59:02 daisy kernel: drmn0: intel_power_well_enable: enabling AUX A
Mar  5 00:59:02 daisy kernel: drmn0: intel_pps_vdd_on_unlocked: Turning [ENCODER:235:DDI A/PHY A] VDD on
Mar  5 00:59:02 daisy kernel: drmn0: intel_pps_vdd_on_unlocked: PP_STATUS: 0x80000008 PP_CONTROL: 0x0000006f
Mar  5 00:59:02 daisy kernel: drmn0: successfully loaded firmware image 'i915/adlp_dmc_ver2_12.bin'
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00000 AUX -> (ret= 15) fffff802de397183h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x02200 AUX -> (ret= 15) fffffe014a2ea5f1h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_read_dpcd_caps: AUX A/DDI A/PHY A: DPCD: fffff802de397183h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00400 AUX -> (ret= 12) fffff802de397250h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_read_desc: AUX A/DDI A/PHY A: DP sink: OUI fffff802de397250hD dev-ID 0xfffff802de397253E HW-rev 0.0 SW-rev 1.253 quirks 0x0000
Mar  5 00:59:02 daisy kernel: drmn0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_12.bin (v2.12)
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00700 AUX -> (ret=  3) fffff802de3971a4h
Mar  5 00:59:02 daisy kernel: drmn0: intel_edp_init_dpcd: eDP DPCD: fffff802de3971a4h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00070 AUX -> (ret=  2) fffff802de397192h
Mar  5 00:59:02 daisy kernel: drmn0: intel_psr_init_dpcd: eDP panel supports PSR version 3
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x02009 AUX -> (ret=  1) fffffe014a2ea5f7h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x0002e AUX -> (ret=  1) fffffe014a2ea5f6h
Mar  5 00:59:02 daisy kernel: drmn0: intel_psr_init_dpcd: PSR2 supported
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x02210 AUX -> (ret=  1) fffffe014a2ea5dfh
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00072 AUX -> (ret=  2) fffffe014a2ea5f4h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00074 AUX -> (ret=  1) fffffe014a2ea5f3h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00010 AUX -> (ret= 16) fffffe014a2ea670h
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00060 AUX -> (ret= 15) fffff802de3971a7h
Mar  5 00:59:02 daisy kernel: drmn0: intel_dp_get_dsc_sink_cap: DSC DPCD: fffff802de3971a7h
Mar  5 00:59:02 daisy kernel: drmn0: intel_dp_get_dsc_sink_cap: FEC CAPABILITY: 0
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x00300 AUX -> (ret=  3) fffffe014a2ea68ch
Mar  5 00:59:02 daisy kernel: [drm] Supported Monitor Refresh rate range is 40 Hz - 60 Hz
Mar  5 00:59:02 daisy kernel: [drm] non_desktop set to 0
Mar  5 00:59:02 daisy kernel: [drm] eDP-1: Assigning EDID-1.4 digital sink color depth as 10 bpc.
Mar  5 00:59:02 daisy kernel: [drm] ELD monitor 
Mar  5 00:59:02 daisy kernel: [drm] ELD size 20, SAD count 0
Mar  5 00:59:02 daisy kernel: [drm] Supported Monitor Refresh rate range is 40 Hz - 60 Hz
Mar  5 00:59:02 daisy kernel: [drm] non_desktop set to 0
Mar  5 00:59:02 daisy kernel: [drm] eDP-1: Assigning EDID-1.4 digital sink color depth as 10 bpc.
Mar  5 00:59:02 daisy kernel: [drm] Supported Monitor Refresh rate range is 40 Hz - 60 Hz
Mar  5 00:59:02 daisy kernel: [drm] non_desktop set to 0
Mar  5 00:59:02 daisy kernel: [drm] eDP-1: Assigning EDID-1.4 digital sink color depth as 10 bpc.
Mar  5 00:59:02 daisy kernel: [drm] OBJ ID: 237 (1)
Mar  5 00:59:02 daisy kernel: drmn0: intel_panel_edid_fixed_mode: [CONNECTOR:236:eDP-1] using preferred mode from EDID: [drm] Modeline "3840x2400": 60 596200 3840 3888 3920 4000 2400 2403 2408 2484 0x48 0xa
Mar  5 00:59:02 daisy kernel: drmn0: intel_drrs_init: VBT doesn't support DRRS
Mar  5 00:59:02 daisy kernel: drmn0: drm_dp_dump_access: AUX A/DDI A/PHY A: 0x007a4 AUX -> (ret=  1) fffffe014a2ea670h
Mar  5 00:59:02 daisy kernel: drmn0: intel_dp_wait_source_oui: Performing OUI wait
Mar  5 00:59:05 daisy kernel: drmn0: intel_pps_vdd_off_sync_unlocked: Turning [ENCODER:235:DDI A/PHY A] VDD off
Mar  5 00:59:05 daisy kernel: drmn0: intel_pps_vdd_off_sync_unlocked: PP_STATUS: 0x80000008 PP_CONTROL: 0x00000067
Mar  5 00:59:05 daisy kernel: drmn0: intel_power_well_disable: disabling AUX A

@dumbbell
Copy link
Member Author

dumbbell commented Mar 6, 2023

Thank you @rhelmot for testing! Sorry I didn't push everything to both my freebsd-src and drm-kmod forks as it's really not working at all currently :-) I made several temporary/testing commits that are not made to be pushed to GitHub...

Once I have something closer to a testable version, I will push again and ask for more help :-)

@rhelmot
Copy link

rhelmot commented Mar 6, 2023

Thanks for the update. Is there anything I can do to facilitate development?

Also, what revision of 5.17 are you targeting? I tried debugging this by comparing the boot logs to my linux 5.17 install on this laptop (ubuntu) and the thing that popped out at me was that linux is using dmc firmware v2.14 and this port seems to want to use v2.12. When I ran the diff tool myself, I found a huge number of unported changes, many of which look like correctness fixes. I attempted to port all the changes I could find which seemed relevant to alder lake, and upgraded the firmware version, but haven't gotten any different results.

@dumbbell
Copy link
Member Author

Patches come from Linux v5.17 tag. Bug fixes backported later are therefore not included.

I selfishly admit I would like to find the problem myself for the challenge and "adrenaline". I'm working on the backport since Linux 5.11 patches and this has been uneventful so far. I kind of need/want to go through this and find the issue.

@dumbbell
Copy link
Member Author

Quick update on the backport of 5.17.

  1. Console is broken for both amdgpu and i915kms, not only the former. I'm working on our vt(4) integration layer to make it behave like Linux' fbdev.
  2. The GuC code in i915kms does not work on FreeBSD. This is enabled by default for Intel Gen12+ but can be disabled. For now, it's the only way to successfully load the driver on these iGPUs. This is equally broken in the 5.15-lts branch by the way.

@dumbbell
Copy link
Member Author

The rewrite of the vt(4) integration layer is almost finished, but it's already working for me; see #243.

The Linux 5.17 backport branch was rebased on top of it. It will be required anyway.

@timp87
Copy link

timp87 commented Mar 17, 2023

@dumbbell thank you so much for your hard work!
I have a question with 14 CURRENT becoming STABLE soon.
What is your general plan for now? Are you going to stop porting linux drm for some time once 5.17 is done?
Or you plan to keep doing that until some point, for example, prepare Linux 6.1 port for FreeBSD 14 as that one is LTS?

@dumbbell
Copy link
Member Author

For now, I still need to work on 5.17 which is a lot harder to port compared to all previous versions.

Once this is done, I'm not sure. Probably, I'll continue with 5.18, 5.19 and so on. That would be really cool to be in sync with Linux!

I'm glad that it was easier than expected to fix the vt(4) integration, though there are still a few things left.

@mekanix
Copy link

mekanix commented Mar 18, 2023

After loading amdgpu, my laptop seams frozen. SSH works, so I can do stuff to figure out what's going on. Relevant pciconf -lv portion:

vgapci0@pci0:7:0:0:	class=0x030000 rev=0xd1 hdr=0x00 vendor=0x1002 device=0x1638 subvendor=0x17aa subdevice=0x509c
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Cezanne [Radeon Vega Series / Radeon Vega Mobile Series]'
    class      = display
    subclass   = VGA

@dumbbell
Copy link
Member Author

I pushed a commit to disable GuC by default as it's clearly causing the computer to freeze when i915kms is loaded. With GuC disabled on the vt(4) integration rewritten in #243 (which this branch is based on), it looks to work with Intel 12th gen GPUs (I tested that very lightly for now).

I also noticed that a function returns an error during amdgpu init. It always did but the error was ignored so far. I will investigate.

@dumbbell
Copy link
Member Author

Sway works fine on 5.15-lts on my Intel 12th gen GPU (i7-1260p), but not on 5.17 where I get a black screen with a working mouse cursor (and lots of errors in /var/log/messages).

@JustAnotherHumanBeing
Copy link

JustAnotherHumanBeing commented Mar 20, 2023

Under a GENERIC kernel, I experience this crash:

panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) linux_wq_exec @ /root/freebsd-src/sys/compat/linuxkpi/common/src/linux_work.c:105

GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) linux_wq_exec @ /root/freebsd-src/sys/compat/linuxkpi/common/src/linux_work.c:105
cpuid = 0
time = 1679351346
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe015c42ca60
vpanic() at vpanic+0x152/frame 0xfffffe015c42cab0
panic() at panic+0x43/frame 0xfffffe015c42cb10
witness_checkorder() at witness_checkorder+0xc5c/frame 0xfffffe015c42ccd0
__mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe015c42cd20
linux_queue_work_on() at linux_queue_work_on+0x9e/frame 0xfffffe015c42cd60
vt_drmfb_drawrect() at vt_drmfb_drawrect+0x5b/frame 0xfffffe015c42cd90
vt_flush() at vt_flush+0x2c4/frame 0xfffffe015c42cdf0
vt_timer() at vt_timer+0xe/frame 0xfffffe015c42ce10
softclock_call_cc() at softclock_call_cc+0x14b/frame 0xfffffe015c42cec0
softclock_thread() at softclock_thread+0xc6/frame 0xfffffe015c42cef0
fork_exit() at fork_exit+0x80/frame 0xfffffe015c42cf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe015c42cf30
--- trap 0xff000000, rip = 0xff000000ff000000, rsp = 0xff000000ff000000, rbp = 0xff000000ff000000 ---
Uptime: 4m50s
Dumping 2503 out of 65301 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

warning: Could not load shared library symbols for i915kms.ko.
Do you need "set solib-search-path" or "set sysroot"?
__curthread () at /root/freebsd-src/sys/amd64/include/pcpu_aux.h:59
59		__asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /root/freebsd-src/sys/amd64/include/pcpu_aux.h:59
#1  doadump (textdump=textdump@entry=1)
    at /root/freebsd-src/sys/kern/kern_shutdown.c:407
#2  0xffffffff80bedc7c in kern_reboot (howto=260)
    at /root/freebsd-src/sys/kern/kern_shutdown.c:528
#3  0xffffffff80bee19f in vpanic (fmt=<optimized out>, 
    ap=ap@entry=0xfffffe015c42caf0)
    at /root/freebsd-src/sys/kern/kern_shutdown.c:972
#4  0xffffffff80bedf23 in panic (fmt=<unavailable>)
    at /root/freebsd-src/sys/kern/kern_shutdown.c:896
#5  0xffffffff80c5fbcc in witness_checkorder (lock=<optimized out>, flags=9, 
    file=file@entry=0xffffffff811c4cfa "/root/freebsd-src/sys/compat/linuxkpi/common/src/linux_work.c", line=line@entry=105, interlock=interlock@entry=0x0)
    at /root/freebsd-src/sys/kern/subr_witness.c:1141
#6  0xffffffff80bc8784 in __mtx_lock_flags (c=0xfffff80001e1e5e0, opts=0, 
    file=0xffffffff811c4cfa "/root/freebsd-src/sys/compat/linuxkpi/common/src/linux_work.c", line=105) at /root/freebsd-src/sys/kern/kern_mutex.c:278
#7  0xffffffff80e8650e in linux_work_exec_unblock (work=0xfffff8019c5162f0)
    at /root/freebsd-src/sys/compat/linuxkpi/common/src/linux_work.c:105
#8  linux_queue_work_on (cpu=<optimized out>, wq=0xfffff80001e1e5c0, 
    work=0xfffff8019c5162f0)
    at /root/freebsd-src/sys/compat/linuxkpi/common/src/linux_work.c:150
#9  0xffffffff836f60ab in vt_drmfb_drawrect () from /boot/modules/drm.ko
#10 0xffffffff80a38044 in vt_set_border (vd=0xffffffff81aec518 <vt_consdev>, 
    area=0xfffff80002aeee00, c=<optimized out>)
    at /root/freebsd-src/sys/dev/vt/vt_core.c:1320
#11 vt_flush (vd=0xffffffff81aec518 <vt_consdev>)
    at /root/freebsd-src/sys/dev/vt/vt_core.c:1409
#12 0xffffffff80a3562e in vt_timer (arg=<unavailable>)
    at /root/freebsd-src/sys/dev/vt/vt_core.c:1435
#13 0xffffffff80c0c88b in softclock_call_cc (
    c=0xffffffff81aec628 <vt_consdev+272>, 
    cc=cc@entry=0xffffffff81ea3e80 <cc_cpu>, direct=direct@entry=0)
    at /root/freebsd-src/sys/kern/kern_timeout.c:721
#14 0xffffffff80c0e0b6 in softclock_thread (
    arg=arg@entry=0xffffffff81ea3e80 <cc_cpu>)
    at /root/freebsd-src/sys/kern/kern_timeout.c:860
#15 0xffffffff80ba45d0 in fork_exit (
    callout=0xffffffff80c0dff0 <softclock_thread>, 
    arg=0xffffffff81ea3e80 <cc_cpu>, frame=0xfffffe015c42cf40)
    at /root/freebsd-src/sys/kern/kern_fork.c:1102
#16 <signal handler called>
#17 0xff000000ff000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0xff000000ff000000
(kgdb) 

I assume you tested i915kms under a GENERIC-NODEBUG kernel.

@dumbbell
Copy link
Member Author

Indeed, because that's my daily driver. I will try on a GENERIC kernel like you. Thanks for the bug report!

@rhelmot
Copy link

rhelmot commented Mar 21, 2023

I can triage this - the locks being held are vt's vd->vd_curwindow->vb_lock spinlock from vt_flush and the work queue lock, a sleep lock (from drm_fb_helper_damage adding to the workqueue). I have absolutely no idea what the implications of these uses of locks are, but I'm gonna try switching the vt lock to a sleep lock and seeing what happens.

edit: as anyone could have predicted, that did not work!

@rhelmot
Copy link

rhelmot commented Mar 21, 2023

I finally twiddled with locks to the point that the driver loads, though X refuses to start. I suspect that's a separate issue.

These patches are assuredly not commit-ready but they should at least help you skip some steps in figuring out where all the paths that are locking erroneously are, since there's more than just the one which is documented above.

kernel:

diff --git a/sys/dev/vt/vt_core.c b/sys/dev/vt/vt_core.c
index 267dd7bf2489..80002b9c21f9 100644
--- a/sys/dev/vt/vt_core.c
+++ b/sys/dev/vt/vt_core.c
@@ -1406,17 +1406,23 @@ vt_flush(struct vt_device *vd)
                vd->vd_flags &= ~VDF_INVALID;
 
                a = teken_get_curattr(&vw->vw_terminal->tm_emulator);
+               /*
+                * The below calls may need to run a sleep lock in order to
+                * dispatch linuxkpi work
+                */
+               vtbuf_unlock(&vw->vw_buf);
                vt_set_border(vd, &vw->vw_draw_area, a->ta_bgcolor);
                vt_termrect(vd, vf, &tarea);
                if (vd->vd_driver->vd_invalidate_text)
                        vd->vd_driver->vd_invalidate_text(vd, &tarea);
                if (vt_draw_logo_cpus)
                        vtterm_draw_cpu_logos(vd);
+               vtbuf_lock(&vw->vw_buf);
        }
 
        if (tarea.tr_begin.tp_col < tarea.tr_end.tp_col) {
-               vd->vd_driver->vd_bitblt_text(vd, vw, &tarea);
                vtbuf_unlock(&vw->vw_buf);
+               vd->vd_driver->vd_bitblt_text(vd, vw, &tarea);
                return (1);
        }

drm-kmod:

diff --git a/drivers/gpu/drm/vt_drmfb.c b/drivers/gpu/drm/vt_drmfb.c
index 116121e14f..62ef106d0c 100644
--- a/drivers/gpu/drm/vt_drmfb.c
+++ b/drivers/gpu/drm/vt_drmfb.c
@@ -257,6 +257,7 @@ vt_drmfb_postswitch(struct vt_device *vd)
 {
        struct fb_info *fbio;
        struct linux_fb_info *info;
+       int locked = mtx_owned(&vd->vd_lock);
 
        fbio = vd->vd_softc;
        info = to_linux_fb_info(fbio);
@@ -266,8 +267,14 @@ vt_drmfb_postswitch(struct vt_device *vd)
        }
 
        if (!kdb_active && !KERNEL_PANICKED()) {
+               if (locked) {
+                       VT_UNLOCK(vd);
+               }
                linux_set_current(curthread);
                info->fbops->fb_set_par(info);
+               if (locked) {
+                       VT_LOCK(vd);
+               }
        } else {
 #ifdef DDB
                db_trace_self_depth(10);

@dumbbell
Copy link
Member Author

Thank you @rhelmot! That's really helpful. Would you mind sharing this again in the #243 pull request, where the vt(4) integration patch is being worked on?

@gizahNL
Copy link

gizahNL commented Apr 8, 2023

Intel DG1:

drmn1: <drmn> on vgapci1
vgapci1: child drmn1 requested pci_enable_io
vgapci1: child drmn1 requested pci_enable_io
vgapci1: 0x1000000 bytes of rid 0x10 res 3 failed (0, 0xffffffffffffffff).
drmn1: _lkpi_pci_iomap: failed to alloc bar 0 type 3 rid 16
drmn1: [drm] *ERROR* failed to map registers
drmn1: 0xfffffe0157b6c788Vdrmn1: Please file a bug on drm/i915; see https://gitlab.freedesktop.org/drm/intel/-/wikis/How-to-file-i915-bugs for details.device_attach: drmn1 attach returned 5

It does populate /dev/dri/renderD128, which is a significant progression :) Unfortunately vainfo doesn't like it and can't open the device (my usecase & primary test case is transcoding video).

I compiled against freebsd/main, and needed the following patch on this PR:

diff --git a/linuxkpi/gplv2/include/linux/device.h b/linuxkpi/gplv2/include/linux/device.h
index 6523393d2d..81f872b722 100644
--- a/linuxkpi/gplv2/include/linux/device.h
+++ b/linuxkpi/gplv2/include/linux/device.h
@@ -3,19 +3,4 @@
 
 #include_next <linux/device.h>
 
-/* allows to add/remove a custom action to devres stack */
-int devm_add_action(struct device *dev, void (*action)(void *), void *data);
-
-static inline int devm_add_action_or_reset(struct device *dev,
-					   void (*action)(void *), void *data)
-{
-	int ret;
-
-	ret = devm_add_action(dev, action, data);
-	if (ret)
-		action(data);
-
-	return ret;
-}
-
 #endif	/* _LINUX_DEVICE_H_ */
diff --git a/linuxkpi/gplv2/src/linux_devres.c b/linuxkpi/gplv2/src/linux_devres.c
index 1243242d08..0ae429605d 100644
--- a/linuxkpi/gplv2/src/linux_devres.c
+++ b/linuxkpi/gplv2/src/linux_devres.c
@@ -17,28 +17,3 @@ static void devm_action_release(struct device *dev, void *res)
 	devres->action(devres->data);
 }
 
-/**
- * devm_add_action() - add a custom action to list of managed resources
- * @dev: Device that owns the action
- * @action: Function that should be called
- * @data: Pointer to data passed to @action implementation
- *
- * This adds a custom action to the list of managed resources so that
- * it gets executed as part of standard resource unwinding.
- */
-int devm_add_action(struct device *dev, void (*action)(void *), void *data)
-{
-	struct action_devres *devres;
-
-	devres = devres_alloc(devm_action_release,
-			      sizeof(struct action_devres), GFP_KERNEL);
-	if (!devres)
-		return -ENOMEM;
-
-	devres->data = data;
-	devres->action = action;
-
-	devres_add(dev, devres);
-	return 0;
-}
-EXPORT_SYMBOL_GPL(devm_add_action);

@orbitz
Copy link

orbitz commented Jul 3, 2023

Hello! I've been busy for a few months and have not been able to try the latest code. Looks like @dumbbell has been busy too. Do you have a chance to say what the latest status is? Is it worth testing?

Evan Quan and others added 18 commits November 27, 2023 11:00
MMHUB PG needs to be disabled for Picasso for stability reasons.

Signed-off-by: Evan Quan <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Cc: [email protected]
Add a quirk in sienna_cichlid_ppt.c to fix some OEM SKU
specific stability issues.

Signed-off-by: Evan Quan <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Cc: [email protected]
Even if PSR is allowed for a present GPU, there might be no eDP link
which supports PSR.

Fixes: 708978487304 ("drm/amdgpu/display: Only set vblank_disable_immediate when PSR is not enabled")
Reviewed-by: Harry Wentland <[email protected]>
Signed-off-by: Michel Dänzer <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
When we switch to dma_resv_wait_timeout() the returned type changes as
well.

Signed-off-by: Christian König <[email protected]>
Fixes: 89aae41d740f ("drm/radeon: use dma_resv_wait_timeout() instead of manually waiting")
Bug: https://bugzilla.kernel.org/show_bug.cgi?id=215600
Reviewed-by: Alex Deucher <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]
In order to fill the drm_display_info structure each time an EDID is
read, the code currently will call drm_add_display_info with the parsed
EDID.

drm_add_display_info will then call drm_reset_display_info to reset all
the fields to 0, and then set them to the proper value depending on the
EDID.

In the color_formats case, we will thus report that we don't support any
color format, and then fill it back with RGB444 plus the additional
formats described in the EDID Feature Support byte.

However, since that byte only contains format-related bits since the 1.4
specification, this doesn't happen if the EDID is following an earlier
specification. In turn, it means that for one of these EDID, we end up
with color_formats set to 0.

The EDID 1.3 specification never really specifies what it means by RGB
exactly, but since both HDMI and DVI will use RGB444, it's fairly safe
to assume it's supposed to be RGB444.

Let's move the addition of RGB444 to color_formats earlier in
drm_add_display_info() so that it's always set for a digital display.

Fixes: da05a5a71ad8 ("drm: parse color format support for digital displays")
Cc: Ville Syrjälä <[email protected]>
Reported-by: Matthias Reichl <[email protected]>
Signed-off-by: Maxime Ripard <[email protected]>
Reviewed-by: Ville Syrjälä <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]
commit 0064b0ce85bb ("drm/amd/pm: enable ASPM by default") enabled ASPM
by default but a variety of hardware configurations it turns out that this
caused a regression.

* PPC64LE hardware does not support ASPM at a hardware level.
  CONFIG_PCIEASPM is often disabled on these architectures.
* Some dGPUs on ALD platforms don't work with ASPM enabled and PCIe subsystem
  disables it

Check with the PCIe subsystem to see that ASPM has been enabled
or not.

Fixes: 0064b0ce85bb ("drm/amd/pm: enable ASPM by default")
Link: https://wiki.raptorcs.com/w/images/a/ad/P9_PHB_version1.0_27July2018_pub.pdf
Link: https://gitlab.freedesktop.org/drm/amd/-/issues/1723
Link: https://gitlab.freedesktop.org/drm/amd/-/issues/1739
Link: https://gitlab.freedesktop.org/drm/amd/-/issues/1885
Link: https://gitlab.freedesktop.org/drm/amd/-/issues/1907
Tested-by: [email protected]
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Mario Limonciello <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Cc: [email protected]
[Why]
Found when running igt@kms_atomic.

Userspace attempts to do a TEST_COMMIT when 0 streams which calls
dc_remove_stream_from_ctx. This in turn calls link_enc_unassign
which ends up modifying stream->link = NULL directly, causing the
global link_enc to be removed preventing further link activity
and future link validation from passing.

[How]
We take care of link_enc unassignment at the start of
link_enc_cfg_link_encs_assign so this call is no longer necessary.

Fixes global state from being modified while unlocked.

Reviewed-by: Jimmy Kizito <[email protected]>
Acked-by: Jasdeep Dhillon <[email protected]>
Signed-off-by: Nicholas Kazlauskas <[email protected]>
Tested-by: Daniel Wheeler <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Cc: [email protected]
The GPU reset function of raven2 is not maintained or tested, so it should be
very unstable.

Now the amdgpu_asic_reset function is added to amdgpu_pmops_suspend, which
causes the S3 test of raven2 to fail, so the asic_reset of raven2 is ignored
here.

Fixes: daf8de0874ab5b ("drm/amdgpu: always reset the asic in suspend (v2)")
Signed-off-by: Chen Gong <[email protected]>
Acked-by: Alex Deucher <[email protected]>
Reviewed-by: Mario Limonciello <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Cc: [email protected]
vkms leverages common amdgpu framebuffer creation, and
also as it does not support FB modifier, there is no need
to check tiling flags when initing framebuffer when virtual
display is enabled.

This can fix below calltrace:

amdgpu 0000:00:08.0: GFX9+ requires FB check based on format modifier
WARNING: CPU: 0 PID: 1023 at drivers/gpu/drm/amd/amdgpu/amdgpu_display.c:1150 amdgpu_display_framebuffer_init+0x8e7/0xb40 [amdgpu]

v2: check adev->enable_virtual_display instead as vkms can be
	enabled in bare metal as well.

Signed-off-by: Leslie Shi <[email protected]>
Signed-off-by: Guchun Chen <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Workstation application ANSA/META v21.1.4 get this error dmesg when
running CI test suite provided by ANSA/META:
[drm:amdgpu_gem_va_ioctl [amdgpu]] *ERROR* Couldn't update BO_VA (-16)

This is caused by:
1. create a 256MB buffer in invisible VRAM
2. CPU map the buffer and access it causes vm_fault and try to move
   it to visible VRAM
3. force visible VRAM space and traverse all VRAM bos to check if
   evicting this bo is valuable
4. when checking a VM bo (in invisible VRAM), amdgpu_vm_evictable()
   will set amdgpu_vm->evicting, but latter due to not in visible
   VRAM, won't really evict it so not add it to amdgpu_vm->evicted
5. before next CS to clear the amdgpu_vm->evicting, user VM ops
   ioctl will pass amdgpu_vm_ready() (check amdgpu_vm->evicted)
   but fail in amdgpu_vm_bo_update_mapping() (check
   amdgpu_vm->evicting) and get this error log

This error won't affect functionality as next CS will finish the
waiting VM ops. But we'd better clear the error log by checking
the amdgpu_vm->evicting flag in amdgpu_vm_ready() to stop calling
amdgpu_vm_bo_update_mapping() later.

Another reason is amdgpu_vm->evicted list holds all BOs (both
user buffer and page table), but only page table BOs' eviction
prevent VM ops. amdgpu_vm->evicting flag is set only for page
table BOs, so we should use evicting flag instead of evicted list
in amdgpu_vm_ready().

The side effect of this change is: previously blocked VM op (user
buffer in "evicted" list but no page table in it) gets done
immediately.

v2: update commit comments.

Acked-by: Paul Menzel <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Qiang Yu <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Cc: [email protected]
SLPC unset param H2G only needs one parameter - the id of the
param.

Fixes: 025cb07bebfa ("drm/i915/guc/slpc: Cache platform frequency limits")

Suggested-by: Umesh Nerlige Ramappa <[email protected]>
Signed-off-by: Vinay Belgaumkar <[email protected]>
Reviewed-by: Umesh Nerlige Ramappa <[email protected]>
Signed-off-by: Ramalingam C <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]
(cherry picked from commit 9648f1c3739505557d94ff749a4f32192ea81fe3)
Signed-off-by: Tvrtko Ursulin <[email protected]>
This JSP2 PCH actually seems to be some special Apple
specific ICP variant rather than a JSP. Make it so. Or at
least all the references to it seem to be some Apple ICL
machines. Didn't manage to find these PCI IDs in any
public chipset docs unfortunately.

The only thing we're losing here with this JSP->ICP change
is Wa_14011294188, but based on the HSD that isn't actually
needed on any ICP based design (including JSP), only TGP
based stuff (including MCC) really need it. The documented
w/a just never made that distinction because Windows didn't
want to differentiate between JSP and MCC (not sure how
they handle hpd/ddc/etc. then though...).

Cc: [email protected]
Cc: Matt Roper <[email protected]>
Cc: Vivek Kasireddy <[email protected]>
Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/4226
Fixes: 943682e3bd19 ("drm/i915: Introduce Jasper Lake PCH")
Signed-off-by: Ville Syrjälä <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]
Acked-by: Vivek Kasireddy <[email protected]>
Tested-by: Tomas Bzatek <[email protected]>
(cherry picked from commit 53581504a8e216d435f114a4f2596ad0dfd902fc)
Signed-off-by: Tvrtko Ursulin <[email protected]>
VRR capable property is not attached by default to the connector
It is attached only if VRR is supported.
So if the driver tries to call drm core set prop function without
it being attached that causes NULL dereference.

Cc: Jani Nikula <[email protected]>
Cc: Ville Syrjälä <[email protected]>
Cc: [email protected]
Signed-off-by: Manasi Navare <[email protected]>
Reviewed-by: Ville Syrjälä <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]
Regression has been reported that suspend/resume may hang with
the previous vm ready check commit.

So bring back the evicted list check as a temp fix.

Bug: https://gitlab.freedesktop.org/drm/amd/-/issues/1922
Fixes: c1a66c3bc425 ("drm/amdgpu: check vm ready by amdgpu_vm->evicting flag")
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Qiang Yu <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Currently we are observing occasional screen flickering when
PSR2 selective fetch is enabled. More specifically glitch seems
to happen on full frame update when cursor moves to coords
x = -1 or y = -1.

According to Bspec SF Single full frame should not be set if
SF Partial Frame Enable is not set. This happened to be true for
ADLP as PSR2_MAN_TRK_CTL_ENABLE is always set and for ADL_P it's
actually "SF Partial Frame Enable" (Bit 31).

Setting "SF Partial Frame Enable" bit also on full update seems to
fix screen flickering.

Also make code more clear by setting PSR2_MAN_TRK_CTL_ENABLE
only if not on ADL_P. Bit 31 has different meaning in ADL_P.

Bspec: 49274

v2: Fix Mihai Harpau email address
v3: Modify commit message and remove unnecessary comment

Tested-by: Lyude Paul <[email protected]>
Fixes: 7f6002e58025 ("drm/i915/display: Enable PSR2 selective fetch by default")
Reported-by: Lyude Paul <[email protected]>
Cc: Mihai Harpau <[email protected]>
Cc: José Roberto de Souza <[email protected]>
Cc: Ville Syrjälä <[email protected]>
Bugzilla: https://gitlab.freedesktop.org/drm/intel/-/issues/5077
Signed-off-by: Jouni Högander <[email protected]>
Reviewed-by: José Roberto de Souza <[email protected]>
Signed-off-by: José Roberto de Souza <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]
(cherry picked from commit 8d5516d18b323cf7274d1cf5fe76f4a691f879c6)
Signed-off-by: Tvrtko Ursulin <[email protected]>
@dumbbell dumbbell marked this pull request as ready for review November 27, 2023 10:04
@dumbbell
Copy link
Member Author

I just marked the branch as ready to review now that #243 is merged!

@thewanderingtraderm:

I have an Intel alder lake i7-12700H with Intel Alder Lake-P - Integrated Graphics Controller and when I load the i915kms graphics driver the whole system freezes.

Does it work with the drm-515-kmod package (5.15-lts branch) otherwise?

does this work with freebsd 14-RELEASE?

It won't because the patches to vt(4) (related to #243) are not backported yet.

@wulf7 wulf7 merged commit 4daf995 into freebsd:master Nov 27, 2023
1 check failed
@dumbbell dumbbell deleted the update-to-v5.17 branch November 27, 2023 13:58
@thewanderingtraderm
Copy link

Does this work with FreeBSD 15-CURRENT?

@dumbbell
Copy link
Member Author

@thewanderingtraderm: Yes, it works with FreeBSD 15-CURRENT

@thewanderingtraderm
Copy link

how do I install it. I tried compiling and installing it but it didn't load because of version mismatch. I'm on FreeBSD 15-CURRENT

@orbitz
Copy link

orbitz commented Dec 11, 2023

@dumbbell Great seeing this merged! Have the installation instructions changed? Is it enough to compile and install the FreeBSD main branch, then the drm-kmod/drm_v5.17 branch?

@dumbbell
Copy link
Member Author

@orbitz: Yes drm-kmod master branch can be compiled against and used with FreeBSD -CURRENT/main branch.

@ko56
Copy link

ko56 commented Dec 12, 2023

Do you think this will make it into 14.0-RELEASE at some point?

@dumbbell
Copy link
Member Author

dumbbell commented Dec 13, 2023

@ko56: 14.0-RELEASE, no, the patches to freebsd-src were not ready in time unfortunately.

I think they can go into the future 14.1-RELEASE however, perhaps 13.x-RELEASE as well, I don't know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.