[BUG][BYT][CHT] Linux 5.11 Cherrytrail/rt5640 emits persistent beeping sound after a short while #3868
Comments
|
Thanks for reporting this @fxthomas Nothing really comes to my mind as a possible issue and our good friend @jwrdegoede has played quite a bit with SOF on CherryTrail. Do you have the ability to test with the latest SOF development kernel (branch topic/sof-dev)? @bardliao anything on the RT5640 that could help explain a sustained 'beeep'? |
|
I guess it is something related to clock. @fxthomas Could you check is the "Platform Clock" widget always on the right state? I.e. It should be On in playback and Off after playback. You can cat the "Platform Clock" node in /sys/kernel/debug/asoc//dapm/. @oder-chiou @shumingfan FYI. |
|
Thanks for the quick answers!
The one that's in https://github.com/thesofproject/linux/tree/topic/sof-dev? Never done a custom kernel build before, but the docs look clear enough. I'm compiling it right now.
It's in |
|
@fxthomas Would you please dump the codec registers?
dump codec registers: BTW, is it possible to measure the MCLK frequency on your side? |
|
I've been testing SOF + BYT/CHT with pipewire instead of pulseaudio as audioserver this weekend and I did notice some instabilities where sound stopped playing (no beep though) until I stopped all audio-playback for a while and then tried again. There were also some issues with pipewire not properly executing the UCM enable-seq for capture, so the capture pipeline was not working while the gnome-control-center sound-pane did try to use it (for the input vu-meter) so that might have been part of what I was seeing. I switched back to SST to try and debug pipewire issues without also having SOF in play, during this I learned: https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/866 that pipewire defaults to 24 bits (or maybe even 32 when advertised?) where pulseaudo will always pick s16_le as format if available. @fxthomas did you specify e.g. "-Dhw:1" when testing with aplay ? Otherwise you are still going through the audio-server (through the alsa-plugin for the audio-server) I believe and then you may have been testing with pipewire. Also did you specify a format with the -f param to aplay ? I wonder if the SOF CI is testing with 24 / 32 bits sample sizes on BYT/CHT and I wonder if the bigger sample size may be causing issues with e.g. the stack on the DSP, or just some things becoming slower enough to sometimes be a problem ? |
Yes this will have impact on FW buffers and processing headroom. 24bit is on the CI test list for CHT, but it's likely not stressed like 16bit is (sinec it's the primary use case on the older HW). The CHT multi format test is here. |
Ok, so that is what I expected, in that case it seems to me that it would be better if pipewire were to just use 16 bit formats on these device ? Also note that the SOF driver is even advertising 32 bit formats (S32_LE) I would not be surprised if pipewire actually picks that one. If you agree that pipewire just using 16bit formats would be better, then the question becomes how we achieve this. I think that ideally the SOF driver should not advertise 24bit/32bit formats (on BYT/CHT) to userspace in that case. Or maybe have a module option to advertise these or not ? |
|
@jwrdegoede The SOF driver just reports what the topology defines, there is no hard-coded format. I would be surprised if there was any difference, the dai is already configured to support 24 bits. |
Yes, I used
In the linked pipewire issue comment I noted the For a quick summary, the only two that worked in the SST driver were: In the SOF driver, the beeping occurs after some time whether I use Also I said (famous last words!): ...but while it boots just fine I'm having some trouble figuring out how to enable SOF in this build, because it is apparently not as simple as the default 5.11 kernel I'm running. I need to read the docs, I'm just a poor newcomer to that world 😉 |
|
@fxthomas there is a kernel parameter to select SOF, by default SST is the default. copy this to any file in /etc/modprobe.d/ options snd-intel-dspcfg dsp_driver=3 |
|
alternatively there's also a build config but I am not sure if it's in our development version just yet. |
That's the one I used, it worked on the official 5.11 kernel but not on the SOF branch when I tried. I'm rebuilding the branch right now from scratch, let's see if I can make that work better.
Looks like it's there in thesofproject/linux@3620c5394b8b. If I can't make it work with the module option I'll try this one next. |
|
I finally got some success! Some observations first:
Tests for v5.11.4-arch1 (SOF, pure ALSA):
Tests for v5.11.4-arch1 (SST, pure ALSA):
Tests for
Tests for
Tests for
|
that's still with the SST driver, yes?
Good.
yes, that's normal. there are two drivers registering for the same ACPI ID, so both will be loaded but only one selected with the run-time detection. So some modules are loaded but their probe will fail. They will still show as loaded modules.
Not following. is this with the SOF or SST driver? |
|
On Thu, Mar 11, 2021, 01:59 Pierre-Louis Bossart ***@***.***> wrote:
yes, that's normal. there are two drivers registering for the same ACPI
ID, so both will be loaded but only one selected with the run-time
detection. So some modules are loaded but their probe will fail. They will
still show as loaded modules.
Thanks, that's good to know!
that's still with the SST driver, yes?
Not following. is this with the SOF or SST driver?
Ah that wasn't clear, sorry. The beeping issue is with SOF.
With SST both kernels play sound fine in S16LE only. Other formats being
advertised in a broken state is a separate issue that's being worked around
in the kernel if I understand correctly.
With SOF all formats seem to play correctly, but `topic/sof-dev` has no
issue, while the official 5.11.0~4 Arch kernels beep after a couple of
minutes as described.
During my tests everything on the system besides the kernel is the same,
and I tried to reuse as much as possible the same build/package
configuration for the `topic/sof-dev` build as the official kernel ; the
SOF-provided kernel config is merged on top of the one from the official
kernel, and that's pretty much it.
—
… You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3868 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAEVZWX2Z4AKSUTXPX5QHZDTDABV7ANCNFSM4YC77CXA>
.
|
|
thanks @fxthomas In this kind of cases, I would try to bisect to try to find more leads. a) start with commit 3620c5394b8b234ca77e562758127a6bb9e64720 ('ASoC: SOF: remove superfluous NULL check in debugfs read'). That's the last commit based on v5.11-rcX in our development tree It'll typically take 15 iterations but to reduce compilation time you can try generating a .config with our script, that compiles in 4mn on my host device since we've removed tons of things that aren't required for audio. see https://github.com/thesofproject/kconfig/blob/master/kconfig-sof-default.sh Sorry I don't have a more direct way of helping. |
|
Got it, thanks for the pointers to the relevant commits and all the help anyway, I already have a much better view of how I troubleshoot this — and learned tons about the Linux audio stack along the way! I'll let you know when I get results from the bisect. |
|
btw I was trying on a Cherrytrail+rt5640 (headset only, no speakers) and I didn't hear anything bad with topic/sof-dev either. |
|
Hmm, I'm actually back to square one. I was watching a video on the same kernel I compiled in my earlier tests from At this point it very well could be an hardware issue, but there's not much of interest in the logs/dmesg... |
|
For more information, I took a recording by plugging the rt5640 output jack to an external recorder's input jack. Here is the output right before the beeping happens:
Zoom on the region:
Interestingly the L/R channels seem to swap after a while (I noticed this earlier, but on the spectrogram above the discontinuities are very visible), see a zoom at one of them:
(Recording available here. The sound is not very clean — you can hear a bit of noise that can't be heard directly with headphones on the machine, but the beeping is exactly as I heard multiple times on this bug) |
|
That part about the channel-swap is interesting, as discussed in #3699 I was seeing this during my testing too, but this got fixed recently. I just checked and the fix for that is not in the 1.6.1 firmware which you are using. In my testing the swapping only happened between open/closing of the device but I never tested for long periods at a time, so maybe the swapping you are seeing is the same; and with some luck it is also the root-cause of the beep (so the swap triggering while playing causes the beep). I asked one of the SOF devs to provide my with a test-build of the upcoming 1.7 firmware which does include the fix for this, so that I could confirm that the channel-swap issue is fixed. It would be good if you can (re)test with that firmware bulld. I've attached that here. To install this, simply replace /lib/firmware/intel/sof/sof-cht.ri with the one from the attached zip and then reboot. |
Done, the beeping is still there but I didn't hear any channel swapping. Given the seemingly random nature of all this I'm not quite ready to call it, but it looks like a small improvement at least! Looking at the curves in my earlier message I get the impression that the beep is just caused by a 2ms buffer (96 samples@48kHz in the recording) not getting updated somewhere in the driver stack and repeating ad infinitum. Does that particular number ring a bell?
Can runtime PM in one of the platform drivers (codec/LPE/something else I'm not aware of) cause this somehow? Is there a way I can see the PM state e.g. in |
|
@fxthomas thanks, I've listened to the ogg file and it sounds like the DMA repeats the last period continually. Can you confirm the following for me.
I suspect here that the SSP port has an error condition and is not able to get new data from DMA. diff --git a/src/drivers/intel/baytrail/ssp.c b/src/drivers/intel/baytrail/ssp.c
index de5dc1346..3711309e7 100644
--- a/src/drivers/intel/baytrail/ssp.c
+++ b/src/drivers/intel/baytrail/ssp.c
@@ -534,6 +534,8 @@ static void ssp_stop(struct dai *dai, int direction)
{
struct ssp_pdata *ssp = dai_get_drvdata(dai);
+ dai_err(dai,"SSSR is 0x%x", ssp_read(dai, SSSR));
+
spin_lock(&dai->lock);
/* stop Rx if neeed */This will dump SSP status at stream stop. It will hopefully show an error condition. |
|
@fxthomas we don't support pm_runtime for SOF on Cherrytrail, so this is likely a firmware issue indeed. |
Thanks for confirming!
I didn't see any XRUNs. In the latest kernels (5.11.4+) The recording I shared was under Pipewire, where the beeping still stays persistent until I restart the PW service (presumably because it retries to establish the link continually).
How do I distinguish between the two? I see a lot of entries for controling DAC/ADC/mixer components like
Unfortunately I don't have a |
ok, so this is useful (I can rule out a few of the questions I asked above) - we have a pipeline like The DMA ALSA position is updated by the host DMA position, but the host DMA position may block if there is not free space in the pipeline (the pipeline is emptied by the DMA I2S - which sounds like it's repeating the last period consistently and not emptying the DATA). The pipeline buffers above have 2 periods so the timing headroom might be tight if PW is writting close to the host DMA position. I assume you have exactly the same error if you do Fwiw, I recommend using audacity to generate a sine wave stereo wav file with 48kHz sample rate. This shows up really well when captured for debugging and shows exactly what's happening (as you know what comes next). Btw, can you run sof-logger when the issue happens. See https://thesofproject.github.io/latest/developer_guides/debugability/logger/index.html The ldc files for logger are in here https://github.com/thesofproject/sof-bin You will also need to build the logger, See |
Yes, using
I will do that next time! 😉
I built 536800a (v1.6.1) that corresponds to the I then tried the following, but that does not seem to do anything on the official 5.11.6 kernel: On the Do I need special debug kernel options to get Alternatively, do I need to rebuild it to work on a different kernel? Sorry for the lot of questions, I hope that does not bother you too much! |
|
FWIW, installing v1.9 from https://github.com/thesofproject/sof-bin/archive/refs/heads/main.zip solved my issue. v2.0 doesn't seem to fix it. |
|
Is it possible that the SOF firmware playing audio in stereo could be part of the problem? I am curious as I noticed the SST firmware plays back in mono when I changed to it. |
|
@sturn42 That would be a bug of the SST firmware/driver or a configuration issue. On some platforms there's also a mono speaker only, so there is a downmix in the codec IIRC. |
|
Hi, Here were my attempts:
|
|
@wwangsa I would bet the differences in results is largely due to the kernel configurations, some kernels enable the DSP firmware trace and some don't, but as can be read above the only current/known work-around is to enable the trace... |
|
I haven't seen the issue coming back lately on Arch Linux, kernel 5.18.2 and SOF 2.1.1-1. I was able to play games and watch YouTube videos without running into this issue. |
|
Still present without snd_sof.sof_debug=1 |
mengdonglin
added
the
won't fix
|
Move BYT and CHT platform to maintenance mode and SOF v2.2 will be the last upstream release with support for this platform. There will be no more FW bug fixing effort on this platform. |
To clarify, does this mean releases following this will have zero support for these platforms whatsoever and users of these will have to figure out some way of staying on v2.2? |
|
the SOF releases will keep providing firmware for BYT, but the version will remain pegged to 2.2 for that SOC. 2.2 is really a long-term support branch, the main development branch is transitioning and will be based on the Zephyr RTOS for newer platforms. |
|
Hello this issue is still present now in Dec. with all updates on Ubuntu / Jammy confirmed on setzer/hp chromebook Lubuntu 5.15.0-57-lowlatency, but the beeping will stop after some time and it will recover and continue playing audio after about 30 seconds. |
|
snd_sof.sof_debug=1 remains a requirement for a work-around. There is no planned fix at this time. |
|
Thank you, will retest that. I can confirm that is also present in 6.0.0-1009-oem #9-Ubuntu and not limited to |
|
Confirming still present as of today on fully upgraded vanilla Ubuntu 20 LTS, even with Anything I forgot to do? Now trying what I understand is the last resort: Update. No surprises, that fixed it. A bit disappointing that a supposedly solid release of the supposedly mainstream distro would have this issue. But I guess it must be hard to get every device to cooperate simultaneously, so no complaints. Thanks for your work. |
|
You'll want to add the debug workaround to /etc/modprobe.d/alsa-base.conf
…On Wed, Dec 21, 2022, 6:02 PM Friptick ***@***.***> wrote:
Confirming still present as of today on fully upgraded vanilla Ubuntu 20
LTS, even with debug workaround. A regression? (I had killed it with the
debug hack on a previous Ubuntu install but had to reinstall Ubuntu and
now it's back.)
$ cat /etc/modprobe.d/sof.conf
options snd_sof sof_debug=1
$ cat /sys/module/snd_sof/parameters/sof_debug
1
$ cat /sys/module/snd_intel_dspcfg/parameters/dsp_driver
0
Anything I forgot to do?
Now trying what I understand is the last resort: options snd-intel-dspcfg
dsp_driver=2. Will edit with update.
—
Reply to this email directly, view it on GitHub
<#3868 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APG6YTWRRQLBEVJFGIXLUYTWOOD75ANCNFSM4YC77CXA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I can also confirm that there is a regression somewhere. Latest stable kernel and sof with sof_debug=1 produces sound freezes. |
|
There's been no change on the firmware and SOF sides, so likely a problem somewhere else. that's what git bisect is for I am afraid. |
|
Do not know where to find clues. Tried bisecting and found that even 5.18 has this freeze. Also tried SOF 2.1.1 and also the same. |
|
After I stepped upon this bug, put intel_idle.max_cstate=2 and disabled intel_iommu. Still testing but even without sof.debug=1 no sound freezes yet. |
|
@Friptick Thanks a lot, adding the line
|
|
I would like to add my expirience with this on intel-z3735 and xubuntu-2204. In alsa-base.conf i activated sof and debug but there was still buzz-sound. In qmplay2 I redirected sound to alsa instead of pulse, this way buzz was much shorter. Then in qmplay2 for video I activated ffmpeg with opengl instead of ffmpeg-vaapi with vulcan, and it seems that buzz-sound doesn't happen anymore. |
|
Just to add my own experience, I had this buzzing problem on an Acer Aspire Switch 10 tablet (Intel Atom Z3745), even with fully updated Ubuntu 22.04.1. |
|
I'd share my experience too. I had this problem on an Asus VivoBook E200HA (Intel Atom x5-Z8300) with fully updated Lubuntu 22.04 (kernel 5.19). |
|
Adding |
Right I believe that the SOF driver is never going to get fixed on this older devices. Fedora has stopped setting CONFIG_SND_INTEL_BYT_PREFER_SOF for a long long time already. I think we should just remove CONFIG_SND_INTEL_BYT_PREFER_SOF from the upstream kernel all together and have the kernel always default to SST (dsp_driver=2) on BYT + CHT hardware. @plbossart what do you think, shall I submit a patch upstream dropping CONFIG_SND_INTEL_BYT_PREFER_SOF ? |
Yes, I agree @jwrdegoede, we've unfortunately dropped the ball on SOF+Baytrail. |
and others
I had this issue while trying to debug https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/530 on an HP Detachable x2 ultrabook, where we're trying to see if the SOF driver is somehow more stable than SST.
This issue happens with raw ALSA (neither Pipewire/Pulseaudio running) as well as with both PW/PA.
The following sequence reproduces the issue 100% of the time:
options snd-intel-dspcfg dsp_driver=3on a 5.11 kerneldmesg | grep sofaplay -D <device name> file.wavfor example)As far as I can tell the beep only goes away when starting (if running raw ALSA) or restarting (if already running) the Pipewire/Pulseaudio server, which restores proper playback until the next beep.
The default SST driver is stable for basic audio output in S16LE, despite issues in other areas.
Environment
Logs:
dmesgat allsof-logger, is there a way for me to compile it without bringing up the whole toolchain?I'm not familiar at all with SOF, so if this is the wrong place for a report do let me know!
The text was updated successfully, but these errors were encountered: