[BUG] sof firmware appears to have caused Ubuntu laptop to fail wakeup

[rogpeppe]

Describe the bug
I just slept my X1 Carbon 7th Gen running stock Ubuntu 20.04 and it failed to wake up.

uname -a output:

Linux snazzy2 5.8.0-38-generic thesofproject/sof#43~20.04.1-Ubuntu SMP Tue Jan 12 16:39:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I'm afraid I have no idea how to find out

Looking in the journalctl log, I saw the following log output, which includes some error messages
that mention "sof", so perhaps that's the underlying reason.

The laptop has been doing this relatively regularly (maybe once every week or so) but
this is the first time I've thought to examine the log, so I don't know if the other problems
were related.

When the machine was "dead", I saw the following output on the screen, which might be
relevant too, I guess:

Here's the log starting just when the machine suspends until it looks like it's definitely
failing to wake up.

Jan 16 11:39:14 snazzy2 systemd[1]: Reached target Sleep.
Jan 16 11:39:14 snazzy2 systemd[1]: Starting Suspend...
Jan 16 11:39:14 snazzy2 systemd-sleep[696419]: Suspending system...
Jan 16 11:39:14 snazzy2 kernel: PM: suspend entry (s2idle)
Jan 16 11:47:57 snazzy2 kernel: Filesystems sync: 0.005 seconds
Jan 16 11:47:57 snazzy2 kernel: Freezing user space processes ... (elapsed 0.002 seconds) done.
Jan 16 11:47:57 snazzy2 kernel: OOM killer disabled.
Jan 16 11:47:57 snazzy2 kernel: Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Jan 16 11:47:57 snazzy2 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Jan 16 11:47:57 snazzy2 kernel: e1000e: EEE TX LPI TIMER: 00000011
Jan 16 11:47:57 snazzy2 kernel: ACPI: EC: interrupt blocked
Jan 16 11:47:57 snazzy2 kernel: ACPI: EC: interrupt unblocked
Jan 16 11:47:57 snazzy2 kernel: sof-audio-pci 0000:00:1f.3: error: no reply expected, received 0x0
Jan 16 11:47:57 snazzy2 kernel: sof-audio-pci 0000:00:1f.3: error: DMA trigger stop failed
Jan 16 11:47:57 snazzy2 kernel: sof-audio-pci 0000:00:1f.3: error: load fw failed ret: -110
Jan 16 11:47:57 snazzy2 kernel: sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000
Jan 16 11:47:57 snazzy2 kernel: sof-audio-pci 0000:00:1f.3: error: failed to reset DSP
Jan 16 11:47:57 snazzy2 kernel: sof-audio-pci 0000:00:1f.3: error: failed to boot DSP firmware after resume -110
Jan 16 11:47:57 snazzy2 kernel: PM: dpm_run_callback(): pci_pm_resume+0x0/0xa0 returns -110
Jan 16 11:47:57 snazzy2 kernel: PM: Device 0000:00:1f.3 failed to resume: error -110
Jan 16 11:47:57 snazzy2 kernel: snd_hda_codec_realtek ehdaudio0D0: Unable to sync register 0x2b8000. -5
Jan 16 11:47:57 snazzy2 kernel: snd_hda_codec_realtek ehdaudio0D0: Unable to sync register 0x2b8000. -5
Jan 16 11:47:57 snazzy2 kernel: OOM killer enabled.
Jan 16 11:47:57 snazzy2 kernel: Restarting tasks ... 
Jan 16 11:47:57 snazzy2 kernel: Core dump to |/usr/share/apport/apport pipe failed
Jan 16 11:47:57 snazzy2 kernel: done.
Jan 16 11:47:57 snazzy2 rtkit-daemon[1485]: The canary thread is apparently starving. Taking action.
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE) event10 - SYNA8005:00 06CB:CD8C Touchpad: kernel bug: Touch jump detected and discarded.
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: See https://wayland.freedesktop.org/libinput/doc/1.15.5/touchpad-jumping-cursors.html for details
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (II) event3  - AT Translated Set 2 keyboard: SYN_DROPPED event - some input events have been lost.
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE)
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: Fatal server error:
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE) systemd-logind disappeared (stopped/restarted?)
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE)
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE)
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: Please consult the The X.Org Foundation support
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]:          at http://wiki.x.org
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]:  for help.
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE) Please also check the log file at "/home/rogpeppe/.local/share/xorg/Xorg.0.log" for additional information.
Jan 16 11:47:57 snazzy2 /usr/lib/gdm3/gdm-x-session[133934]: (EE)
Jan 16 11:47:57 snazzy2 kernel: thermal thermal_zone5: failed to read out thermal zone (-61)
Jan 16 11:47:57 snazzy2 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
Jan 16 11:47:57 snazzy2 rtkit-daemon[1485]: Demoting known real-time threads.
Jan 16 11:47:57 snazzy2 tailscaled[1057]: magicsock: starting endpoint update (periodic)
Jan 16 11:47:57 snazzy2 tailscaled[1057]: logtail: upload: log upload of 161 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/98d087e157212145e6bc70d52accf449e582f16a63a695d3600522d9eb0b254e": context deadline exceeded
Jan 16 11:47:57 snazzy2 tailscaled[1057]: logtail: backoff: 13 msec
Jan 16 11:47:57 snazzy2 tailscaled[1057]: magicsock: [0xc0001e80c0] derp.Recv(derp-8): derp.Recv: read tcp4 192.168.1.225:53706->167.71.139.179:443: i/o timeout
Jan 16 11:47:57 snazzy2 tailscaled[1057]: magicsock: ReSTUN: endpoint update active, need another later ("derp-recv-error")
Jan 16 11:47:57 snazzy2 tailscaled[1057]: derp-8: backoff: 8 msec
Jan 16 11:47:57 snazzy2 systemd[1]: systemd-logind.service: Killing process 847 (systemd-logind) with signal SIGABRT.
Jan 16 11:47:57 snazzy2 rtkit-daemon[1485]: Demoted 0 threads.

I've afraid I have no idea how to find out which sof firmware I'm using, or the topology file, or what "platform" might mean in this context. Please let me know if there's any more info I can provide and how. The sof-logger command doesn't seem to exist on my system.

[juimonen]

@rogpeppe I have the X1 myself, and installed stock 20.04, but upgraded kernel and firmware many times, because doing development... could you try to upgrade your kernel etc? I just tried the suspend and didn't see problems, but yes I'm on different kernel than you.

[rogpeppe]

@juimonen Which kernel version would you recommend that I install? I was looking at the installer here and concerned that it might install a kernel that's not stable yet.

[juimonen]

@rogpeppe maybe just try "apt-get upgrade" so it should be then some kind of official stable ubuntu kernel if it gets updated... you should probably get also other parts like firmware and topology files with that command. Or have you done that already?

@kv2019i any ideas about the kernel version?

[kv2019i]

@rogpeppe Thanks for the report. I'd also recommend trying a system upgrade first. The newer SOF1.6 firwmare was updated to 20.04 in linux-firmware 1.187.6, you should get that via apt-get upgrade.

In case that still fails, next steps:

• can you share/upload output of alsa-info.sh
• please try a newer kernel version -- you can do this as a one-off test to see whether latest upstream kernel affects the bug, and then return to stock Ubuntu kernel

[rogpeppe]

I just tried an apt-get upgrade and rebooting, but it doesn't seem to have updated the kernel version at all (uname -a reports the same as before).

This issue has always been very sporadic (and it may well not have been the audio driver at fault in other cases!). The last time it failed to wake up, I didn't have any messages in the log at all.

alsa-info.sh is this, presumably? https://wiki.ubuntu.com/Audio/AlsaInfo

I'm not familiar with selecting different kernel versions. I'll have a google around.

[kv2019i]

@rogpeppe wrote:

I just tried an apt-get upgrade and rebooting, but it doesn't seem to have updated the kernel version at all (uname -a reports the same as before).

Ack.

This issue has always been very sporadic (and it may well not have been the audio driver at fault in other cases!). The last time it failed to wake up, I didn't have any messages in the log at all.

Ack, this is good info. There is a communication error in the screenshot you shared, so this is worth looking at in any case. I found one older bug where same error status code is seen, but this was fixed long time back and should be fixed in 20.04 kernel+fw for sure:
https://github.com/thesofproject/sof/issues/3752

alsa-info.sh is this, presumably? https://wiki.ubuntu.com/Audio/AlsaInfo

Yes, exactly that. On Ubuntu, it's actually installed by default and you can just run "sudo alsa-info".

I'm not familiar with selecting different kernel versions. I'll have a google around.

If you are unfamiliar with running customer kernels, maybe better not to try this yet. Instructions are at https://wiki.ubuntu.com/Kernel/MainlineBuilds but let's try to gather data from the stock Ubuntu kernel first.

[kv2019i]

@rogpeppe any update?

[rogpeppe]

It hasn't happened again so far. When it was happening quite a bit, I was on battery power, but since then I've been mostly on external power, in case that makes a difference.

[keyonjie]

set to P3 as no reproduction here.

[plbossart]

let's close for now. Hopefully the combination of SOF 1.7 and ubuntu updates solved this.