Driver support for firmware recovery

[plbossart]

When the firmware gets stuck or unresponsive (IPC timeouts), the driver just bails out and the only way to recover is to reboot. That's not acceptable for a number of products.

The SOF driver needs to implement a recovery mechanism where it detects the firmware has gone off tracks (e.g. with a 5s watchdog), and restores the configuration (hardware reset, firmware download, pipelines and controls restored). We already have this sequence for resume so it shouldn't be too hard.

Note that this request is not asking for seamless operation. Applications will likely time out and likely stop, so collaboration with userspace is likely required to reenable playback/capture. The ask is really to avoid a reboot and restart cleanly.

[wenqingfu]

extended work of thesofproject/sof#710?

[plbossart]

On 12/17/18 5:29 PM, Michael Fu wrote: extended work of thesofproject/sof#710 <thesofproject/sof#710>?

No, different problem. recovery on firmware stuck means reset and redownload. recovery on underflow means closing and reopening the PCM device.

[ranj063]

@plbossart looking into the hda_dsp_core_reset_power_down() method in hds-dsp.c, we seem to be placing the core in reset before powering down and when the firmware is unresponsive, this also seems to fail. This is required when we want to shutdown gracefully.

In the case of panic, would it be acceptable to just power down the cores instead of placing them in reset?

[plbossart]

On 12/21/18 1:14 PM, Ranjani Sridharan wrote: @plbossart <https://github.com/plbossart> looking into the hda_dsp_core_reset_power_down() method in hds-dsp.c. We seem to be placing the core in reset before powering down and when the firmware is unresponsive, this also seems to fail. This is required when we want to shutdown gracefully. In the case of panic, would it be acceptable to just power down the cores instead of placing them in reset?

Sorry, no idea. The current behavior is I believe tracking what was implemented for the Skylake driver, not sure though if it was based on experiments or guidance from hardware.

…

[xiulipan]

@plbossart @ranj063
Actually now when a panic happened. If runtime_pm is not disabled, it will automatically recovery the DSP by runtime_suspend and runtime_resume.

This now brings some difficulty in debug as all the logs and dumped exception info is gone with that suspend/resume.

As for the recovery mode, I think we may need to add flag or debugfs controller to disable the FW recovery for DEBUG usage.

[plbossart]

On 2/10/19 9:01 AM, xiulipan wrote: @plbossart <https://github.com/plbossart> @ranj063 <https://github.com/ranj063> Actually now when a panic happened. If runtime_pm is not disabled, it will automatically recovery the DSP by runtime_suspend and runtime_resume. This now brings some difficulty in debug as all the logs and dumped exception info is gone with that suspend/resume. As for the recovery mode, I think we may need to add flag or debugfs controller to disable the FW recovery for DEBUG usage.

Panic is one aspect, but the more critical part is timeouts when the DSP gets stuck or irresponsive. You are correct that when recovery happens we should keep a trace of what happened before the recovery, otherwise it's just awfully hard to test.

[wenqingfu]

@kv2019i , would you be able to take this one? thanks!

[kv2019i]

@wenqingfu wrote:

@kv2019i , would you be able to take this one? thanks!

Ack. I'm currently looking at tracing issues during suspend and runtime supsned (old issue #248
and newer #486), will work on these first (good way to learn the tracing part first) and then move to this.

[kv2019i]

Hello, I now pushed initial patchset to address this issue (PR841). I ended up taking a slightly different path than what you've discussed earlier, so please check the code and see whether this is on the right track or not.