Improve IPC flood error reports

[plbossart]

in CI report: https://sof-ci.01.org/linuxpr/PR2201/build3950/devicetest/
one can see:

kernel: [ 96.248808] sof-audio-pci 0000:00:1f.3: error: debugfs write failed to idle -16

This is problematic because this is reported in the test_speaker.sh test while the error above is related to an IPC flood issue.

We do need to have a better reporting of IPC flood errors...

[ranj063]

@plbossart I'm not sure this is a kernel issue. I think this is a problem in the test script. The error should have been reported for the IPC flood test as a failure because the device couldnt suspend after the IPC flood test completed. And of course its a different issue as to why the device failed to idle in the first place.

[fredoh9]

@plbossart @ranj063 I will assign this to myself.

First ipc_flood_test didn't go through. But it returned PASS. Looks I need to udpate IPC Flood test script.

2020-06-16 14:49:33 UTC [REMOTE_INFO] Dumping test logs!
2020-06-16 14:49:33 UTC [REMOTE_INFO] Test Result: PASS!

in good case,

2020-06-16 14:48:00 UTC [REMOTE_INFO] Dumping test logs!
[   70.091602] sof-audio-pci 0000:00:1f.3: IPC Flood count: 10000, Avg response time: 50780ns
[   70.091603] sof-audio-pci 0000:00:1f.3: Max response time: 452093ns
[   70.091603] sof-audio-pci 0000:00:1f.3: Min response time: 32028ns
2020-06-16 14:48:00 UTC [REMOTE_INFO] Test Result: PASS!

2nd, the device failed to idle. Need to debug why so.

kernel: [ 96.248808] sof-audio-pci 0000:00:1f.3: error: debugfs write failed to idle -16

[fredoh9]

Correction: above logs difference is due to dynamic log verbosity difference. We need to match same verbosity for all CI devices. I will submit a issue.

This error is treated to ignore for a long time.
[ 96.248808] sof-audio-pci 0000:00:1f.3: error: debugfs write failed to idle -16

__pm_runtime_suspend() return -16, Device or resource busy. @ranj063 @plbossart not sure we should worry about this error. Looks we don't have any issue with this error.

For one failure in CI report: https://sof-ci.01.org/linuxpr/PR2201/build3950/devicetest/
I don't think it is related with debugfs write failure.

16:Write error: -32,Broken pipe
2020-06-16 14:54:40 UTC [REMOTE_ERROR] speaker test failed

[fredoh9]

Sorry for lazy update on this issue. This issue have evolved to dynamic message filter and log filtering/scanning.

1. Same sof-dyndbg.conf is deployed to all test DUTs, this is done and up and running
2. Remove sof-audio before scanning kernel log, Include errors outside sof-audio before scanning dmesg for errors, ignore only specific known issues sof-test#290 (+1 approved)
3. Giving a warning when sof-dyndbg.conf is not available in /etc/modprobe.d/, env-check: add sof-dyndbg.conf file check sof-test#260 (waiting for approval)

I'm planning to close this when 2nd item is merged.

[fredoh9]

thesofproject/sof-test#290 is merged. I will monitor test results for next a few days.