Catch firmware errors reported in logs

[marc-hb]

Overdue issue for PR:

• Catch firmware errors #1075

Is your feature request related to a problem? Please describe.

Validation should stop ignoring firmware error messages at last.

Describe the solution you'd like

Catch firmware errors: implementation finalized in #1075 but held back by... firmware errors.

cc:

• Check for a valid trace for ALL tests #297

[marc-hb]

As a first step we should catch only Zephyr errors using the is_zephyr() function and add a FIXME for stable-v2.2

[lgirdwood]

This is more of a second order test since the feature tests will provide the definitive yes/no on overall health, this is more providing useful feedback that errors (maybe recoverable or incorrectly labelled have occurred). There are a few more higher priority feature test items so will make this P2 for time being

[marc-hb]

All the sof-test coding has already been done a long time ago. Now it's just the matter of finding a time window where there is no false positive in the logs and turning it on. This has been surprisingly low priority indeed.

Besides finding actual bugs, the other value is not drowning logs in false positives messages when a test fail for any reason.

[marc-hb]

since the feature tests will provide the definitive yes/no on overall health,

I'm not sure what you mean by first order / "feature tests"... Besides ALSABAT (aplay-only, no arecord yet - internal issue 279), the only thing sof-test checks is just the lack of error messages in the kernel logs.

[plbossart]

I tend to agree with @marc-hb. I don't know what the 'feature tests' are, and if you are referring to the sof-test scripts they only track kernel logs.

If someone adds an error message in the firmware, it means we have a non-functional solution but we grade it as acceptable? That's not ok in my book. Either it's not an error, and the log needs to be updated, or the problem is real and needs to be fixed.

P2 means in general "won't fix" so I am not really ok with the directions here.

[plbossart]

And to be clear, ignoring fw logs should only be allowed for devices that are in early stages of development. For "mature" products based on cAVS or ACE1.x, we have no excuses, do we?

[marc-hb]

For "mature" products based on cAVS or ACE1.x, we have no excuses, do we?

See some samples over the last year in #1075

It's mostly OK but there's always some regression or false positive creeping up. It's the usual game of whack-a-mole as always when something is not in CI or left red for more than a couple days.