[BUG]ERROR ewt in etrace

[XiaoyunWu6666]

Describe the bug
Multiple 'ERROR ewt' in etrace log when check-runtime-pm-status on apl-up2-nocodec

To Reproduce
TPLG=sof-apl-nocodec.tplg ~/sof-test/test-case/check-runtime-pm-status.sh -l 15

Reproduction Rate
100%

Environment
Kernel Branch: topic/sof-dev
Kernel Commit: dcea8052
Kernel PR: 2938
SOF Branch: main
SOF Commit: 636cbef

Platform
APL_UP2_NOCODEC

Screenshots or console output

[  8008757921.656386] (           0.000000) c0 dma-trace             src/trace/dma-trace.c:335  ERROR FW ABI 0x3012001 DBG ABI 0x5003000 tag v1.8-rc1-114-g636cbefc62c9 src hash 0x85629837 (ldc hash 0x85629837)
[     7214463.254990] (     7214463.500000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    15517671.102133] (     8303208.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    16210527.532935] (      692856.437500) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    24186156.747262] (     7975629.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    32125762.994270] (     7939606.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    32455645.481162] (      329882.500000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    41285201.692807] (     8829556.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    50000448.794411] (     8715247.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    50010666.814838] (       10218.020508) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    58100591.857957] (     8089925.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt
[    66559722.407238] (     8459131.000000) c0 wait                         src/lib/wait.c:45   ERROR ewt

[lgirdwood]

Thanks, this is not a very helpful message and it's the last of the legacy message types.

int poll_for_register_delay(uint32_t reg, uint32_t mask,
			    uint32_t val, uint64_t us)
{
	uint64_t tick = clock_us_to_ticks(PLATFORM_DEFAULT_CLOCK, us);
	uint32_t tries = DEFAULT_TRY_TIMES;
	uint64_t delta = tick / tries;

	if (!delta) {
		delta = us;
		tries = 1;
	}

	while ((io_reg_read(reg) & mask) != val) {
		if (!tries--) {
			tr_err(&wait_tr, "ewt");
			return -EIO;
		}
		wait_delay(delta);
	}
	return 0;
}

@juimonen can you send a quick PR that prints a poll timeout message and includes the register, mask, val and us fields. This will tell us which user (dw-dma, ssp or pm on Intel platforms is showing the issue). I would also make all users check the return value too and print an error.

[keyonjie]

This is from the ssp_stop(). the change from @iuliana-prodan to stop ssp dai before stopping DMA introduce this error for us, we don't see this issue on byt/bdw as we configure CONFIG_DMA_SUSPEND_DRAIN there. @juimonen FYI in case you will work on it. The quick fix could be configure CONFIG_DMA_SUSPEND_DRAIN for all Intel platforms, but I am not very happy with that.

#if CONFIG_DMA_SUSPEND_DRAIN
		ret = dma_stop(dd->chan);
		dai_trigger(dd->dai, cmd, dev->direction);
#else
		dai_trigger(dd->dai, cmd, dev->direction);
		ret = dma_stop(dd->chan);
#endif

[juimonen]

@lgirdwood @keyonjie yeah I can do the prints, but you need to decide what is the actual fix :)

[lgirdwood]

@lgirdwood @keyonjie yeah I can do the prints, but you need to decide what is the actual fix :)

@juimonen can you do the trace update and probably @keyonjie can fix the SSP ?

[juimonen]

#4251, printing here. Only one (I think) who was not checking return in some way was sai.c

[marc-hb]

FWIW I've seen this ERROR ewt messages since forever. That's because I've been working on the logger for some time.

EDIT: they were rare though.

I made some enhancements recently but generally speaking sof-test has not cared about errors in the firmware logs:

thesofproject/sof-test#297 (comment)
thesofproject/sof-test#666
etc.

[lgirdwood]

Debug PRs merged, so it looks like we are polling on HW change that's not happening and will need inputs from @slawblauciak