[BUG] DSM standalone Multicore Stream Stall

[zrombel]

Describe the bug
In DSM test where DSM component is run on CORE 1 and rest of topology on default CORE 0 after triggering pipelines start, playback pipeline stops copying data. Test were performed to check if PR #4088 fixes issue #3947. Root cause seems to be different - error is caused by pipeline XRUN visible in attached trace_logs.

Topology

    +----------------------------------------------+
    | +------+   +---+   +- -+   +---+   +-------+ |
    | | Host |-->|Buf|-->|Dsm|-->|Buf|-->|SSP Dai|-------+
    | +------+   +---+   +---+   +---+   +-------+ |     |
    +------------------+   ^   +-------------------+     |
                       |   |   |                         |
                       | +---+ |                         |
                       | |Buf| |                         |
                       | +---+ |                         |
                       +---^---+                         |
    +----------------------|------------------------+    |
    | +------+   +---+   +-|--+   +---+   +-------+ |    |
    | | Host |<--|Buf|<--|Dmux|<--|Buf|<--|SSP Dai|<-----+
    | +------+   +---+   +----+   +---+   +-------+ |
    +-----------------------------------------------+

To Reproduce
Run Python test:
22_05_TestDsmMulticoreSsp48000Hz24b32b2ch or 22_05_TestDsmMulticoreSsp48000Hz32b32b2ch
with parameters:
--playback-iterations=100

Reproduction Rate
~2/100

Environment

1. Name of the platform(s) on which the bug is observed.
   • Platform: TGL B0 RVP
2. Firmware branch name and commit
   • Branch: main
   • Hash: 3500d0e
   • Config: defconfig + chromeconfig
   • Config: defconfig
3. Python tests branch name and commit
   • Branch: master
   • Hash: 825533f4e0d3ff5e2425abb50e9785916d7d1044

Logs
22_05_TestDsmMulticoreSsp_StreamStall.zip

[keyonjie]

@zrombel do you mean PR #4088 is not enough to fix this? Can you help to do a quick check if PR #4089 help?

[keyonjie]

#4089 can't address this issue, I can reproduce this and have some clue about it, but need to debug it after May vacation.

[zrombel]

Description updated, bug is also present in build with def_config only.

[lgirdwood]

This is an asynchronous scheduling issue. We have everything running synchronously on core 0 and DSM asynchronously (wrt core 0 pipeline copy status).
The fix is to make sure that

1. DSM source/sink buffers are large enough to cope with scheduling differences.
2. DSM should continue if reference buffer is empty- treat as 0s
3. Demux should continue if reference buffer is full.

[zrombel]

Just to share quick tests results:
Unfortunately #4089 doesn't fix DSM multicore issue. Results look very similar as main branch results.
Test info: Simple - 1 iteration, Stress 20 - 20 playback iterations, Stress 200 - 200 playback iterations

		sof_main_c2fe4ff10ba_defconfig	sof_main_c2fe4ff10ba_chromeconfig	sof_keyon_ll_37f11e33_defconfig	sof_keyon_ll_37f11e33_chromeconfig
22_00_TestDsmSsp48000Hz24b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	PASS	PASS	PASS	PASS
	Stress 200	PASS	Glitch 1/200	PASS	Silent begginig 1/200
22_00_TestDsmSsp48000Hz32b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	PASS	PASS	PASS	PASS
	Stress 200	Glitch 1/200	PASS	PASS	PASS
22_05_TestDsmMulticoreSsp48000Hz24b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	Glitch 4/20	Stream Stall	Stream Stall	Stream Stall
	Stress 200	Stream Stall	Stream Stall	Stream Stall	Stream Stall
22_05_TestDsmMulticoreSsp48000Hz32b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	Minor Glitch 1/20	Glitch 4/20	Stream Stall	Stream Stall
	Stress 200	Stream Stall	Stream Stall	Stream Stall	Stream Stall

[lgirdwood]

@zrombel thanks for the valuable info. This looks like a fix for rc2 and TGL12 is needed. @keyonjie we need to fix this in topology to account for the async nature of the work running on core0 and core1 with respect to each other.

[keyonjie]

@lgirdwood @zrombel @slawblauciak I just root caused this today, will submit fix tomorrow.

[keyonjie]

@zrombel I just updated my PR #4089 which address the stream stall issue at my end, can you help to try it?