Interrupt status missed by the host pipeline task during the playback of input stream over mux

[maruthi-machani]

Describe the bug
host pipeline task is not scheduled.
The timing issue in the scheduler as playback dma interrupt would occur in between the scheduling of two tasks /pipeline(host and mux) state check of playback, so one task(host) didn't update the state and the other task(mux, which is registrable task clears) cleared the interrupt leading to skip the execution of the host task and creating underrun in the pipeline and buzz noise.

To Reproduce
Long run: playback of input stream with mux comp in pipe, buzz noise observed within 2 hours.

Observations:

1. Host pipeline not scheduled attaching the screenshot and logs with and without host pipeline schedule.
   logs.zip
2. host component state is not changed to pending
   
   

Detatils of Topology

1. Three pipelines are used to support the mux usecase.
2. two input host pipelines connected to the mux-dai pipeline.
   Attaching the topology for reference.
   sof-acp-rmb-tdm8ch-dmic4ch.m4.txt
   
   

[plbossart]

The topology description doesn't match the pictures above

# PCM0 ----> buffer ------+
#                         |----> muxdemux ----> buffer ---->  ACPSP_play
# PCM1 ----> buffer ----->
#
# PCM0 <---- muxdemux <---- buffer <---- ACPSP_Capture

Why are there two ACPHS0 and ACPHS1 widgets connected?

[maruthi-machani]

The above graph is created from the attached topology, input stream from host pipes are tdm using the mux component, streamed over ACPHS0 in mux-dai pipe.
ACPHS0 is enabled for DSP_A mode, to support the tdm streaming.
ACPHS1 is inactive wrt sof FW, used only to provide the hw_params required for the driver.

[bhiregoudar]

Hi Liam,
I tried to assign to author, but could not as author name didn't found to assign. Could you please assign to the right person.

[lgirdwood]

@maruthi-machani is this now resolved with recent v2.6 mux fixes ?

[maruthi-machani]

@lgirdwood this issue is not related to the recent patches merged #7590.

[lgirdwood]

@bhiregoudar how are you pipelines being scheduled ? i.e which pipelies are using DMA IRQ and which using timer IRQ. I think in the whole topology diagram above that everything has to be scheduled on the same timer tick to be synchronous.

[bhiregoudar]

Playback and capture pipelines use the DMA IRQ and trace uses the timer IRQ.

The host pipelines and MUX-DAI pipelines of playback are scheduled based on the I2S DMA interrupt of MUX-DAI pipeline.

The issue occurs when there are multiple streams like playback and capture are running simultaneously.

When any of the registered I2S DMA interrupt or PDM DMA interrupt occurs, the scheduler will look for scheduling of all the pipelines in the list by checking corresponding interrupt in "domain_is_pending" through "schedule_ll_is_pending" function.

If any other pipeline interrupt also has occurs in middle of the process then there is possibility of missing scheduling pipelines. For example, there is PDM interrupt and scheduler enabled the PDM pipeline and start checking for the I2S pipelines scheduling, if there is no I2S DMA interrupt while checking the host pipelines but, I2S DMA interrupt occurs while checking the MUX-DAI pipeline, then the MUX-DAI pipeline is scheduled and I2S interrupt is cleared as MUX-DAI pipeline task is a registrable pipeline task.

In this case, the playback will suffer starvation as the host pipelines are not scheduled.

This issue is observed only when running multiple streams like capture and playback.

This issue won't be observed if only playback stream is played.

The snapshot of the logs attached by Maruthi shows the above mentioned behavior.

This is a timing issue. I believe this issue would occur with mixer topology along with DMIC capture.

[lgirdwood]

Playback and capture pipelines use the DMA IRQ and trace uses the timer IRQ.

The host pipelines and MUX-DAI pipelines of playback are scheduled based on the I2S DMA interrupt of MUX-DAI pipeline.

The issue occurs when there are multiple streams like playback and capture are running simultaneously.

Not sure if this work with DMA IRQs as they are asynchronous and I'm also not sure if others have tried a similar topology using DMA IRQs. The only way this may work with DMA mode is using larger buffers for more headroom and probably an ASRC to deal with drift between the 2 DMA IRQs.

Can you adapt your topology and schedule everything on a single timer tick ? I know this works as Intel has similar topologies using timer domain. i.e. they are synchronous and always executed in same order on each schedule tick.

[bhiregoudar]

Playback and capture pipelines use the DMA IRQ and trace uses the timer IRQ.
The host pipelines and MUX-DAI pipelines of playback are scheduled based on the I2S DMA interrupt of MUX-DAI pipeline.
The issue occurs when there are multiple streams like playback and capture are running simultaneously.

Not sure if this work with DMA IRQs as they are asynchronous and I'm also not sure if others have tried a similar topology using DMA IRQs.

There is "domain->full_sync" option available in dma_multi_chan_domain.c, we are using that option. I believe NXP/IMX is using the DMA IRQs with mixer topology which is similar to AMD's

The only way this may work with DMA mode is using larger buffers for more headroom and probably an ASRC to deal with drift between the 2 DMA IRQs.

Yaa, may work if different period times for different streams or with larger buffers.

Can you adapt your topology and schedule everything on a single timer tick ? I know this works as Intel has similar topologies using timer domain. i.e. they are synchronous and always executed in same order on each schedule tick.

We have an issue with timer domain scheduler on AMD platforms. I am not sure what is the issue, but time domain scheduler is missing to schedule the pipelines for some time period and recover again. So, we can't use the timer domain.

[lgirdwood]

Not sure if this work with DMA IRQs as they are asynchronous and I'm also not sure if others have tried a similar topology using DMA IRQs.

There is "domain->full_sync" option available in dma_multi_chan_domain.c, we are using that option. I believe NXP/IMX is using the DMA IRQs with mixer topology which is similar to AMD's

The only way this may work with DMA mode is using larger buffers for more headroom and probably an ASRC to deal with drift between the 2 DMA IRQs.

Yaa, may work if different period times for different streams or with larger buffers.

Can you adapt your topology and schedule everything on a single timer tick ? I know this works as Intel has similar topologies using timer domain. i.e. they are synchronous and always executed in same order on each schedule tick.

We have an issue with timer domain scheduler on AMD platforms. I am not sure what is the issue, but time domain scheduler is missing to schedule the pipelines for some time period and recover again. So, we can't use the timer domain.

ok - the only other thing I can suggest is that DMA mode should support a synchronised trigger(start). i.e. if one stream is running then the other stream start is blocked until the original stream DMA IRQ is asserted. This would only work when both stream share the same audio clock, but would gurantee timing was always deterministic.

Sorry, not used the full_sync option with DMA IRQs, so best to ask @dbaluta for advice here.

[bhiregoudar]

@dbaluta : Could you please advise how to synchronize the streams based on DMA IRQ's for multiple streams with multiple pipelines in full_sync mode?

[lgirdwood]

@bhiregoudar thinking aloud - timer based pipelines support a synchronised start on the next timer LL tick today, it may not take much effort to do a quick prof of concept to force this with your DMA test case. Again, this would only work if both DMAs shared the same audio time domain. Could be worth a quick try.