add resume_false_interruption and pause/resume the audio output

[longcw]

No description provided.

[longcw]

@theomonnom can you review this one?

[theomonnom]

this lg, I'm just not sure about the scenario where a SpeechHandle created using say/generate_reply should be "restarted".

[theomonnom]

Just saw one of those examples in dz's PR:

        await self.session.say("connecting you to the customer now.")
        await self.session_manager.merge_calls()

[longcw]

this lg, I'm just not sure about the scenario where a SpeechHandle created using say/generate_reply should be "restarted".

we can add the source back to the event, and skip retry for source say in the default callback.

[theomonnom]

The tricky thing if splitting the frames in smaller size, is that as soon as the asyncio loop is slower or the cpu usage is high, the audio may be stuttering.

You should be able to see that when using the stress cmd on mac

[theomonnom]

I remember having a lot of issues around that when initially developing agents. This is why we can push faster than realtime from the Python side (from this PR)

[longcw]

but we still have a reasonable buffer (200ms) for the AudioSource, the chunking here is just to ensure a single frame is not too big that we paused too late.

[theomonnom]

I see, so the interruptions has a min of 200ms latency here? Would be good to check if 200ms is enough. The only real reason we need big buffers is because users may write slow code on the event loop

[longcw]

we can use 100ms or 200ms, but I am wondering, what will happen if we call audio_source.clear_queue() when await audio_source.capture(frame) didn't return, how can we get the queued audio back from the audio_source?

if we can get the audio back, then the queue size and frame size don't matter, I can implement the pause in a different way when pause is called.

[theomonnom]

There is no easy way to get the audio back. But it's a complete OK assumption to assume that the playout was realtime. So 2s passed = 2s to discard

[longcw]

right now we will have a max of frame_size latency to pause the audio, and will drop max to 200ms (audio_source queue size) of audio when pause.

[theomonnom]

maybe 200ms is completely fine in both naturalness and cpu stress

[longcw]

keep this event or replace it with a callback?

[theomonnom]

We should deprecate it for now, to avoid breaking changes, logging a warning is fine for now

[theomonnom]

Suggested change

	agent_false_interruption_timeout: float | None = 2.0,
	false_interruption_timeout: float | None = 2.0,

I think it's fine to call it this way, user_false_interruption_timeout wouldn't make a ton of sense anyway

[longcw]

this will be a breaking change, should we change a name and log deprecate warning if user set agent_false_interruption_timeout?

[theomonnom]

lgtm!!