Bug: Response Generation Completes Without Proper Indicator

[ajmalp5]

Description

There is a case where llma.response returns data and isModelGenerating is false, but the model is still generating a response. Due to this inconsistency, it becomes impossible to determine when the response generation has fully completed, making it challenging to handle the response appropriately in the database.

Steps to reproduce

1. Start with an initial model where llama.response and llama.isModelGenerating work as expected.
2. Switch to a different model.
3. Generate responses multiple times after switching models.
4. Observe that the issue occurs where llama.response and llama.isModelGenerating is false, even though the response is still being generated

Expected Behavior
The response generation process should have a clear indicator when it has fully completed, ensuring data is stored to the database only after generation finishes.
Actual Behavior
llama.response and !llama.isModelGenerating trigger prematurely, leading to saving the response multiple times during generation.

useEffect(() => {
  const { response, isModelGenerating } = llama;
  console.log(response, isModelGenerating);
  if (llama.response && !llama.isModelGenerating) {
    console.log('Message is stored to DB after finishing generating');
    addMessage(db, parseInt(chatIdRef.current), {
      content: llama.response,
      role: Role.Bot,
    });
  }
}, [llama.response, llama.isModelGenerating]);

React Native Executorch version

0.1.2

React Native version

0.76.3

Platforms

Android, iOS

JavaScript runtime

Hermes

Workflow

Expo Dev Client

Architecture

Fabric (New Architecture)

Build type

Debug mode

Device

Real device

Device model

Iphone 14 pro

AI model

llama-3.2-1B/QLoRA/llama3_2_qat_lora.pte , v0.1.0/llama-3.2-3B/spinquant/llama3_2_3B_spinquant.pte

Performance logs

No response

Acknowledgements

Yes

[NorbertKlockiewicz]

Hello,

Thank you for reporting this issue. We are aware of it and plan to resolve it in an upcoming release.

[ajmalp5]

You're doing a great job. Thanks!