NPU models fail during Agent Framework fan‑out (parallel inference) while CPU models succeed

[leestott]

Issue reported via Foundry Discord https://aka.ms/foundry/discord
Foundry Local

Description
When using Foundry Local with the Microsoft Agent Framework, CPU models run successfully, but NPU‑backed models fail at the fan‑out stage when multiple parallel LLM calls are made.
This occurs even on systems with sufficient shared memory (32 GB). Single, serialized inference works on NPU; failures only appear when fan‑out / parallel orchestration is used.
This makes NPU models unusable today for multi‑agent or fan‑out workflows.

Repro steps
Install Foundry Local on a Copilot+ PC with NPU support (Windows 11 24H2)
Clone and run:
https://github.com/leestott/agentframework--foundrylocal
Configure the agent to use:

CPU model → ✅ succeeds
NPU model → ❌ fails during fan‑out

Observe failure when multiple concurrent LLM calls are triggered

Expected behavior
NPU models should either:
Support fan‑out / parallel inference, or
Fail gracefully with a clear error indicating that parallel execution is not supported

Actual behavior
CPU models complete successfully
NPU models fail at fan‑out / parallel execution stage
Failure appears related to NPU execution provider session / memory constraints rather than total system memory

Environment
Foundry Local: latest 0.8.119 (as of March 2026)
OS: Windows 11 24H2
Hardware: Copilot+ PC with NPU QNN ARM
Memory: 32 GB shared system memory
Agent Framework: used via agentframework--foundrylocal sample

Notes
NPU works correctly for single-agent, serialized inference
Issue only occurs with parallel / fan‑out agent execution
This appears to be a limitation of current NPU execution providers (session concurrency, buffer/KV cache duplication)

Clarifying whether this is a known limitation (and documenting it) would help developers avoid confusion when targeting NPUs with Agent Framework.

User asked to validate issue confirm using NPU model Qwen2.5-7b
Upload logs and Screenshot of demo and issue

[leestott]

I ran the demo on a ARM Snapdragon machine

NO ISSUE Found and demo runs successfully Validated on ARM64 Windows 11 AI PC

Spec
System Model: 21N10007UK
System Type: ARM64-based PC
Processor(s): 1 Processor(s) Installed.
BIOS Version: LENOVO N42ET95W (2.25 ), 23/10/2025
Total Physical Memory: 32,317 MB
Windows Version 10.0.26200 Build 26200
Version 25H2

Foundry Local Version
Foundry Local version 0.8.117

Log file from /users/username/.foundry/logs

.log file No error shown

2026-03-10 19:16:23.235 +00:00 [INF] Loaded cached model info for 30 models. SavedAt:10/03/2026 19:05:07
2026-03-10 19:16:23.255 +00:00 [INF] HandleChatCompletionRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:(null) maxTokens:(null) temperature:(null) topP:(null)
2026-03-10 19:16:23.302 +00:00 [INF] AppendTokenSequences -> numTokensAppended:684
2026-03-10 19:16:28.846 +00:00 [INF] Using new generator
2026-03-10 19:16:48.961 +00:00 [INF] HandleChatCompletionRequest elapsed time:00:00:25.7042709
2026-03-10 19:16:48.963 +00:00 [INF] UserAgent:agent-framework-python/0.0.0 Command:OpenAIChatCompletions Status:Success Direct:True Time:25741ms
2026-03-10 19:16:50.556 +00:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:(null) maxTokens:(null) temperature:(null) topP:(null)
2026-03-10 19:16:50.568 +00:00 [INF] Loaded cached model info for 30 models. SavedAt:10/03/2026 19:05:07
2026-03-10 19:16:50.584 +00:00 [INF] HandleChatCompletionRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:(null) maxTokens:(null) temperature:(null) topP:(null)
2026-03-10 19:16:50.605 +00:00 [INF] AppendTokenSequences -> numTokensAppended:853
2026-03-10 19:16:58.036 +00:00 [INF] Using new generator
2026-03-10 19:17:17.964 +00:00 [INF] HandleChatCompletionRequest elapsed time:00:00:27.3783167
2026-03-10 19:17:17.964 +00:00 [INF] UserAgent:agent-framework-python/0.0.0 Command:OpenAIChatCompletions Status:Success Direct:True Time:27407ms
2026-03-10 19:18:54.096 +00:00 [INF] Command:SessionInteractive Status:Success Direct:False Time:697270ms
2026-03-10 19:18:54.096 +00:00 [INF] Command:ModelRun Status:Success Direct:True Time:719474ms

Application and multi agent successfully executed NO ERRORS

[natke]

Can you please check your NPU driver version?

• Open the Device Manager on your Windows system.
• Look for "Neural Processors" in the list of devices.
• Right-click on the NPU device and select "Properties".
• In the Properties window, you will find the driver version listed under the "Driver Version" tab.

[leestott]

Confirmed working with following driver

To check your driver run the Powershell Command

Get-WmiObject Win32_PnPSignedDriver | Where-Object { $_.DeviceName -match "Hexagon.*NPU" } | Select-Object DeviceName, DriverVersion, DriverDate, Manufacturer, InfName | Format-List 

My machine is running

DeviceName : Snapdragon(R) X Elite - X1E78100 - Qualcomm(R) Hexagon(TM) NPU
DriverVersion : 30.0.146.0
DriverDate : 20250208000000.***+
Manufacturer : Qualcomm Technologies, Inc.
InfName : oem211.inf

[JasonB-IBCP]

I'm running into a similar issue. I'll post the log tomorrow.
Here's my driver version

Get-WmiObject Win32_PnPSignedDriver | Where-Object { $_.DeviceName -match "Hexagon.*NPU" } | Select-Object DeviceName, DriverVersion, DriverDate, Manufacturer, InfName | Format-List

DeviceName : Snapdragon(R) X Elite - X1E80100 - Qualcomm(R) Hexagon(TM) NPU
DriverVersion : 30.0.31.180
DriverDate : 20240902000000.***+
Manufacturer : Qualcomm Technologies, Inc.
InfName : oem349.inf

[natke]

Please update to the latest driver here: https://softwarecenter.qualcomm.com/catalog/item/Qualcomm_HND

[leestott]

@JasonB-IBCP please update your driver I have validated that this works successfully with
DriverVersion : 30.0.146.0

[JasonB-IBCP]

I found a driver pack for the surface last night on Microsoft's site. it brought me to

DeviceName : Snapdragon(R) X Elite - X1E80100 - Qualcomm(R) Hexagon(TM) NPU
DriverVersion : 30.0.219.1000
DriverDate : 20251109000000.***+
Manufacturer : Qualcomm Technologies, Inc.
InfName : oem424.inf

I'll test with it and a NPU model this morning

[JasonB-IBCP]

11:10:50 app.foundry_boot INFO Bootstrapping Foundry Local with alias 'qwen2.5-7b' …
11:10:52 app.foundry_boot INFO ✅ Foundry Local ready → endpoint=http://127.0.0.1:59543/v1 model=qwen2.5-7b-instruct-qnn-npu:2
Model : qwen2.5-7b-instruct-qnn-npu:2 (alias: qwen2.5-7b)
Endpoint: http://127.0.0.1:59543/v1

Log
2026-03-11 11:07:47.760 -04:00 [INF] 🟢 Model qwen2.5-7b-instruct-qnn-npu:2 loaded successfully
2026-03-11 11:07:47.760 -04:00 [INF] Command:ModelLoad Status:Success Direct:False Time:9497ms
2026-03-11 11:07:47.761 -04:00 [INF] Accepting pipe incoming pipeName:inference_agent numOfSession:13
2026-03-11 11:08:06.261 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
2026-03-11 11:08:06.265 -04:00 [INF] Loaded cached model info for 30 models. SavedAt:3/10/2026 4:40:18 PM
2026-03-11 11:08:06.273 -04:00 [INF] UserAgent:FoundryLocal OpenAI/2.2.0-beta.4 (.NET 9.0.8; Microsoft Windows 10.0.26100) Command:OpenAIChatCompletions Status:Success Direct:True Time:10ms
2026-03-11 11:08:06.273 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
2026-03-11 11:08:06.305 -04:00 [INF] AppendTokenSequences -> numTokensAppended:39
2026-03-11 11:08:06.549 -04:00 [INF] Using new generator
2026-03-11 11:08:15.716 -04:00 [INF] TTFT: 0 ms, TTST: 83
2026-03-11 11:08:15.716 -04:00 [INF] Completion stats: Total Tokens/Second: 12.3, Time to First Token: 276 ms, Average Token Generation Time: 80.1 ms, Total Time: 9444 ms, Total Tokens: 116
2026-03-11 11:08:15.717 -04:00 [INF] Ran qwen2.5-7b-instruct-qnn-npu:2 on QNNExecutionProvider, TimeToFirstTokenMs:276, TotalTimeMs:9444, TotalTokens:116, Turns:1
2026-03-11 11:08:37.188 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
2026-03-11 11:08:37.193 -04:00 [INF] Loaded cached model info for 30 models. SavedAt:3/10/2026 4:40:18 PM
2026-03-11 11:08:37.199 -04:00 [INF] UserAgent:FoundryLocal OpenAI/2.2.0-beta.4 (.NET 9.0.8; Microsoft Windows 10.0.26100) Command:OpenAIChatCompletions Status:Success Direct:True Time:11ms
2026-03-11 11:08:37.199 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
2026-03-11 11:08:37.240 -04:00 [INF] AppendTokenSequences -> numTokensAppended:175
2026-03-11 11:08:37.942 -04:00 [INF] Using new generator
2026-03-11 11:09:19.326 -04:00 [INF] TTFT: 1 ms, TTST: 83
2026-03-11 11:09:19.326 -04:00 [INF] Completion stats: Total Tokens/Second: 11.6, Time to First Token: 740 ms, Average Token Generation Time: 85.2 ms, Total Time: 42125 ms, Total Tokens: 490
2026-03-11 11:09:19.328 -04:00 [INF] Ran qwen2.5-7b-instruct-qnn-npu:2 on QNNExecutionProvider, TimeToFirstTokenMs:740, TotalTimeMs:42125, TotalTokens:490, Turns:2
2026-03-11 11:10:33.142 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
2026-03-11 11:10:33.147 -04:00 [INF] Loaded cached model info for 30 models. SavedAt:3/10/2026 4:40:18 PM
2026-03-11 11:10:33.153 -04:00 [INF] UserAgent:FoundryLocal OpenAI/2.2.0-beta.4 (.NET 9.0.8; Microsoft Windows 10.0.26100) Command:OpenAIChatCompletions Status:Success Direct:True Time:11ms
2026-03-11 11:10:33.153 -04:00 [INF] HandleChatCompletionAsStreamRequest -> model:qwen2.5-7b-instruct-qnn-npu:2 MaxCompletionTokens:2048 maxTokens:(null) temperature:(null) topP:(null)
2026-03-11 11:10:33.207 -04:00 [INF] AppendTokenSequences -> numTokensAppended:676
2026-03-11 11:10:35.891 -04:00 [INF] Using new generator
2026-03-11 11:10:38.147 -04:00 [INF] TTFT: 1 ms, TTST: 90
2026-03-11 11:10:38.147 -04:00 [INF] Completion stats: Total Tokens/Second: 5.2, Time to First Token: 2738 ms, Average Token Generation Time: 188 ms, Total Time: 4992 ms, Total Tokens: 26
2026-03-11 11:10:38.148 -04:00 [INF] Ran qwen2.5-7b-instruct-qnn-npu:2 on QNNExecutionProvider, TimeToFirstTokenMs:2738, TotalTimeMs:4992, TotalTokens:26, Turns:3
2026-03-11 11:10:39.859 -04:00 [INF] Command:SessionInteractive Status:Success Direct:False Time:172098ms
2026-03-11 11:10:39.859 -04:00 [INF] Command:ModelRun Status:Success Direct:True Time:182152ms
2026-03-11 11:10:39.859 -04:00 [INF] Stream disconnected
2026-03-11 11:10:50.307 -04:00 [INF] Starting Foundry Local CLI with 'service status'
2026-03-11 11:10:50.318 -04:00 [INF] Found service endpoints: http://127.0.0.1:59543
2026-03-11 11:10:50.338 -04:00 [INF] UserAgent:foundry-local-CLI/0.8.119 Command:ServiceStatus Status:Success Direct:True Time:0ms
2026-03-11 11:10:50.339 -04:00 [INF] Command:ServiceStatus Status:Success Direct:True Time:27ms
2026-03-11 11:10:50.339 -04:00 [INF] Stream disconnected
2026-03-11 11:10:50.935 -04:00 [INF] Starting Foundry Local CLI with 'service status'
2026-03-11 11:10:50.950 -04:00 [INF] Found service endpoints: http://127.0.0.1:59543
2026-03-11 11:10:50.969 -04:00 [INF] Command:ServiceStatus Status:Success Direct:True Time:28ms
(.venv) PS C:\Users\JBenway.foundry\logs>

[JasonB-IBCP]

Foundry version: 0.8.119

System Model: Microsoft Surface Laptop, 7th Edition
System Type: ARM64-based PC
Processor(s): 1 Processor(s) Installed. [01]: ARMv8 (64-bit) Family 8 Model 1 Revision 201 Qualcomm Technologies Inc ~1670 Mhz
BIOS Version: Microsoft Corporation 175.138.235, 8/19/2025
Total Physical Memory: 65,017 MB
Windows Version 10.0.26100 N/A Build 26100
Version 24H2

[natke]

Is that working for you now? @JasonB-IBCP

[JasonB-IBCP]

The fan-out is still erroring.

13:36:00 app.foundry_boot INFO Bootstrapping Foundry Local with alias 'qwen2.5-7b' …
13:36:10 app.foundry_boot INFO ✅ Foundry Local ready → endpoint=http://127.0.0.1:59543/v1 model=qwen2.5-7b-instruct-qnn-npu:2
Model : qwen2.5-7b-instruct-qnn-npu:2 (alias: qwen2.5-7b)

@natke

[natke]

What is the error? Can you set foundry service set --default-log-level Debug and run again please?

[leestott]

@JasonB-IBCP

So your driver is still an old version, so I think the Issues is with your Qualcomm Snapdragon
Fix to try: Install this latets driver https://softwarecenter.qualcomm.com/catalog/item/QHND (you will need a Qualcomm Developer Account to access the download)

[natke]

@leestott what is failing here and how do we observe the failure?