feat: live dashboard monitor + serve loop improvements#5
Open
msitarzewski wants to merge 1 commit intodanveloper:mainfrom
Open
feat: live dashboard monitor + serve loop improvements#5msitarzewski wants to merge 1 commit intodanveloper:mainfrom
msitarzewski wants to merge 1 commit intodanveloper:mainfrom
Conversation
Dashboard: - ncurses-based htop-style terminal monitor (dashboard.c) - Reads /tmp/flash-moe-stats.json written by the inference server - Shows real-time status, progress bars, TTFT, tok/s, rolling averages - Auto-adapts to terminal width, clean exit with q or Ctrl+C Serve loop: - SSE streaming with per-token delta events - Dashboard stats reporting (server state, prefill progress, generation metrics) - Tool call parsing from model output (<tool_call> blocks) - Session state save/restore for multi-turn conversations - GPU KV buffer increased to 32K pre-allocation - CPU 2-bit expert forward path for fallback compute Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AndrewFarley
added a commit
to AndrewFarley/flash-moe
that referenced
this pull request
Apr 29, 2026
Applied PRs danveloper#5 and danveloper#14: Dashboard: - ncurses-based htop-style terminal monitor (dashboard.c) - Reads /tmp/flash-moe-stats.json written by inference server - Real-time status, progress bars, TTFT, tok/s, rolling averages Serve loop improvements: - SSE streaming with per-token delta events - Non-streaming JSON response mode (stream: false) - Tool call parsing from <tool_call> blocks in model output - Full OpenAI messages array parsing for generic clients - Dashboard stats reporting (server state, prefill progress, generation) - GPU KV buffer increased to 32K pre-allocation - CPU 2-bit expert forward path for fallback compute - CMD1+CMD2 merge optimization for linear attention layers - select() loop for idle stats updates, GET /stats endpoint 8-bit gate dequant (PR danveloper#14): - dequant_matvec_8bit Metal kernel (FMA-optimized) - cpu_dequant_matvec_8bit CPU fallback - BatchMatvecSpec.bits field for per-tensor bit-width dispatch - Auto-detection of gate quantization from config.json - Gate bits applied dynamically (4-bit or 8-bit based on model) Additional fixes: - BPE byte marker decoding in SSE output (Ġ→space, Ċ→newline)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dashboard.c) that reads/tmp/flash-moe-stats.jsonand shows real-time inference status, progress bars, TTFT, tok/s, and rolling averages/v1/chat/completions(OpenAI-compatible streaming)<tool_call>blocks in model output and returns structured tool_calls in the responseTesting
Tested end-to-end on Apple M5 Max (128GB RAM):
make && make dashboardbuilds cleanly./infer --serve 6601 --2bit+./dashboard— live monitoring works across idle/prefilling/generating states./infer --serve 6601(4-bit) — 10.5 tok/s served with correct SSE streamingqexit all work correctlyTest plan
make clean && make && make chat && make dashboard— all targets build./infer --serve 6601 --2bit+./chat --port 6601— interactive chat works./dashboardshows live stats during generation🤖 Generated with Claude Code