Development Roadmap (2025 H2)

[chenghuaWang]

Here is the development roadmap for H2 2025. We will pin this roadmap in Issues, and most of our subsequent work will be updated in this roadmap within Issues. In MLLM's documentation, we will archive each version of the roadmap and provide some outlooks. Contributions and feedback are welcome.

We plan to release a major MLLM version every year. The version for H2 2025 will be 2.0.0, and the main updates to be implemented in this version can be found in the Focus section.

Focus

• Refactoring from mllm-v1: Implement a more streamlined project structure; introduce a simple and user-friendly eager mode; provide MLLM static graph IR
• Support for more backends: P0-CANN, and P1-CUDA/AMD NPU
• Experimental attempt: Compilation from MLLM static graph IR to NPU backend
• Provide user-friendly components such as pymllm, mllm-cli, and MllmCSdk to expand the adoption of the MLLM project
• Enhance the benchmarking system with a focus on optimizing Arm Kernels

Engine

• ✔️ Async IO between Disk and Host memory for MoE model @chenghuaWang
  • feat(engine): add Tracy profiler support and CPU memory-disk async I/O #497
• ✔️ inplace and redirect API for resue memory. Check mllm/models/qwen3/modeling_qwen3_fa2.hpp. @chenghuaWang
  • feat(cpu): add inplace rmsnorm implementations for fp32 and fp16 #483
• ✔️ Op Plugin system. @chenghuaWang @oreomaker
  • feat(plugin): implement plugin system for dynamic op loading #479

Model coverage

• ✔️ DeepSeek-OCR @chenghuaWang
  • feat(deepseek-ocr): deepseek-ocr support(On working) #486
• ✔️ SmolLM3-3B @nuozhihan
• InternLM2.5-1.8B @Sp0tless
• ✔️ MiniCPM2.6-Omni @oreomaker
  • feat：Add support of MiniCPM-o-2_6 model #495
  • refactor(minicpm_o): reformat code and remove unused includes #496
  • refactor(minicpm_o2_6): optimize tensor operations and memory access  #499
• Gemma3 @jialilve
• InternVL @jialilve
• GPT-OSS 20B
• Qwen3-VL
• Qwen3-Omni-30B-A3B
• Qwen3-Next

Kernels

• Arm: KleidiAI SME Kernels are supported, as the latest SoC includes SME capability. @chenghuaWang
• Arm: Improve Mllm-Blas performance.
• X86: FP32 Kernels built on top of highway. @HayzelHan
• X86: Quantized kernel, using GGUF format. @HayzelHan
• ✔️ Commons: Paged Attention kernels based on mllm's zen-file system. @chenghuaWang
  • feat(lmcache, paged_attn): implement ZenFS with mmap-based blob storage and recovery. Coworks with special paged_attn kernel #459

Backends

• Qualcomm NPU: Migrated and refactored the QNN backend from mllm-v1 to mllm-v2, leveraging the framework’s static-graph compilation to execute on QNN. @oreomaker @jialilve @Sp0tless
  • doc(Qnn): Qnn Documents #488
  • feat(qnn): Basic QNN Prefill on v2 #485
• CUDA: Future-proofed for VLA models with native FP8 and MXFP4 support. (CUDA + TileLang) @jialilve @chenghuaWang
  • feat(cuda): add CUDA backend initialization and device info test #478
• CANN: Ready for next-gen VLA models, optimized for Ascend AI accelerators. @lywbarca @yuerqiqi @chenghuaWang

Performance

• Benchmark MLLM, llama.cpp and mnn using q4_k like quantization settings. @jialilve
  • feat: Add benchmark for Qwen3 and update readme about benchmark #487
• ✔️ Fast version of qwen3 using: 1. Manually memory planning 2. Fused kernels 3. Inplace Operators etc. @chenghuaWang
  • feat(cpu): add inplace rmsnorm implementations for fp32 and fp16 #483
• ✔️ Using tracy and perfetto for performance measurement @chenghuaWang
  • feat(tracy): integrate Tracy performance profiling support #498
  • feat(perf): integrate Perfetto for performance tracing #382
• (Optional) ARM PMU Tools setup

Quantization

• kai: Quantize on any machine, packing on ARM (make mllm-convertor --pipeline xxx_kai_pipeline available on any devices).
• GGUF: GGUF Q4_K and Q6_K Quantization method on .mllm file. @HayzelHan

Compile

KV Cache Management

• Quantized KVCache: int8 per token.
• ✔️ Prefix Cache and Paged Attn: Support multi-turn chat. @chenghuaWang
  • feat(lmcache, paged_attn): implement ZenFS with mmap-based blob storage and recovery. Coworks with special paged_attn kernel #459
  • feat(service): implement interactive chat loop in qwen3 service  #462

Pymllm

• ✔️ Waiting: Awaiting PyPI's approval of our organization's application. pymllm now is available on MacOS(pip install pymllm).
  • https://pypi.org/project/pymllm/

Production Stack

• ✔️ mllm-cli: API Server and CLI Chat Interface. @yuerqiqi
  • feat: Add mllm-cli support for Qwen3 and update docs #493
• MllmCSdk: Mllm C Wrapper for mllm-cli and other language usage. @yuerqiqi

[chenghuaWang]

[P0] The MLLM plugin interface requires refactoring to leverage TVM-FFI bindings and ensure ABI compatibility.