Implement cross-platform Voice WebRTC AI Programmer with Flutter

[Copilot]

Scaffolds complete voice-driven AI programming assistant enabling natural language code generation through Voice→Realtime API→MCP→GitHub workflow.

Framework Selection

Evaluated Flutter, Kotlin Multiplatform, Qt, and React Native against WebRTC support, Realtime API integration, MCP protocol capability, cross-platform coverage, and developer experience. Selected Flutter (score 9.05/10) for superior WebRTC support via flutter_webrtc, comprehensive platform coverage, and mature tooling.

Architecture

Core Pipeline

• lib/webrtc/ - Microphone capture (24kHz PCM), audio buffering, base64 encoding
• lib/realtime/ - WebSocket client for OpenAI Realtime API, response handler for transcriptions/function calls
• lib/mcp/ - JSON-RPC 2.0 client, tool executor for file operations (create/edit/delete/read)
• lib/github/ - Repository management, code generation, commit handling
• lib/ui/ - Material Design 3 interface with Provider state management

State Flow

AppStateManager orchestrates services:
  WebRTCService.startStreaming() 
    → RealtimeAPIClient.sendAudio()
    → ResponseHandler.onFunctionCall()
    → MCPToolExecutor.executeCodeAction()
    → GitHubCopilotService.updateFile()

Implementation Details

Audio Processing

• WebRTC constraints: echo cancellation, noise suppression, auto gain control
• Chunk-based buffering (100ms default) for streaming
• Proper base64 encoding via dart:convert

Realtime API Integration

• Session configuration with tools schema for MCP function calling
• Server-side VAD (threshold 0.5, 500ms silence detection)
• Message handler for 15+ event types (transcription, audio deltas, function calls)

MCP Protocol

• Mock server responses for development/testing
• Extensible transport layer (HTTP/stdio/SSE ready)
• Tool discovery and execution framework

Security

• Memory-only credential storage, no persistence
• Explicit GitHub Actions permissions (contents: read)
• Auto-fetch SHA for GitHub file updates when not provided

Testing & CI/CD

• Unit tests for WebRTC, Realtime API, MCP, AppState
• Multi-platform CI: Android (APK), iOS (IPA), Linux/macOS/Windows builds
• CodeQL scan: 0 vulnerabilities

Documentation

Comprehensive docs covering framework evaluation rationale, system architecture with data flow diagrams, configuration guide, quickstart, security policy, and contributing guidelines.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://storage.googleapis.com/flutter_infra_release/flutter/a5cb96369ef86c7e85abf5d662a1ca5d89775053/dart-sdk-linux-x64.zip
  • Triggering command: /update-job-proxy /update-job-proxy ACCEPT b35095b483b1479d/etc/ssl/certs/DigiCert_Trusted_Root_G4.pem sed -e A.pem -e test -e s/,/_/g test sed -e ial_Root_CA_1.pem -e /usr/bin/sed -e s/,/_/g /usr/local/sbin/-t sed (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Create cross-platform Voice WebRTC AI Programmer (Android/iOS/Desktop). Scaffold: src/webrtc/ (mic streaming), src/realtime/ (OpenAI Realtime API client), src/mcp/ (MCP protocol), src/github/ (Copilot integration), src/ui/ (connect, mic toggle, logs, repo selector), tests/, .github/workflows/. Add README.md, docs/FRAMEWORK_EVAL.md (evaluate Flutter/KMP/Qt/ReactNative & select: WebRTC, Realtime API, MCP, cross-platform, dev experience, ecosystem). Voice→Realtime→MCP→Copilot→code workflow.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[paulpv]

After evaluating cross platforms I am closing this Flutter based PR and will open a new one to implement this in Compose Multi Platform.