feat: add comprehensive multimodal driver support

[JacobFV]

Multimodal Driver Support - Node SDK Updates

This update adds comprehensive multimodal support to the Node SDK to match the new agi-driver capabilities.

Changes Made

Protocol Updates (src/driver/protocol.ts)

New Event Types

• AudioTranscriptEvent: Audio transcript from buffer
• VideoFrameEvent: Video frame from camera/screen
• SpeechStartedEvent: TTS playback started
• SpeechFinishedEvent: TTS playback finished
• TurnDetectedEvent: Voice turn detection

New Command Types

• GetAudioTranscriptCommand: Request audio transcript
• GetVideoFrameCommand: Request video frame

New Interfaces

• MCPServerConfig: MCP server configuration
• AgentIdentity: Agent identity information
• ToolChoice: Tool choice configuration type

Updated StartCommand

Added fields for multimodal configuration:

• agent_identity?: AgentIdentity - Agent identity (default: agi-2-claude by AGI Company)
• tool_choice?: ToolChoice - Tool choice mode
• mcp_servers?: MCPServerConfig[] - MCP server configurations
• audio_input_enabled?: boolean, audio_buffer_seconds?: number
• turn_detection_enabled?: boolean, turn_detection_silence_ms?: number
• speech_output_enabled?: boolean, speech_voice?: string
• camera_enabled?: boolean, camera_buffer_seconds?: number
• screen_recording_enabled?: boolean, screen_recording_buffer_seconds?: number

Exports (src/driver/index.ts)

Added exports for all new event and command types, plus helper interfaces.

Usage Examples

Basic Multimodal Session

import { AgentDriver } from '@agi-inc/agi-node';

const driver = new AgentDriver({
  mode: 'local',
  agent_name: 'agi-2-claude'
});

// Start with multimodal features
await driver.start({
  goal: 'Help me with my computer',
  mode: 'local',
  agent_name: 'agi-2-claude',

  // Voice features
  audio_input_enabled: true,
  turn_detection_enabled: true,
  speech_output_enabled: true,
  speech_voice: 'alloy',

  // Video features
  camera_enabled: true,
  screen_recording_enabled: true,

  // MCP servers
  mcp_servers: [
    {
      name: 'filesystem',
      command: 'npx',
      args: ['-y', '@modelcontextprotocol/server-filesystem', '/path/to/dir'],
      env: {}
    }
  ],

  // Tool choice
  tool_choice: 'auto'
});

Handling New Events

driver.on('audio_transcript', (event: AudioTranscriptEvent) => {
  console.log(`Transcript: ${event.transcript}`);
});

driver.on('video_frame', (event: VideoFrameEvent) => {
  // event.frame_base64 contains JPEG frame
  saveFrame(event.frame_base64);
});

driver.on('speech_started', (event: SpeechStartedEvent) => {
  console.log(`🔊 Speaking: ${event.text}`);
});

driver.on('speech_finished', () => {
  console.log('✓ Finished speaking');
});

driver.on('turn_detected', (event: TurnDetectedEvent) => {
  console.log(`You said: ${event.transcript}`);
});

Voice-Only Mode

await driver.start({
  goal: '(voice input)',
  mode: 'local',
  audio_input_enabled: true,
  turn_detection_enabled: true,
  turn_detection_silence_ms: 1000,  // 1 second of silence = turn complete
  speech_output_enabled: true,
  speech_voice: 'alloy'  // or: echo, fable, onyx, nova, shimmer
});

MCP Servers

const mcpServers: MCPServerConfig[] = [
  {
    name: 'filesystem',
    command: 'npx',
    args: ['-y', '@modelcontextprotocol/server-filesystem', '/Users/you/Documents']
  },
  {
    name: 'database',
    command: 'python',
    args: ['-m', 'my_db_server'],
    env: { DATABASE_URL: 'postgresql://...' }
  }
];

await driver.start({
  goal: 'Analyze my documents',
  mode: 'local',
  mcp_servers: mcpServers
});

Tool Choice Configuration

// Auto (default)
tool_choice: 'auto'

// Required - must use at least one tool
tool_choice: 'required'

// None - no tool use
tool_choice: 'none'

// Specific tool
tool_choice: { type: 'tool', name: 'filesystem__read_file' }

Breaking Changes

⚠️ This is a breaking change with no backwards compatibility.

• StartCommand interface has many new optional fields
• New event types may be emitted
• agent_name should be set to "agi-2-claude" for new agents

Testing

# Install updated SDK
npm install

# Build TypeScript
npm run build

# Run tests
npm test

# Try a voice session
node -e "
const { AgentDriver } = require('./dist');

(async () => {
  const driver = new AgentDriver({ mode: 'local' });
  const result = await driver.start({
    goal: 'Test voice',
    mode: 'local',
    audio_input_enabled: true,
    speech_output_enabled: true
  });
  console.log(result);
})();
"

Related PRs

• agi-api (driver): https://github.com/agi-inc/agents/pull/344
• agi-python: https://github.com/agi-inc/agi-python/pull/8
• agi-csharp: TBD

Greptile Overview

Greptile Summary

This PR extends the driver protocol to support multimodal capabilities (audio transcript, video frames, speech/turn events) and adds corresponding command types and config interfaces. The AgentDriver is updated to forward new event types and to optionally enable voice/camera/screen/MCP features by populating the new StartCommand fields.

Key merge blockers are around MCP config handling and documentation accuracy: the default ~ path expansion currently resolves to /.agi/... (root) rather than the user’s home, the MCP loader is typed as any[] and can silently disable MCP on malformed configs, and the new MULTIMODAL_UPDATES.md examples don’t match the actual exported JS/TS API (agentName + positional start(...)).

Confidence Score: 3/5

• This PR adds useful protocol support but has a couple of correctness issues that will break expected MCP behavior and user-facing docs.
• Multimodal protocol additions look consistent, but the MCP config default path expansion resolves to filesystem root and the MCP config loader is typed as any[] and swallows parse/shape errors, making MCP silently not work in common cases. Additionally, documentation examples currently do not match the SDK API and will fail for users.
• src/driver/driver.ts, MULTIMODAL_UPDATES.md

Important Files Changed

Filename	Overview
MULTIMODAL_UPDATES.md	Adds multimodal usage doc, but examples use non-existent agent_name option and object-based start signature that doesn't match AgentDriver.start.
package.json	Bumps package version to 0.5.0; otherwise unchanged.
src/driver/driver.ts	Adds multimodal toggles and MCP config loading and wires into StartCommand; current MCP ~ expansion is incorrect and config loader is typed as any[] and swallows JSON errors.
src/driver/index.ts	Exports newly added protocol types and configs; no issues found.
src/driver/protocol.ts	Extends protocol with multimodal events/commands and helper types; parseEvent/serializeCommand remain exported.
package-lock.json	Lockfile updates for version bump / dependency metadata; no functional code changes.

Sequence Diagram

sequenceDiagram
  autonumber
  participant App as User App
  participant SDK as AgentDriver
  participant FS as MCP Config File
  participant Driver as agi-driver

  App->>SDK: new AgentDriver(options)
  App->>SDK: start(goal, screenshot?, w?, h?, mode?)
  SDK->>Driver: spawn(binaryPath)
  Driver-->>SDK: stdout {event:"ready"}
  SDK->>SDK: build StartCommand (multimodal fields)
  alt MCP enabled
    SDK->>FS: read mcpConfig path
    FS-->>SDK: JSON config
    SDK->>SDK: map config -> mcp_servers[]
  end
  SDK->>Driver: stdin serializeCommand(StartCommand)
  loop driver emits events
    Driver-->>SDK: stdout DriverEvent (e.g., action/thinking/audio_transcript)
    SDK-->>App: emit(event.event)
  end
  Driver-->>SDK: stdout {event:"finished"}
  SDK-->>App: resolve start() with DriverResult

Loading

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

src/driver/protocol.ts
StartCommand now required

StartCommand now requires mode and agent_name (protocol.ts:228-231), but src/driver/driver.ts still constructs StartCommand with mode: mode ?? this.mode where this.mode can be undefined and with agent_name: this.agentName || undefined (driver.ts:247-251). With the new types this becomes a type error, and at runtime it can emit invalid JSON to the driver. Either keep these fields optional in the protocol, or update DriverOptions/constructor defaults so mode/agent_name are always set before sending start.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/driver/protocol.ts
Line: 219:232

Comment:
**StartCommand now required**

`StartCommand` now requires `mode` and `agent_name` (`protocol.ts:228-231`), but `src/driver/driver.ts` still constructs `StartCommand` with `mode: mode ?? this.mode` where `this.mode` can be `undefined` and with `agent_name: this.agentName || undefined` (`driver.ts:247-251`). With the new types this becomes a type error, and at runtime it can emit invalid JSON to the driver. Either keep these fields optional in the protocol, or update `DriverOptions`/constructor defaults so `mode`/`agent_name` are always set before sending `start`.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

_{5 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

src/driver/driver.ts
Docs/examples don’t match API

AgentDriver.start is still a positional-args API (start(goal, screenshot?, width?, height?, mode?)) in src/driver/driver.ts:166-173, but MULTIMODAL_UPDATES.md documents calling start({...}) with a StartCommand-shaped object. As-is, those examples will throw at runtime (object passed as goal string) and the multimodal options can’t be provided via the public API. Either update the docs to match the actual signature, or (if intended) change AgentDriver.start to accept an options object and forward the new multimodal fields.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/driver/driver.ts
Line: 166:173

Comment:
**Docs/examples don’t match API**

`AgentDriver.start` is still a positional-args API (`start(goal, screenshot?, width?, height?, mode?)`) in `src/driver/driver.ts:166-173`, but `MULTIMODAL_UPDATES.md` documents calling `start({...})` with a `StartCommand`-shaped object. As-is, those examples will throw at runtime (object passed as `goal` string) and the multimodal options can’t be provided via the public API. Either update the docs to match the actual signature, or (if intended) change `AgentDriver.start` to accept an options object and forward the new multimodal fields.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

_{5 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{6 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[NamanGarg20]

lgtm