Add native post-processing module for silence removal and pitch-preserving speed adjustment

[Copilot]

Implements offline post-processing for recorded clips: removes silence via audio amplitude analysis and applies pitch-preserving speed adjustment (default 1.15x, configurable 1.0-2.0x). Supports single clip and batch processing.

Implementation

New native module: modules/video-postprocess/

• iOS (AVFoundation): Complete implementation with RMS-based silence detection, hysteresis filtering, and time-scaled speed adjustment
• Android/Web: Stubs (not implemented)
• TypeScript: Full type definitions and module interface
• Tests: Swift test suite covering silence detection, speed adjustment, and full pipeline

Architecture

Silence detection:

• AVAssetReader extracts 16-bit PCM audio samples
• Calculates RMS amplitude per buffer → dB scale
• Applies threshold (-40 dB default) with minimum duration filter (500ms default)
• Returns CMTimeRange array of silent periods

Processing pipeline:

1. Analyze audio → detect silence periods
2. Create AVMutableComposition excluding silent ranges
3. Apply time scaling to video/audio tracks (preserves pitch)
4. Export with AVAssetExportSession

Progress events:

• Phases: analyzing → removing_silence → adjusting_speed → finalizing
• Progress: 0.0-1.0 via Expo event system

API

import VideoPostprocess from '@/modules/video-postprocess';

// Single clip
await VideoPostprocess.processClip(inputURL, outputURL, {
  speedFactor: 1.15,        // 1.0-2.0x
  silenceThreshold: -40,    // dB
  minSilenceDuration: 500   // ms
});

// Batch
await VideoPostprocess.processClips([url1, url2], outputDir, options);

// Progress tracking
VideoPostprocess.addListener('onProgress', (event) => {
  // event.progress: 0.0-1.0, event.phase: string
});

Files

• ios/VideoPostprocessModule.swift (417 lines) - Core implementation
• src/VideoPostprocess.types.ts - TypeScript interface
• test/video/RunPostprocessTests.swift (283 lines) - Test suite
• README.md - API documentation

Platform support: iOS only (Android/Web throw not-implemented errors)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• cdp.expo.dev
  • Triggering command: /usr/local/bin/node /home/REDACTED/work/pulse/pulse/node_modules/@expo/cli/build/src/utils/telemetry/clients/flushFetchDetached.js /tmp/415733b71f22cb3e52ebd05828dd72f0/expo-telemetry.json (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Post-processing: Silence Removal + Pitch-Preserving Speed Up for Clips in Pulse (iOS)</issue_title>
<issue_description>### Summary
Add an offline post-processing step for recorded clips in Pulse that:

1. Removes silence using debounced hysteresis (to avoid choppy micro-cuts).
2. Optionally speeds up the remaining audio/video while preserving pitch.
   • Default speed factor: 1.15x
   • User-selectable (e.g. 1.0–2.0x).

Applies to a single clip or batch (all clips).

---

Background / Motivation

The initial PoC records “high-signal” screen content by gating on audio, but it may still leave:

• Short silent gaps.
• Slower pacing than ideal for “shorts” style content.

We want a quick post-process pass that:

• Further removes low-value silent stretches.
• Lets users slightly speed up the entire clip while keeping voices natural (no chipmunking).
• Runs entirely on-device using standard AV APIs.

This is purely an offline process on an existing clip (or a list of clips).

---

Scope

In scope:

• iOS-only implementation.
• A post-processing function that:
  • Accepts one clip (file URL) or a list of clips.
  • Produces a new processed clip (silence removed, optional speed-up).
• Simple UX to:
  • Apply to a single clip.
  • Optionally “Process all clips” (batch mode).
  • Set speed factor, defaulting to 1.15x.

Out of scope (for this ticket):

• Fancy timeline UI or visual editing.
• Cloud processing / progress sync.
• Android.

---

High-Level Requirements

1. Post-Processing API

   • Add a native function (Swift) such as:

     func processClip(
       at inputURL: URL,
       speedFactor: Float = 1.15,
       silenceThreshold: Float = defaultValue,
       completion: (Result<URL, Error>) -> Void
     )

   • Optionally a batch wrapper: processClips(_ urls: [URL], ...).

2. Silence Removal with Hysteresis

   • Read audio track via AVAssetReader → PCM buffers.
   • Compute audio level per small window (e.g. 10–50 ms):
     • Use RMS / average absolute amplitude.
   • Use debounced hysteresis to classify windows as:
     • silent (below threshold).
     • active (above threshold).
   • Build a set of keep-ranges:
     • Require min active duration to start a kept region.
     • Require min silence duration to break it.
     • Avoid micro-cuts (don’t chop for 100 ms of silence).
   • Build an AVMutableComposition:
     • For each keep-range, insert both audio and video time ranges sequentially (concatenate kept segments).

3. Speed-Up with Pitch Preservation

   • After the composition with silence removed is built:
     • If speedFactor != 1.0:
       • Scale the full composition time range by 1 / speedFactor (shorten it).
       • Set the audio track’s audioTimePitchAlgorithm, e.g.:
         • AVAudioTimePitchAlgorithmSpectral or TimeDomain.
       • This should keep pitch stable while shortening duration.
   • Alternatively (if needed later): use AVAudioEngine + AVAudioUnitTimePitch, but for PoC, prefer the simpler composition + export approach.

4. Export

   • Export the processed composition using AVAssetExportSession to a new file:
     • H.264 video, AAC audio.
     • Save to app sandbox (e.g. Documents/Processed).
   • Return the new file URL to the caller (and optionally delete the original only if the caller wants).

5. React Native Integration

   • Add a RN native module function:

     processClip(options: {
       inputPath: string;
       speedFactor?: number; // default 1.15
     }): Promise<{ outputPath: string }>

   • And optionally:

     processAllClips(options: { speedFactor?: number }): Promise<void>

   • Wire to a simple UI:

     • Per-clip “Process” button.
     • A small control (slider or segmented control) to choose speed: e.g. 1.0, 1.15 (default), 1.3, 1.5.

6. Config / Tuning

   • Expose tweakable constants in code (or config struct):
     • silenceThreshold
     • minActiveDuration
     • minSilenceDuration
   • For now, okay to hard-code defaults; no need for a full UX to tune them.

---

Technical Notes / Implementation Details

• Silence Detection

  • Audio: mono or stereo PCM; treat peak/RMS across channels.
  • Sample approach:
    • Window size: 20 ms.
    • Compute RMS per window.
    • If RMS < threshold for N windows → mark silent.
    • If RMS > threshold for M windows → mark active.

• Hysteresis

  • Use different thresholds/time for entering and leaving active state to avoid flickering.
  • Example:
    • Enter active: 0.3 s above threshold.
    • Return to silent: 0.7 s below threshold.

• Composition Assembly

  • Iterate keep-ranges in chronological order.
  • For each, insert the corresponding CMTimeRange from original asset’s video and au...

• Fixes Post-processing: Silence Removal + Pitch-Preserving Speed Up for Clips in Pulse (iOS) #225

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.