[DMP 2026]: Create Intelligent Closed Caption (CC) Suggestion Tool

[keerthiseelan-planetread]

Ticket Contents

Description

Our goal is to develop an AI-powered tool that intelligently identifies moments in a video where a Closed Caption (CC) annotation is genuinely necessary — such as when a non-speech audio event meaningfully affects the speakers or the scene — and suggests contextually relevant CC text, without over-captioning routine or low-impact sounds. The tool will analyze both the audio and visual tracks together to determine whether a non-speech event is significant enough to warrant a CC, reducing the manual effort of editors and accessibility teams who currently add CC annotations by hand.

Goals & Mid-Point Milestone

Goals

• Goal 1: Sound Event Detection Module Automatically detect and classify non-speech audio events in a given video file with confidence scores and timestamps. Steps Involved: The video file is taken as input. The audio track is extracted and passed through an open-source sound event detection model. The model classifies events such as honking, explosions, laughter, music, glass breaking, alarms, and applause. The output is a list of detected events with confidence scores and start/end timestamps.

• Goal 2: Speaker Reaction Detection Module (Mid-Point Milestone) Detect visible speaker or scene reactions to audio events using visual analysis of video frames. Steps Involved: At each detected audio event timestamp, the corresponding video frames are extracted. A visual analysis model detects reactions such as head turns, startled body language, paused speech, or facial expressions. A reaction confidence score is assigned per event and stored alongside the audio event data for downstream combination.

• Goal 3: CC Decision Engine & SRT/SLS Output Combine audio event signals and visual reaction signals to make a CC/no-CC decision and generate a labelled output file. Steps Involved: The audio event confidence and visual reaction confidence are combined to determine whether a CC is warranted. A CC text label is auto-generated for each accepted event (e.g., [honking], [gunshot], [crowd cheering]). The accepted suggestions are exported with correct timestamps into a standard SRT or SLS file. The tool is tested on a sample set of Hindi and regional-language content and feedback is collected from editors on suggestion accuracy.

• The midpoint milestones will be completion of Goal 1 and Goal 2.

Setup/Installation

No response

Expected Outcome

The Intelligent Closed Caption (CC) Suggestion Tool is a Python-based backend pipeline that accepts any video file as input and produces a ready-to-use SRT or SLS file containing only contextually meaningful, non-speech closed caption annotations — reducing manual effort for accessibility editors and teams working on Hindi and regional-language content.

Acceptance Criteria

The tool should successfully detect non-speech audio events, assess speaker/scene reaction, and produce a CC-annotated SRT or SLS file for any given video file. It must avoid over-captioning ambient sounds that do not affect the speakers or narrative.

Implementation Details

Open-source stack — Python, audio event detection model (e.g., YAMNet or PANNs), OpenCV (frame extraction), MediaPipe or similar (pose and expression analysis), decision combiner logic, SRT/SLS file output.

Mockups/Wireframes

No response

Product Name

Intelligent Closed Caption (CC) Suggestion Tool

Organisation Name

Planet Read

Domain

⁠Education

Tech Skills Needed

Artificial Intelligence, Computer Vision, Python, Machine Learning

Mentor(s)

@abinash-sketch @keerthiseelan-planetread

Category

Backend, Machine Learning, AI

[amitkumarashutosh]

Hi @abinash-sketch and @keerthiseelan-planetread,
I am interested in contributing to this project. The signal fusion part, combining audio event confidence with visual reaction scores to make a CC/no-CC call, seems like it will be the trickiest bit to get right, especially around avoiding over-captioning sounds that do not really affect the scene.

One thing I wanted to ask early: is there a preference between YAMNet and PANNs for the audio model? YAMNet is much lighter to run, but PANNs gives finer-grained classifications which might matter depending on how specific the CC labels need to be. Happy to benchmark both if that is useful.
I have already started working on an initial implementation and will update you soon.

Also wanted to flag that the link to this issue on the C4GT Badal platform seems to redirect to a different issue instead of landing here and shows issue has been deleted.

[Eshaan-byte]

Hi @abinash-sketch and @keerthiseelan-planetread

I'd love to work on this project; the idea of combining audio and visual signals to decide when a CC actually matters is a really interesting problem.

I've been reading through the goals, and a few things came to mind:

For the audio side, let's start with YAMNet since it's light and fast, and we can always compare it against PANNs later if we need finer classifications. One thing I'm curious about, though: since YAMNet is trained on AudioSet, which is mostly English-centric data, how well do we expect it to handle sounds common in Hindi/regional content? Things like dhol drums, festival firecrackers, or devotional music might not map cleanly to existing classes. Would there be a sample set of Hindi videos I could test against early on?

For the decision engine, I was thinking the fusion logic should probably weigh things differently depending on the type of sound, like an explosion is worth captioning even without a visible reaction, but background music probably shouldn't get a CC unless someone on screen is clearly reacting to it. A category-aware threshold system could help avoid the over-captioning problem.

I've started putting together an initial prototype and proposal, will share updates as I go.

[Jeevanjot19]

Hi @abinash-sketch and @keerthiseelan-planetread,
I'm interested in contributing to this project and have already built a working end-to-end implementation. Here's a quick overview of my approach:
Architecture: Three independently testable modules — audio detection → visual reaction scoring → fusion engine → SRT/SLS output. Each module is swappable without touching the others.
Audio (Goal 1): Implemented two backends — a heuristic RMS detector with adaptive noise floor (zero dependencies, runs on WAV instantly) and a YAMNet backend via MediaPipe AudioClassifier with a speech/silence/music blocklist to ensure only non-speech events pass through. Timestamps use chunk-index × hop-size calculation rather than result.timestamp_ms which is unreliable in AUDIO_CLIPS mode.
Visual (Goal 2): Two backends — OpenCV frame-diff scoring with scene-cut detection (cuts are discounted 80% before fusion to avoid false positives) and a MediaPipe Pose + FaceMesh backend with independently normalised landmark-delta scoring for head turns, flinches, and body language.
Fusion (Goal 3): score = α·audio_confidence + β·reaction_score (default α=0.60, β=0.40) with hard override thresholds for high-confidence single-signal cases. All weights and thresholds are fully configurable via YAML with no hardcoded magic numbers. Caption splitting ensures no single CC exceeds 3 seconds (subtitle standard).
On the Hindi/regional content question raised above: YAMNet's AudioSet training does have gaps for Indian-specific sounds. My config includes a VAD pre-filter (WebRTC VAD) to suppress speech segments before detection, which helps, but I agree that benchmarking on actual PlanetRead Hindi content and potentially adding a PANNs fallback would be the right approach during the coding period.
Evaluation: Built an IoU-based precision/recall/F1/overcaption-rate evaluation framework and tested on a real suspense short film — heuristic backend filtered 27 candidates down to 4 accepted CCs (5.6s), YAMNet filtered 20 down to 2 with rich class labels (20.9s).
I have a working prototype as well
Happy to discuss any architectural decisions. Will be submitting a proposal on Badal plateform as well.

[vaibhav45sktech]

Hey @abinash-sketch @keerthiseelan-planetread

I've spent time going through this issue carefully and I want to share my thinking before writing any code.

The core problem as I understand it: your editors are currently watching video manually and deciding when a non-speech sound is important enough to caption. That judgment call "does this honk / gunshot / laugh actually affect what's happening on screen" is what makes this hard to automate. A naive approach would just dump every detected sound into the SRT file and make the editor's job worse, not better. So the entire design needs to be built around that one constraint: only suggest a CC when the audio event visibly changes something on screen.

Here's how I'm planning to build it.

---

Architecture & Data Flow

The full stack is open-source throughout YAMNet via TensorFlow Hub, MediaPipe, OpenCV, and ffmpeg.

Goals 1, 2, and 3 aren't independent modules I'm designing them around a fixed data contract so the outputs chain directly without reshaping:

• Goal 1 produces: { label, confidence, start_time, end_time }
• Goal 2 attaches: reaction_score to each event
• Goal 3 reads both and makes one CC decision

This means swapping YAMNet for PANNs later, or MediaPipe for something else, won't touch the decision engine at all.

---

Goal 1: Sound Event Detection

Audio Extraction: ffmpeg pulls the audio from any video container (mp4, mkv, avi, mov) and converts it to mono 16kHz WAV the format YAMNet requires.

Processing: YAMNet returns a (frames, 521) score matrix with ~0.48s hops. I take the argmax per window to get the top predicted class and confidence.

Speech Filtering: YAMNet class indices 0–6 cover Speech, Male speech, Female speech, Child speech, Conversation, Narration, Whispering. I drop all of these. For Hindi/regional content this filter is critical the dialogue track is dense enough that without it, almost every window comes back as speech.

Thresholding: Confidence threshold default 0.3 (CLI arg). Below this, dropped.

Event Merging: Consecutive windows with the same label get merged into one event with peak confidence taken.

Merge check: same_label && new_window.start_time <= previous_event.end_time → extend.

---

Goal 2: Speaker Reaction Detection (Midpoint Milestone)

This is the part that separates meaningful CCs from noise. For every event from Goal 1, I seek to the midpoint frame using cap.set(CAP_PROP_POS_FRAMES, frame_number) and run three detection passes:

Flinch / Startle (Pose model): MediaPipe Pose gives 33 body landmarks in normalized coordinates. I check the Y-difference between LEFT_SHOULDER (11) and RIGHT_SHOULDER (12). Asymmetry above 0.05 normalized units = likely startle. Scaled linearly to 0–1, capped at 1.0 above 0.15.

Head Turn (Pose model, same pass): Using LEFT_EAR (7), RIGHT_EAR (8), and NOSE (0) X-coordinates. Forward-facing = nose centered at ~0.5 ratio between ears. A deviation above 0.15 from center signals a head turn toward the sound source. This contributes to the pose sub-score.

Surprise / Gasp (FaceMesh model): 468 face landmarks. Landmarks 13 (upper inner lip) and 14 (lower inner lip) measure mouth openness. Closed = 0.01–0.02 normalized. Gasp = above 0.05. Scaled 0.02→0.08 into 0–1.

Paused Speech (from Goal 1 data): The ticket lists paused speech as a reaction signal. A single frame can't tell you this, but Goal 1 already can if the window just before the event timestamp was speech (indices 0–6) and the event window is not, that silence transition is a real reaction signal. I flag this as speech_paused: true and carry it into Goal 3.

Final reaction score: (pose_sub_score + face_sub_score) / 2

If MediaPipe finds no person (wide shots, scene-only reactions), reaction_score defaults to 0.0 and the event still reaches Goal 3 where audio confidence alone decides. This is the correct fallback for scene reactions the ticket mentions.

I'm extracting one frame per event (midpoint) for now. Hindi drama scenes sometimes have a slight delay between the sound and the visible reaction if you'd like 2–3 frames sampled around the timestamp instead, I can build that in.

---

Goal 3: CC Decision Engine + SRT/SLS Output

Combined score:
combined = (confidence × 0.6) + (reaction_score × 0.4)
If speech_paused is true: +0.2 bonus before thresholding.

The 0.6/0.4 weighted split means a sound with strong audio confidence but no visible reaction still scores 0.6 just above the default threshold of 0.4. This is intentional. But ambient sounds like distant traffic or background music typically fail both low audio confidence AND no visible reaction so they get filtered out without a manual blocklist. That's what prevents over-captioning.

Label lookup table:

• "Gunshot, gunfire" → [gunshot]
• "Glass" → [glass breaking]
• "Crowd" → [crowd cheering]
• "Honking" → [honking]

Anything outside the table: [{first word of class name}]

Output: Standard HH:MM:SS,mmm --> HH:MM:SS,mmm SRT blocks. Both .srt and .sls supported.

CLI:

python main.py video.mp4 --format srt --threshold 0.4 --audio-weight 0.6

Default threshold is set at 0.35 (slightly higher than raw YAMNet default) because Hindi/regional content tends to have heavy background scores running under dialogue a lower threshold triggers constant [music] captions which is exactly the over-captioning problem this tool is meant to solve.

---

Three things I need from you before I open the PR

1. Test clip
The ticket mentions collecting editor feedback on suggestion accuracy as part of Goal 3 validation. I'd like to run the full pipeline on a real clip from your content and share the SRT output with your editors for that feedback loop. Even 2–3 minutes with at least one clear non-speech event is enough. Tuning thresholds on a random YouTube video won't reflect what your editors actually see.

2. Camera framing
Is the camera in your content typically close-up (faces clearly visible) or wide-shot (full body)? This changes whether FaceMesh or Pose carries more weight as the primary reaction signal and I want to weight them correctly from the start.

3. SLS format
Is there an existing SLS schema your editors are already working with, or should I model it on the SRT structure?

I'll open the PR once you confirm the approach works for you.

[ravencore06]

Hi, I’d like to contribute.

Initial plan (MVP):

• Extract audio from video
• Speech-to-text (Whisper)
• Generate basic subtitle output (.srt)

Will start with a simple pipeline and iterate with caption refinement.
Open to feedback on scope.

[Naitik120gupta]

Hi @keerthiseelan-planetread

I'm Naitik, a 3rd year Computer Science student interested in contributing to this project for DMP 2026.

I've gone through the issue carefully and the three-goal structure makes a lot of sense — especially the decision to keep sound event detection and visual reaction analysis as separate modules before combining them in the CC Decision Engine. That separation keeps the pipeline modular and testable.

A quick question before I start building: for the Sound Event Detection module (Goal 1), are you leaning towards a pretrained AudioSet-based model like panns_inference for classifying events such as applause, laughter, and alarms — or would you prefer a lighter approach using librosa features with a custom classifier trained on Indian language video content?

I have prior experience with both librosa for audio processing and OpenCV for visual analysis, so I'm confident I can contribute meaningfully across all three goals. I'd love to coordinate further on Discord and get started on a small PoC for the audio module.

Thanks for putting this together — the impact on accessibility for Hindi and regional language content is what drew me to this project.

Linkedin: https://linkedin.com/in/naitik-gupta

[Rishabhmishra3129]

Hey @abinash-sketch and @keerthiseelan-planetread,

I’m Rishabh Mishra, a 3rd-year Computer Science student, and I’m really interested in contributing to this project.

I went through the project description, and I genuinely liked the idea of focusing on contextual necessity instead of over-captioning every background sound. It’s a very practical problem—especially since most automated captions today feel cluttered and miss what actually matters for understanding the scene, particularly in Hindi and regional-language content.

I have a background in Python and Machine Learning, and I’d like to start by contributing to Goal 1 (Sound Event Detection Module). My plan is to build a clean pipeline for:

• extracting audio from video
• running it through a model like YAMNet or PANNs
• generating structured outputs with timestamps and confidence scores

Once that is stable, I’d be interested in gradually exploring how these outputs can be filtered or refined to better support the downstream decision logic.

I also find the multimodal aspect (combining audio with visual reactions) very interesting, and I’d love to contribute toward that after getting comfortable with the initial pipeline.

I’ll keep my contributions modular and well-documented so they integrate smoothly with the rest of the system.

Please let me know if there’s a good issue or starting point I should pick—I’d love to get started.

Thanks and looking forward to contributing!

Best regards,
Rishabh Mishra