Skip to content

VER-277: Update prompts in Stage 3 to utilize Google Search result better #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
quancao-ea merged 3 commits into from
Nov 3, 2025
Merged

VER-277: Update prompts in Stage 3 to utilize Google Search result better #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e2089a2
Update user prompt and system instruction in Stage 3 for utilizing we…
quancao-ea Oct 29, 2025
b780a4b
Update prompts/Stage_3_analysis_prompt.md
quancao-ea Oct 29, 2025
bf0af1f
Include the current datetime in Stage 3 user prompt
quancao-ea Oct 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 51 additions & 24 deletions prompts/Stage_3_analysis_prompt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,25 @@
# **Task Overview**

You are provided with an audio clip containing a potential disinformation snippet that has been flagged by Stage 1 of an audio processing pipeline.
You are provided with the following information:
1. An audio clip that requires independent verification and analysis. It has been flagged by Stage 1 of an audio processing pipeline for review, but you must objectively determine whether it contains accurate or false information. Do not assume the content is disinformation - verify all claims thoroughly before making any determination.

The audio clip contains 3 parts:
- The part before the snippet
- The detected snippet
- The part after the snippet

You are also provided with the metadata of the audio clip, which contains:
2. The metadata of the audio clip, which contains:

- `duration`: the duration of the entire audio clip, in MM:SS format
- `start_time`: the start time of the snippet within the audio clip, in MM:SS format
- `end_time`: the end time of the snippet within the audio clip, in MM:SS format
- `transcription`: the transcription of the disinformation snippet within the audio clip
- `transcription`: the transcription of the flagged snippet within the audio clip
- Note that this is NOT the transcription of the entire audio clip
- If this transcription is inaccurate, or not found within the audio clip, disregard it and use your own transcription for analysis.
- `disinformation_categories`: the disinformation categories assigned to the snippet by Stage 1 of the pipeline
- `additional_info`: other relevant metadata, such as recording date, time, and location

3. The current date and time, which may be relevant for verifying time-sensitive claims made in the audio clip.

Your tasks are:

Expand Down Expand Up @@ -85,7 +90,7 @@ Perform the following steps:
- Highlight the main points discussed.

- **Explanation:**
- Provide a detailed explanation of why the snippet constitutes disinformation.
- Provide a detailed explanation of your analysis findings. If disinformation is detected, explain why the content is false or misleading.
- Reference specific elements from the audio, including vocal cues and linguistic features.
- Use the detailed heuristics and examples to support your analysis.
- Consider cultural contexts and how they may influence interpretation.
Expand Down Expand Up @@ -121,39 +126,53 @@ Perform the following steps:
- Ensure that the `main` part of the `context` matches with the transcription in the provided metadata.
- In case the provided transcription in the metadata is inaccurate (eg, it's not found within the audio clip), utilize your own transcription to identify and extract the snippet and its surrounding context.

##### **H. Confidence Scoring and Self-Review Process**
##### **H. Confidence Scoring and Verification Requirements**

The confidence score represents your degree of certainty that the content contains demonstrably false or misleading claims. This is NOT:
- A measure of confidence in your analysis
- A measure of how controversial or partisan the content is
- A measure of whether you agree with the positions expressed

**Initial Scoring Framework:**
**Verification Requirement:**

Before assigning confidence scores, verify all factual claims using Grounding with Google Search. The verification outcome determines the maximum possible score.

**Scoring Framework Based on Verification:**

High Confidence Scores (80-100) require:
1. **High Confidence Scores (80-100) require:**
- Strong contradictory evidence from reputable sources (news articles, official statements, fact-checkers stating claim is false)
- Specific factual claims that can be definitively proven false
- Direct contradictions of well-documented facts
- Demonstrably false statements or deliberate misrepresentation
Example: "The COVID vaccine contains microchips for mind control"
Example: "The COVID vaccine contains microchips for mind control" OR search finds recent news showing person alive when death claimed

Medium Confidence Scores (40-79) require:
2. **Medium Confidence Scores (40-79) require:**
- Some contradictory evidence found, but not definitive
- Misleading claims that omit crucial context
- Deceptive presentation of real facts
- Misrepresentation of causation vs correlation
**IMPORTANT**: If NO contradictory evidence is found, maximum score is 40, not higher
Example: "Immigrants are causing crime rates to spike" (when data shows no correlation)

Low Confidence Scores (1-39) apply to:
3. **Low Confidence Scores (1-39) apply when:**
- No contradictory evidence found via web search (no search results or only old/unrelated information)
- Unsubstantiated claims without clear evidence
- Exaggerated interpretations of real events
- Misleading but not entirely false statements
- Limited coverage (common for recent events or local news)
Example: "The government is hiding the truth about inflation"

Zero Confidence Score (0) applies when:
4. **Zero Confidence Score (0) applies when:**
- Web search confirms claims are true (verified by reputable sources like CNN, BBC, Reuters, AP)
- Content makes no demonstrably false claims
- Content expresses opinions without misrepresenting facts
- Content may be partisan or controversial but is factually accurate
Example: "We need stricter immigration policies"

**Key Principle: Absence of evidence is not evidence of absence.**

No search results does NOT mean the claim is false. Unusual or surprising claims can be true. Recent events may have limited coverage. When you cannot find contradictory information, score conservatively (0-40), not as disinformation (80-100).

##### **I. Required Self-Review Process**

After completing your initial analysis, perform this structured review:
Expand All @@ -172,10 +191,13 @@ After completing your initial analysis, perform this structured review:
- [ ] Am I scoring falsity rather than controversy?
- [ ] Would these scores be defensible to fact-checkers?
- [ ] Are my explanations consistent with my scores?
- [ ] For uncertain claims: Did I score them 0-40 maximum?

3. **Score Adjustment Protocol**
If any validation fails:
- Reduce score to 0 if you cannot cite specific false claims
- Reduce to 0 if you cannot cite specific false claims
- Reduce to 0 if web search confirms claims are true
- Reduce to 0-40 if you found no reliable information (uncertainty)
- Adjust scores to match available evidence
- Document reasoning for any score changes
- Ensure final scores reflect only demonstrably false content
Expand All @@ -186,6 +208,7 @@ After completing your initial analysis, perform this structured review:
- Confusing controversial content with false content
- Treating bias as equivalent to disinformation
- Scoring based on disagreement rather than falsity
- Treating "no search results" as evidence of falsity (it is uncertainty)

##### **J. Emotional Tone Analysis**
The emotional tone analysis identifies and measures emotions expressed in the content. Like our confidence scoring, this requires evidence-based assessment:
Expand Down Expand Up @@ -422,7 +445,7 @@ Ensure your output strictly adheres to this schema.
"description": "Explanation in English."
}
},
"description": "Detailed explanation of why the snippet constitutes disinformation."
"description": "Detailed explanation of the analysis findings, including why content is scored as disinformation or verified as accurate."
},
"disinformation_categories": {
"type": "array",
Expand Down Expand Up @@ -537,14 +560,16 @@ Ensure your output strictly adheres to this schema.
"evidence_provided",
"scoring_falsity",
"defensible_to_factcheckers",
"consistent_explanations"
"consistent_explanations",
"uncertain_claims_scored_low"
],
"properties": {
"specific_claims_quoted": { "type": "boolean" },
"evidence_provided": { "type": "boolean" },
"scoring_falsity": { "type": "boolean" },
"defensible_to_factcheckers": { "type": "boolean" },
"consistent_explanations": { "type": "boolean" }
"consistent_explanations": { "type": "boolean" },
"uncertain_claims_scored_low": { "type": "boolean" }
}
},
"score_adjustments": {
Expand Down Expand Up @@ -1514,7 +1539,8 @@ Below is a complete example showing all required fields:
"evidence_provided": true,
"scoring_falsity": true,
"defensible_to_factcheckers": true,
"consistent_explanations": true
"consistent_explanations": true,
"uncertain_claims_scored_low": true
},
"score_adjustments": {
"initial_score": 95,
Expand Down Expand Up @@ -1689,14 +1715,15 @@ By following these instructions and listening closely using the detailed heurist
Please proceed to analyze the provided audio content following these guidelines:

1. Listen carefully to capture all spoken content.
2. Apply the detailed heuristics for disinformation analysis.
3. Base political orientation assessment solely on observable content elements.
4. Document all findings with specific evidence from the content.
5. Structure your output according to the provided JSON schema.
6. Confidence scores must be based on demonstrably false claims, not controversy.
7. Each non-zero score requires specific evidence.
8. Complete the self-review process for every analysis.
9. When in doubt, score conservatively (lower).
2. Verify all factual claims using Google Search before assigning scores (Section H), searching for information relevant to the recording datetime and/or current datetime.
3. Apply the detailed heuristics for disinformation analysis.
4. Base political orientation assessment solely on observable content elements.
5. Document all findings with specific evidence from the content.
6. Structure your output according to the provided JSON schema.
7. Confidence scores must be based on contradictory evidence and demonstrably false claims, not absence of confirmation or controversy.
8. Each non-zero score requires specific evidence.
9. Complete the self-review process for every analysis.
10. When in doubt, score conservatively (0-40 for uncertain claims).

# Important Note

Expand Down
8 changes: 5 additions & 3 deletions prompts/Stage_3_output_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
"description": "Explanation in English."
}
},
"description": "Detailed explanation of why the snippet constitutes disinformation."
"description": "Detailed explanation of the analysis findings, including why content is scored as disinformation or verified as accurate."
},
"disinformation_categories": {
"type": "array",
Expand Down Expand Up @@ -185,14 +185,16 @@
"evidence_provided",
"scoring_falsity",
"defensible_to_factcheckers",
"consistent_explanations"
"consistent_explanations",
"uncertain_claims_scored_low"
],
"properties": {
"specific_claims_quoted": { "type": "boolean" },
"evidence_provided": { "type": "boolean" },
"scoring_falsity": { "type": "boolean" },
"defensible_to_factcheckers": { "type": "boolean" },
"consistent_explanations": { "type": "boolean" }
"consistent_explanations": { "type": "boolean" },
"uncertain_claims_scored_low": { "type": "boolean" }
}
},
"score_adjustments": {
Expand Down
3 changes: 2 additions & 1 deletion prompts/Stage_3_system_instruction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ You are an advanced language model specialized in in-depth disinformation and po
**General Guidelines:**
- **Accuracy:** Ensure precise transcription and translation, preserving the original meaning and cultural nuances.
- **Cultural Sensitivity:** Be mindful of cultural contexts, idiomatic expressions, and dialects specific to Spanish and Arabic-speaking immigrant communities especially.
- **Evidence-Based Analysis:** Base all analytical conclusions, including disinformation detection and political orientation assessments, strictly on demonstrably false claims and explicitly stated positions. Each non-zero confidence score must be supported by specific false claims that can be definitively proven incorrect.
- **Evidence-Based Analysis:** Verify all factual claims using Grounding with Google Search before scoring. High scores (80-100) require strong contradictory evidence from reputable sources, not absence of confirmation. Base all conclusions on demonstrably false claims that can be definitively proven incorrect with cited sources.
- **Searching Guidelines:** When verifying factual claims, prioritize information that matches or is recent to the recording date of the provided audio file. Do not rely on old or unrelated information that predates the recording. If the recording references recent events or claims about current status, search for the most up-to-date information available.
- **Objectivity:** Maintain strict neutrality by focusing solely on verifiable facts rather than assumptions or inferences. Distinguish between controversial content and demonstrably false claims.
- **Self-Review:** Systematically validate all assessments through structured self-review, adjusting scores when specific evidence cannot be cited.
- **Structured Output:** All output must strictly conform to the provided JSON schema.
6 changes: 4 additions & 2 deletions src/processing_pipeline/stage_3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from datetime import datetime
from datetime import datetime, timezone
import os
import time
from google import genai
Expand Down Expand Up @@ -275,7 +275,9 @@ def run(

# Prepare the user prompt
user_prompt = (
f"{cls.USER_PROMPT}\n\nHere is the metadata of the attached audio clip:\n{json.dumps(metadata, indent=2)}"
f"{cls.USER_PROMPT}\n\n"
f"Here is the metadata of the attached audio clip:\n{json.dumps(metadata, indent=2)}\n\n"
f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %-d, %Y %-I:%M %p UTC')}\n\n"
Comment on lines +278 to +280
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Platform compatibility issue: Unix-specific datetime format codes.

The format codes %-d and %-I use the dash modifier to suppress leading zeros, which only works on Unix/Linux/macOS platforms. On Windows, these will raise a ValueError, potentially breaking the pipeline.

Apply this diff to use platform-independent formatting:

-            f"{cls.USER_PROMPT}\n\n"
-            f"Here is the metadata of the attached audio clip:\n{json.dumps(metadata, indent=2)}\n\n"
-            f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %-d, %Y %-I:%M %p UTC')}\n\n"
+            f"{cls.USER_PROMPT}\n\n"
+            f"Here is the metadata of the attached audio clip:\n{json.dumps(metadata, indent=2)}\n\n"
+            f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %d, %Y %I:%M %p UTC').replace(' 0', ' ')}\n\n"

Alternatively, use zero-padded formats if leading zeros are acceptable:

-            f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %-d, %Y %-I:%M %p UTC')}\n\n"
+            f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %d, %Y %I:%M %p UTC')}\n\n"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
f"{cls.USER_PROMPT}\n\n"
f"Here is the metadata of the attached audio clip:\n{json.dumps(metadata, indent=2)}\n\n"
f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %-d, %Y %-I:%M %p UTC')}\n\n"
f"{cls.USER_PROMPT}\n\n"
f"Here is the metadata of the attached audio clip:\n{json.dumps(metadata, indent=2)}\n\n"
f"Here is the current date and time: {datetime.now(timezone.utc).strftime('%B %d, %Y %I:%M %p UTC').replace(' 0', ' ')}\n\n"
🤖 Prompt for AI Agents 
In src/processing_pipeline/stage_3.py around lines 278 to 280, the f-string uses
Unix-only strftime codes %-d and %-I which raise ValueError on Windows; replace
those with platform-independent formatting by using zero-padded %d and %I (or
build the date/time string from datetime attributes) and then remove leading
zeros in a platform-safe way (e.g., format with %d/%I and strip the leading '0'
or construct the day/hour using dt.day and 12-hour conversion) so the output
remains the same but works on Windows and Unix.

)

try:
Expand Down
7 changes: 4 additions & 3 deletions src/processing_pipeline/stage_3_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ class Summary(BaseModel):


class Explanation(BaseModel):
spanish: str = Field(description="Explanation of the snippet in Spanish")
english: str = Field(description="Explanation of the snippet in English")
spanish: str = Field(description="Explanation of the analysis findings in Spanish")
english: str = Field(description="Explanation of the analysis findings in English")


class DisinformationCategory(BaseModel):
Expand Down Expand Up @@ -49,6 +49,7 @@ class ValidationChecklist(BaseModel):
scoring_falsity: bool
defensible_to_factcheckers: bool
consistent_explanations: bool
uncertain_claims_scored_low: bool


class ScoreAdjustments(BaseModel):
Expand Down Expand Up @@ -137,7 +138,7 @@ class Stage3Output(BaseModel):
translation: str = Field(description="Translation of the transcription into English")
title: Title = Field(description="Descriptive title of the snippet")
summary: Summary = Field(description="Objective summary of the snippet")
explanation: Explanation = Field(description="Detailed explanation of why the snippet constitutes disinformation")
explanation: Explanation = Field(description="Detailed explanation of the analysis findings, including why content is scored as disinformation or verified as accurate")
disinformation_categories: list[DisinformationCategory] = Field(
description="Disinformation categories that the snippet belongs to"
)
Expand Down