Skip to content

feat: Verify repro by directly checking Sentry telemetry #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lcian wants to merge 4 commits into from
Closed

feat: Verify repro by directly checking Sentry telemetry #10

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e62e8a7
feat: Verify repro by directly checking Sentry telemetry
lcian Jan 28, 2026
fca8648
check for missing deps
lcian Jan 28, 2026
fb8c9b0
Merge branch 'main' into lcian/feat/auto-verify
lcian Jan 28, 2026
42438d4
improve
lcian Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 91 additions & 8 deletions .claude/skills/repro/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,12 @@ Create a minimal, self-contained reproduction that:

**Guidelines for reproductions:**
- Keep it minimal - only include what's needed to show the bug
- Install the Sentry SDK and version mentioned in the issue description.
- Fall back to the latest version if no specific version is provided.
- Fall back to the best fitting Sentry SDK you can detect for the reproduction.
- Keep the DSN for Sentry empty and remind users to `export SENTRY_DSN=` in the instructions
- Install the Sentry SDK and version mentioned in the issue description.
- Fall back to the latest version if no specific version is provided.
- Fall back to the best fitting Sentry SDK you can detect for the reproduction.
- Configure the reproduction to read the DSN from the `SENTRY_DSN` environment variable
- **ALWAYS initialize Sentry with `debug: true`** (or language equivalent) so that event/transaction IDs are logged to console for easier retrieval
- In the README, instruct users to `export SENTRY_DSN=<their-dsn>` before running
- Use standard tooling where possible (npm, uv, bundle)
- If the issue description mentions a specific framework (django, rails, express, opentelemetry), set up that framework minimally
- Include any necessary configuration files
Expand All @@ -86,6 +88,76 @@ Before committing, verify the reproduction actually works:
3. Confirm the bug manifests as described
4. If you get stuck, continue with Step 7 but mention it clearly in the readme and still create the PR

#### Telemetry Verification (when relevant)

For issues involving runtime behavior, errors, exceptions, or telemetry data (not build/config issues), verify with actual Sentry telemetry:

1. **Determine if verification is needed**: Analyze the issue to see if it involves:
- Runtime errors or exceptions
- Event capture behavior
- Telemetry data (breadcrumbs, contexts, tags)
- Transactions or spans

Skip telemetry verification for:
- Build/compilation errors
- Configuration/setup issues
- Documentation issues

2. **Check prerequisites for telemetry verification**:
- Verify the `SENTRY_DSN` environment variable is set:
```bash
echo $SENTRY_DSN
```
- Verify Sentry MCP tools are available by attempting to use them
- Check if Sentry MCP requires authentication

**If any prerequisites are missing**:
- Report the issue to the user using `AskUserQuestion`
- Ask if they want to verify with telemetry
- If yes, inform them what needs to be set up:
- Missing SENTRY_DSN: "Set your Sentry DSN with `export SENTRY_DSN=<your-dsn>`"
- Sentry MCP not installed: "Install the Sentry MCP server (provide setup instructions)"
- Sentry MCP needs auth: "Authenticate the Sentry MCP server with your credentials"
- If no, skip telemetry verification and document in the README that it was skipped
Copy link

@cursor cursor bot Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing wait step after user confirms prerequisites setup

Medium Severity

When prerequisites are missing and the user says "yes" to telemetry verification, the instructions only say to inform them what needs to be set up. There's no instruction to wait for the user to complete setup, confirm readiness, or re-check prerequisites before proceeding to step 3 (Run with telemetry). This could cause the agent to proceed immediately without the prerequisites being met.

Fix in Cursor Fix in Web


3. **Run with telemetry**: Execute the reproduction, which will send events to Sentry.

4. **Retrieve telemetry using Sentry MCP**:
- Check the console output for logged event/transaction IDs (available because `debug: true`)
- Use available MCP tools to fetch the telemetry:
- `get_issue_details` - Retrieve Sentry issues (errors) created by the reproduction
- `get_event_attachment` - Get event attachments if relevant
- `get_trace_details` - Retrieve trace data for transactions/spans
- `get_profile` - Get profiling data if applicable
Copy link

@cursor cursor bot Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing propagation delay handling before fetching telemetry

Medium Severity

The telemetry verification workflow has no instruction to wait for events to propagate through Sentry's ingestion pipeline before fetching. Step 3 runs the reproduction, and step 4 immediately retrieves telemetry. Since Sentry has processing delays (events aren't instantly available via API), the agent may query before events appear, receive empty results, and incorrectly conclude the reproduction failed to capture telemetry—a false negative that defeats the purpose of verification.

Fix in Cursor Fix in Web


The debug output will show IDs like "Event sent: <event-id>" or "Transaction: <transaction-id>" for easier lookup.

5. **Verify against GitHub issue**: Compare the Sentry telemetry with the reported issue:
- Does the error message match what's described in the GitHub issue?
- Does the exception type match?
- Are the expected stack frames present?
- Do breadcrumbs/contexts show the expected behavior?
- For transactions/spans, do they match the expected structure and behavior?

6. **Document results**: In the README.md, add a "Verification" section:
- If telemetry matches: Note that the reproduction was verified with actual Sentry telemetry
- If telemetry doesn't match: Document the discrepancy clearly - what was expected vs what was captured
- If SENTRY_DSN not set: Note that telemetry verification was not performed

Example:
```markdown
## Verification
✅ Verified with Sentry telemetry - captured error matches reported issue

Or:

⚠️ Telemetry discrepancy: Expected `TypeError` but captured `ReferenceError`. See Sentry issue [link].

Or:

ℹ️ Telemetry verification skipped - SENTRY_DSN not set. Set your DSN and run to verify.
```


### Step 7: Write the README.md

Expand All @@ -101,12 +173,17 @@ The README.md in the reproduction directory should include:

## Steps to Reproduce

1. Install dependencies:
1. Set your Sentry DSN:
```bash
export SENTRY_DSN=<your-dsn>
```

2. Install dependencies:
```bash
npm install
```

2. Run the reproduction:
3. Run the reproduction:
```bash
[command to run]
```
Expand All @@ -117,6 +194,9 @@ The README.md in the reproduction directory should include:
## Actual Behavior
[What actually happens - the bug]

## Verification
[Include telemetry verification results if performed - see Step 6]

## Environment
- Node.js: [version if relevant]
- [Package name]: [version]
Expand Down Expand Up @@ -149,8 +229,8 @@ The README.md in the reproduction directory should include:

### Step 9: Backlink to Original Issue

IMPORTANT: Only do this, if you successfully reproduced the bug described in the issue!
Before posting the github response on the issue, ask for permission (`AskUserQuestion` tool).
IMPORTANT: Only do this if you successfully reproduced the bug described in the issue!
Before posting the GitHub response on the issue, ask for permission (`AskUserQuestion` tool).

Comment on the original GitHub issue with a link to the PR:

Expand All @@ -159,6 +239,9 @@ gh issue comment <issue-number> --repo <owner>/<repo> --body "I've created a rep

You can run it by cloning the repo and following the instructions in the README.

[If telemetry was verified, add:]
✅ Verified with actual Sentry telemetry - the reproduction successfully captures the reported issue."

🤖 Generated with [Claude Code](https://claude.ai/code)"
```

Comment on lines 239 to 247
Copy link

@sentry sentry bot Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The bash command template in SKILL.md has a syntax error with two closing double quotes in the --body argument, which will cause the command to fail.
Severity: MEDIUM

Suggested Fix

Remove the closing double quote from the optional telemetry verification line. The --body argument's string should only be terminated by the single closing quote at the very end of the multi-line body content.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: .claude/skills/repro/SKILL.md#L239-L247

Potential issue: The `gh issue comment` command template in `SKILL.md` contains a syntax
error. The template provides an optional line for telemetry verification which includes
a closing double quote. However, the final line of the template also has a closing
double quote. If an agent includes the optional telemetry line as instructed, the
resulting command will have two closing quotes for the `--body` argument, which is
invalid bash syntax. This will cause the command to fail, preventing the agent from
successfully posting the GitHub comment.

Did we get this right? 👍 / 👎 to inform future reviews.

Expand Down