feat: Verify repro by directly checking Sentry telemetry#10
feat: Verify repro by directly checking Sentry telemetry#10
Conversation
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
|
|
||
| You can run it by cloning the repo and following the instructions in the README. | ||
|
|
||
| [If telemetry was verified, add:] | ||
| ✅ Verified with actual Sentry telemetry - the reproduction successfully captures the reported issue." | ||
|
|
||
| 🤖 Generated with [Claude Code](https://claude.ai/code)" | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Bug: The bash command template in SKILL.md has a syntax error with two closing double quotes in the --body argument, which will cause the command to fail.
Severity: MEDIUM
Suggested Fix
Remove the closing double quote from the optional telemetry verification line. The --body argument's string should only be terminated by the single closing quote at the very end of the multi-line body content.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: .claude/skills/repro/SKILL.md#L239-L247
Potential issue: The `gh issue comment` command template in `SKILL.md` contains a syntax
error. The template provides an optional line for telemetry verification which includes
a closing double quote. However, the final line of the template also has a closing
double quote. If an agent includes the optional telemetry line as instructed, the
resulting command will have two closing quotes for the `--body` argument, which is
invalid bash syntax. This will cause the command to fail, preventing the agent from
successfully posting the GitHub comment.
Did we get this right? 👍 / 👎 to inform future reviews.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| - Missing SENTRY_DSN: "Set your Sentry DSN with `export SENTRY_DSN=<your-dsn>`" | ||
| - Sentry MCP not installed: "Install the Sentry MCP server (provide setup instructions)" | ||
| - Sentry MCP needs auth: "Authenticate the Sentry MCP server with your credentials" | ||
| - If no, skip telemetry verification and document in the README that it was skipped |
There was a problem hiding this comment.
Missing wait step after user confirms prerequisites setup
Medium Severity
When prerequisites are missing and the user says "yes" to telemetry verification, the instructions only say to inform them what needs to be set up. There's no instruction to wait for the user to complete setup, confirm readiness, or re-check prerequisites before proceeding to step 3 (Run with telemetry). This could cause the agent to proceed immediately without the prerequisites being met.
| - `get_issue_details` - Retrieve Sentry issues (errors) created by the reproduction | ||
| - `get_event_attachment` - Get event attachments if relevant | ||
| - `get_trace_details` - Retrieve trace data for transactions/spans | ||
| - `get_profile` - Get profiling data if applicable |
There was a problem hiding this comment.
Missing propagation delay handling before fetching telemetry
Medium Severity
The telemetry verification workflow has no instruction to wait for events to propagate through Sentry's ingestion pipeline before fetching. Step 3 runs the reproduction, and step 4 immediately retrieves telemetry. Since Sentry has processing delays (events aren't instantly available via API), the agent may query before events appear, receive empty results, and incorrectly conclude the reproduction failed to capture telemetry—a false negative that defeats the purpose of verification.
|
will build on using MCP in a gradual way see #11 |
2 participants
Just an idea, feel free to change this or close it if you don't like it.
Most of the time when reproducing something you need to check Sentry.
With the current setup, the agent might believe it correctly reproduced an issue when in fact it hasn't.
This makes the agent use the Sentry MCP to directly check the telemetry by itself.
We could also instruct it to iterate until the sample properly reproduces the issue, but that might consume a lot of tokens and fill up the context after a while.