diff --git a/.claude/skills/repro/SKILL.md b/.claude/skills/repro/SKILL.md index f1b3c91..6ba47a8 100644 --- a/.claude/skills/repro/SKILL.md +++ b/.claude/skills/repro/SKILL.md @@ -69,10 +69,12 @@ Create a minimal, self-contained reproduction that: **Guidelines for reproductions:** - Keep it minimal - only include what's needed to show the bug -- Install the Sentry SDK and version mentioned in the issue description. - - Fall back to the latest version if no specific version is provided. - - Fall back to the best fitting Sentry SDK you can detect for the reproduction. -- Keep the DSN for Sentry empty and remind users to `export SENTRY_DSN=` in the instructions +- Install the Sentry SDK and version mentioned in the issue description. + - Fall back to the latest version if no specific version is provided. + - Fall back to the best fitting Sentry SDK you can detect for the reproduction. +- Configure the reproduction to read the DSN from the `SENTRY_DSN` environment variable +- **ALWAYS initialize Sentry with `debug: true`** (or language equivalent) so that event/transaction IDs are logged to console for easier retrieval +- In the README, instruct users to `export SENTRY_DSN=` before running - Use standard tooling where possible (npm, uv, bundle) - If the issue description mentions a specific framework (django, rails, express, opentelemetry), set up that framework minimally - Include any necessary configuration files @@ -86,6 +88,76 @@ Before committing, verify the reproduction actually works: 3. Confirm the bug manifests as described 4. If you get stuck, continue with Step 7 but mention it clearly in the readme and still create the PR +#### Telemetry Verification (when relevant) + +For issues involving runtime behavior, errors, exceptions, or telemetry data (not build/config issues), verify with actual Sentry telemetry: + +1. **Determine if verification is needed**: Analyze the issue to see if it involves: + - Runtime errors or exceptions + - Event capture behavior + - Telemetry data (breadcrumbs, contexts, tags) + - Transactions or spans + + Skip telemetry verification for: + - Build/compilation errors + - Configuration/setup issues + - Documentation issues + +2. **Check prerequisites for telemetry verification**: + - Verify the `SENTRY_DSN` environment variable is set: + ```bash + echo $SENTRY_DSN + ``` + - Verify Sentry MCP tools are available by attempting to use them + - Check if Sentry MCP requires authentication + + **If any prerequisites are missing**: + - Report the issue to the user using `AskUserQuestion` + - Ask if they want to verify with telemetry + - If yes, inform them what needs to be set up: + - Missing SENTRY_DSN: "Set your Sentry DSN with `export SENTRY_DSN=`" + - Sentry MCP not installed: "Install the Sentry MCP server (provide setup instructions)" + - Sentry MCP needs auth: "Authenticate the Sentry MCP server with your credentials" + - If no, skip telemetry verification and document in the README that it was skipped + +3. **Run with telemetry**: Execute the reproduction, which will send events to Sentry. + +4. **Retrieve telemetry using Sentry MCP**: + - Check the console output for logged event/transaction IDs (available because `debug: true`) + - Use available MCP tools to fetch the telemetry: + - `get_issue_details` - Retrieve Sentry issues (errors) created by the reproduction + - `get_event_attachment` - Get event attachments if relevant + - `get_trace_details` - Retrieve trace data for transactions/spans + - `get_profile` - Get profiling data if applicable + + The debug output will show IDs like "Event sent: " or "Transaction: " for easier lookup. + +5. **Verify against GitHub issue**: Compare the Sentry telemetry with the reported issue: + - Does the error message match what's described in the GitHub issue? + - Does the exception type match? + - Are the expected stack frames present? + - Do breadcrumbs/contexts show the expected behavior? + - For transactions/spans, do they match the expected structure and behavior? + +6. **Document results**: In the README.md, add a "Verification" section: + - If telemetry matches: Note that the reproduction was verified with actual Sentry telemetry + - If telemetry doesn't match: Document the discrepancy clearly - what was expected vs what was captured + - If SENTRY_DSN not set: Note that telemetry verification was not performed + + Example: + ```markdown + ## Verification + ✅ Verified with Sentry telemetry - captured error matches reported issue + + Or: + + ⚠️ Telemetry discrepancy: Expected `TypeError` but captured `ReferenceError`. See Sentry issue [link]. + + Or: + + ℹ️ Telemetry verification skipped - SENTRY_DSN not set. Set your DSN and run to verify. + ``` + ### Step 7: Write the README.md @@ -101,12 +173,17 @@ The README.md in the reproduction directory should include: ## Steps to Reproduce -1. Install dependencies: +1. Set your Sentry DSN: + ```bash + export SENTRY_DSN= + ``` + +2. Install dependencies: ```bash npm install ``` -2. Run the reproduction: +3. Run the reproduction: ```bash [command to run] ``` @@ -117,6 +194,9 @@ The README.md in the reproduction directory should include: ## Actual Behavior [What actually happens - the bug] +## Verification +[Include telemetry verification results if performed - see Step 6] + ## Environment - Node.js: [version if relevant] - [Package name]: [version] @@ -149,8 +229,8 @@ The README.md in the reproduction directory should include: ### Step 9: Backlink to Original Issue -IMPORTANT: Only do this, if you successfully reproduced the bug described in the issue! -Before posting the github response on the issue, ask for permission (`AskUserQuestion` tool). +IMPORTANT: Only do this if you successfully reproduced the bug described in the issue! +Before posting the GitHub response on the issue, ask for permission (`AskUserQuestion` tool). Comment on the original GitHub issue with a link to the PR: @@ -159,6 +239,9 @@ gh issue comment --repo / --body "I've created a rep You can run it by cloning the repo and following the instructions in the README. +[If telemetry was verified, add:] +✅ Verified with actual Sentry telemetry - the reproduction successfully captures the reported issue." + 🤖 Generated with [Claude Code](https://claude.ai/code)" ```