From e62e8a782df23ac53c81f8b8717b672b8efaf5e8 Mon Sep 17 00:00:00 2001 From: lcian <17258265+lcian@users.noreply.github.com> Date: Wed, 28 Jan 2026 11:57:02 +0100 Subject: [PATCH 1/3] feat: Verify repro by directly checking Sentry telemetry --- .claude/skills/repro/SKILL.md | 90 +++++++++++++++++++++++++++++++---- 1 file changed, 81 insertions(+), 9 deletions(-) diff --git a/.claude/skills/repro/SKILL.md b/.claude/skills/repro/SKILL.md index 904f926..af21e66 100644 --- a/.claude/skills/repro/SKILL.md +++ b/.claude/skills/repro/SKILL.md @@ -69,10 +69,12 @@ Create a minimal, self-contained reproduction that: **Guidelines for reproductions:** - Keep it minimal - only include what's needed to show the bug -- Install the Sentry SDK and version mentioned in the issue description. - - Fall back to the latest version if no specific version is provided. - - Fall back to the best fitting Sentry SDK you can detect for the reproduction. -- Keep the DSN for Sentry empty and remind users to `export SENTRY_DSN=` in the instructions +- Install the Sentry SDK and version mentioned in the issue description. + - Fall back to the latest version if no specific version is provided. + - Fall back to the best fitting Sentry SDK you can detect for the reproduction. +- Configure the reproduction to read the DSN from the `SENTRY_DSN` environment variable +- **ALWAYS initialize Sentry with `debug: true`** (or language equivalent) so that event/transaction IDs are logged to console for easier retrieval +- In the README, instruct users to `export SENTRY_DSN=` before running - Use standard tooling where possible (npm, uv, bundle) - If the issue description mentions a specific framework (django, rails, express, opentelemetry), set up that framework minimally - Include any necessary configuration files @@ -86,6 +88,65 @@ Before committing, verify the reproduction actually works: 3. Confirm the bug manifests as described 4. If you get stuck, continue with Step 7 but mention it clearly in the readme and still create the PR +#### Telemetry Verification (when relevant) + +For issues involving runtime behavior, errors, exceptions, or telemetry data (not build/config issues), verify with actual Sentry telemetry: + +1. **Determine if verification is needed**: Analyze the issue to see if it involves: + - Runtime errors or exceptions + - Event capture behavior + - Telemetry data (breadcrumbs, contexts, tags) + - Transactions or spans + + Skip telemetry verification for: + - Build/compilation errors + - Configuration/setup issues + - Documentation issues + +2. **Check for SENTRY_DSN**: Verify the `SENTRY_DSN` environment variable is set: + ```bash + echo $SENTRY_DSN + ``` + If not set, document in the README that telemetry verification was skipped and the user should set their DSN to test. + +3. **Run with telemetry**: Execute the reproduction, which will send events to Sentry. + +4. **Retrieve telemetry using Sentry MCP**: + - Check the console output for logged event/transaction IDs (available because `debug: true`) + - Use available MCP tools to fetch the telemetry: + - `get_issue_details` - Retrieve Sentry issues (errors) created by the reproduction + - `get_event_attachment` - Get event attachments if relevant + - `get_trace_details` - Retrieve trace data for transactions/spans + - `get_profile` - Get profiling data if applicable + + The debug output will show IDs like "Event sent: " or "Transaction: " for easier lookup. + +5. **Verify against GitHub issue**: Compare the Sentry telemetry with the reported issue: + - Does the error message match what's described in the GitHub issue? + - Does the exception type match? + - Are the expected stack frames present? + - Do breadcrumbs/contexts show the expected behavior? + - For transactions/spans, do they match the expected structure and behavior? + +6. **Document results**: In the README.md, add a "Verification" section: + - If telemetry matches: Note that the reproduction was verified with actual Sentry telemetry + - If telemetry doesn't match: Document the discrepancy clearly - what was expected vs what was captured + - If SENTRY_DSN not set: Note that telemetry verification was not performed + + Example: + ```markdown + ## Verification + ✅ Verified with Sentry telemetry - captured error matches reported issue + + Or: + + ⚠️ Telemetry discrepancy: Expected `TypeError` but captured `ReferenceError`. See Sentry issue [link]. + + Or: + + ℹ️ Telemetry verification skipped - SENTRY_DSN not set. Set your DSN and run to verify. + ``` + ### Step 7: Write the README.md @@ -101,12 +162,17 @@ The README.md in the reproduction directory should include: ## Steps to Reproduce -1. Install dependencies: +1. Set your Sentry DSN: + ```bash + export SENTRY_DSN= + ``` + +2. Install dependencies: ```bash npm install ``` -2. Run the reproduction: +3. Run the reproduction: ```bash [command to run] ``` @@ -117,6 +183,9 @@ The README.md in the reproduction directory should include: ## Actual Behavior [What actually happens - the bug] +## Verification +[Include telemetry verification results if performed - see Step 6] + ## Environment - Node.js: [version if relevant] - [Package name]: [version] @@ -149,15 +218,18 @@ The README.md in the reproduction directory should include: ### Step 9: Backlink to Original Issue -IMPORTANT: Only do this, if you successfully reproduced the bug described in the issue! -Before posting the github response on the issue, ask for permission (`AskUserQuestion` tool). +IMPORTANT: Only do this if you successfully reproduced the bug described in the issue! +Before posting the GitHub response on the issue, ask for permission (`AskUserQuestion` tool). Comment on the original GitHub issue with a link to the PR: ```bash gh issue comment --repo / --body "I've created a reproduction for this issue: [PR-URL] -You can run it by cloning the repo and following the instructions in the README." +You can run it by cloning the repo and following the instructions in the README. + +[If telemetry was verified, add:] +✅ Verified with actual Sentry telemetry - the reproduction successfully captures the reported issue." ``` ## Error Handling From fca864808d6c40996dd10d69b077a8bb0eafff42 Mon Sep 17 00:00:00 2001 From: lcian <17258265+lcian@users.noreply.github.com> Date: Wed, 28 Jan 2026 12:03:13 +0100 Subject: [PATCH 2/3] check for missing deps --- .claude/skills/repro/SKILL.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/.claude/skills/repro/SKILL.md b/.claude/skills/repro/SKILL.md index af21e66..ff38481 100644 --- a/.claude/skills/repro/SKILL.md +++ b/.claude/skills/repro/SKILL.md @@ -103,11 +103,22 @@ For issues involving runtime behavior, errors, exceptions, or telemetry data (no - Configuration/setup issues - Documentation issues -2. **Check for SENTRY_DSN**: Verify the `SENTRY_DSN` environment variable is set: - ```bash - echo $SENTRY_DSN - ``` - If not set, document in the README that telemetry verification was skipped and the user should set their DSN to test. +2. **Check prerequisites for telemetry verification**: + - Verify the `SENTRY_DSN` environment variable is set: + ```bash + echo $SENTRY_DSN + ``` + - Verify Sentry MCP tools are available by attempting to use them + - Check if Sentry MCP requires authentication + + **If any prerequisites are missing**: + - Report the issue to the user using `AskUserQuestion` + - Ask if they want to verify with telemetry + - If yes, inform them what needs to be set up: + - Missing SENTRY_DSN: "Set your Sentry DSN with `export SENTRY_DSN=`" + - Sentry MCP not installed: "Install the Sentry MCP server (provide setup instructions)" + - Sentry MCP needs auth: "Authenticate the Sentry MCP server with your credentials" + - If no, skip telemetry verification and document in the README that it was skipped 3. **Run with telemetry**: Execute the reproduction, which will send events to Sentry. From 42438d43e3d175ce24a1028d9f96679df45575ac Mon Sep 17 00:00:00 2001 From: lcian <17258265+lcian@users.noreply.github.com> Date: Wed, 28 Jan 2026 12:10:53 +0100 Subject: [PATCH 3/3] improve --- .claude/skills/repro/SKILL.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/.claude/skills/repro/SKILL.md b/.claude/skills/repro/SKILL.md index 31cc0e3..6ba47a8 100644 --- a/.claude/skills/repro/SKILL.md +++ b/.claude/skills/repro/SKILL.md @@ -242,8 +242,6 @@ You can run it by cloning the repo and following the instructions in the README. [If telemetry was verified, add:] ✅ Verified with actual Sentry telemetry - the reproduction successfully captures the reported issue." -You can run it by cloning the repo and following the instructions in the README. - 🤖 Generated with [Claude Code](https://claude.ai/code)" ```