Skip to content

fix: skip AI analysis on /approve cancel#1007

Merged
myakove merged 2 commits intomainfrom
fix-no-ai-on-cancel
Mar 1, 2026
Merged

fix: skip AI analysis on /approve cancel#1007
myakove merged 2 commits intomainfrom
fix-no-ai-on-cancel

Conversation

@rnetser
Copy link
Copy Markdown
Collaborator

@rnetser rnetser commented Mar 1, 2026

Summary

  • Fix test oracle (AI analysis) being triggered on /approve cancel
  • Only trigger AI analysis when approving, not when cancelling the approval

Closes #1006

Changes

  • webhook_server/libs/handlers/issue_comment_handler.py — add and not remove check
  • webhook_server/tests/test_issue_comment_handler.py — add test for cancel path

Test plan

  • All 1315 tests pass
  • 90.65% coverage
  • ruff check + format clean

Summary by CodeRabbit

  • Bug Fixes

    • Approval handling tightened so cancel/remove variants no longer trigger oracle processing or spawn unwanted background tasks, preventing unintended side effects when an approval is retracted.
  • Tests

    • Added a test verifying that canceling an approval does not invoke the oracle or create background work, ensuring correct behavior for approve-cancel flows.

The test oracle was triggered even when the approve label was being
removed via /approve cancel. Only trigger AI analysis when approving,
not when cancelling.
@myakove-bot
Copy link
Copy Markdown
Collaborator

Report bugs in Issues

Welcome! 🎉

This pull request will be automatically processed with the following features:

🔄 Automatic Actions

  • Reviewer Assignment: Reviewers are automatically assigned based on the OWNERS file in the repository root
  • Size Labeling: PR size labels (XS, S, M, L, XL, XXL) are automatically applied based on changes
  • Issue Creation: Disabled for this repository
  • Pre-commit Checks: pre-commit runs automatically if .pre-commit-config.yaml exists
  • Branch Labeling: Branch-specific labels are applied to track the target branch
  • Auto-verification: Auto-verified users have their PRs automatically marked as verified
  • Labels: All label categories are enabled (default configuration)

📋 Available Commands

PR Status Management

  • /wip - Mark PR as work in progress (adds WIP: prefix to title)
  • /wip cancel - Remove work in progress status
  • /hold - Block PR merging (approvers only)
  • /hold cancel - Unblock PR merging
  • /verified - Mark PR as verified
  • /verified cancel - Remove verification status
  • /reprocess - Trigger complete PR workflow reprocessing (useful if webhook failed or configuration changed)
  • /regenerate-welcome - Regenerate this welcome message

Review & Approval

  • /lgtm - Approve changes (looks good to me)
  • /approve - Approve PR (approvers only)
  • /automerge - Enable automatic merging when all requirements are met (maintainers and approvers only)
  • /assign-reviewers - Assign reviewers based on OWNERS file
  • /assign-reviewer @username - Assign specific reviewer
  • /check-can-merge - Check if PR meets merge requirements

Testing & Validation

  • /retest tox - Run Python test suite with tox
  • /retest build-container - Rebuild and test container image
  • /retest python-module-install - Test Python package installation
  • /retest pre-commit - Run pre-commit hooks and checks
  • /retest conventional-title - Validate commit message format
  • /retest all - Run all available tests

Container Operations

  • /build-and-push-container - Build and push container image (tagged with PR number)
    • Supports additional build arguments: /build-and-push-container --build-arg KEY=value

Cherry-pick Operations

  • /cherry-pick <branch> - Schedule cherry-pick to target branch when PR is merged
    • Multiple branches: /cherry-pick branch1 branch2 branch3

Label Management

  • /<label-name> - Add a label to the PR
  • /<label-name> cancel - Remove a label from the PR

✅ Merge Requirements

This PR will be automatically approved when the following conditions are met:

  1. Approval: /approve from at least one approver
  2. LGTM Count: Minimum 1 /lgtm from reviewers
  3. Status Checks: All required status checks must pass
  4. No Blockers: No WIP, hold, conflict labels
  5. Verified: PR must be marked as verified (if verification is enabled)

📊 Review Process

Approvers and Reviewers

Approvers:

  • myakove
  • rnetser

Reviewers:

  • myakove
  • rnetser
Available Labels
  • hold
  • verified
  • wip
  • lgtm
  • approve
  • automerge

💡 Tips

  • WIP Status: Use /wip when your PR is not ready for review
  • Verification: The verified label is automatically removed on each new commit
  • Cherry-picking: Cherry-pick labels are processed when the PR is merged
  • Container Builds: Container images are automatically tagged with the PR number
  • Permission Levels: Some commands require approver permissions
  • Auto-verified Users: Certain users have automatic verification and merge privileges

For more information, please refer to the project documentation or contact the maintainers.

@qodo-code-review
Copy link
Copy Markdown

Review Summary by Qodo

Skip AI analysis on /approve cancel command

🐞 Bug fix 🧪 Tests

Grey Divider

Walkthroughs

Description
• Skip AI analysis when cancelling approval via /approve cancel
• Add condition to check approval is not being removed
• Add test case verifying test oracle not called on cancel
Diagram
flowchart LR
  A["User runs /approve cancel"] --> B["label_by_user_comment called"]
  B --> C{"Check: APPROVE_STR<br/>and not remove?"}
  C -->|Yes| D["Call test oracle"]
  C -->|No| E["Skip test oracle"]
Loading

Grey Divider

File Changes

1. webhook_server/libs/handlers/issue_comment_handler.py 🐞 Bug fix +1/-1

Add remove check to skip AI analysis

• Modified condition to check _command == APPROVE_STR and not remove
• Prevents test oracle from being triggered when approval is cancelled
• Ensures AI analysis only runs on approval, not cancellation

webhook_server/libs/handlers/issue_comment_handler.py


2. webhook_server/tests/test_issue_comment_handler.py 🧪 Tests +24/-0

Add test for approve cancel path

• Added new test test_approve_cancel_does_not_call_test_oracle
• Verifies that /approve cancel command does not trigger test oracle
• Uses mocks to isolate the behavior and assert oracle is not called

webhook_server/tests/test_issue_comment_handler.py


Grey Divider

Qodo Logo

@qodo-code-review
Copy link
Copy Markdown

qodo-code-review Bot commented Mar 1, 2026

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider


Remediation recommended

1. Cancel arg not normalized🐞 Bug ✓ Correctness
Description
The new oracle guard relies on remove being set, but remove is only true when `_args ==
"cancel" exactly; _args is not stripped/normalized. As a result, /approve  cancel or /approve
cancel  can still be treated as an approval (remove=False`) and will continue to trigger AI
analysis.
Code

webhook_server/libs/handlers/issue_comment_handler.py[358]

+            if _command == APPROVE_STR and not remove:
Evidence
Commands are extracted from comment lines without trimming whitespace (only / is stripped), then
parsed via split(" ", 1) leaving _args potentially containing leading/trailing spaces. Since
remove is computed via exact string equality to "cancel", whitespace variants won’t set
remove=True, so the new and not remove condition still evaluates true and schedules the oracle
task. The added test covers only the idealized single-space form.

webhook_server/libs/handlers/issue_comment_handler.py[92-92]
webhook_server/libs/handlers/issue_comment_handler.py[170-172]
webhook_server/libs/handlers/issue_comment_handler.py[209-210]
webhook_server/libs/handlers/issue_comment_handler.py[358-365]
webhook_server/tests/test_issue_comment_handler.py[1490-1511]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`/approve cancel` is intended to set `remove=True` so the oracle trigger is skipped. Today, `remove` is only set when `_args == &amp;quot;cancel&amp;quot;` exactly, but `_args` is not trimmed/normalized, so whitespace variants (double spaces, trailing space) can still trigger AI analysis.
### Issue Context
Commands are extracted from comment lines with only `/` stripped (not whitespace). Args are parsed via `split(&amp;quot; &amp;quot;, 1)` which preserves leading/trailing whitespace in the remainder.
### Fix Focus Areas
- webhook_server/libs/handlers/issue_comment_handler.py[92-92]
- webhook_server/libs/handlers/issue_comment_handler.py[170-210]
- webhook_server/libs/handlers/issue_comment_handler.py[358-365]
- webhook_server/tests/test_issue_comment_handler.py[1490-1511]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

ⓘ The new review experience is currently in Beta. Learn more

Grey Divider

Qodo Logo

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Mar 1, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5fe14f0 and 24e7be6.

📒 Files selected for processing (1)
  • webhook_server/tests/test_issue_comment_handler.py

Walkthrough

Tighten the /approve handling to avoid triggering the test oracle when the command is a cancellation (remove flag set) and add a test to ensure /approve cancel does not call the oracle.

Changes

Cohort / File(s) Summary
Handler Logic
webhook_server/libs/handlers/issue_comment_handler.py
Modified condition that triggers call_test_oracle from if _command == APPROVE_STR: to if _command == APPROVE_STR and not remove: to skip oracle invocation when approval is being removed.
Test Coverage
webhook_server/tests/test_issue_comment_handler.py
Added test_approve_cancel_does_not_call_test_oracle to assert that /approve cancel does not invoke call_test_oracle (no background task created).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • fix: add-allowed-user #806 — Modifies IssueCommentHandler.user_commands logic; related area where approve/command handling was edited.

Suggested labels

size/XS, commented-rnetser, commented-coderabbitai[bot]

Suggested reviewers

  • myakove
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix: skip AI analysis on /approve cancel' directly and clearly describes the main change: preventing AI analysis from being invoked when canceling an approval.
Linked Issues check ✅ Passed The pull request fully addresses all coding requirements from issue #1006: adds 'and not remove' check to the approve condition and includes the test_approve_cancel_does_not_call_test_oracle test method.
Out of Scope Changes check ✅ Passed All changes are tightly scoped to the issue objectives: one-line condition fix and one test method addition, with no extraneous modifications.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix-no-ai-on-cancel

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@rnetser
Copy link
Copy Markdown
Collaborator Author

rnetser commented Mar 1, 2026

/build-and-push-container

@myakove-bot
Copy link
Copy Markdown
Collaborator

New container for ghcr.io/myk-org/github-webhook-server:pr-1007 published

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@webhook_server/tests/test_issue_comment_handler.py`:
- Around line 1495-1511: The cancel-path test should also assert that no
background tasks are scheduled: patch asyncio.create_task (e.g. with
patch("asyncio.create_task", new_callable=AsyncMock) or patch.object(asyncio,
"create_task", new_callable=AsyncMock)) around the call to
issue_comment_handler.user_commands and then assert that the mocked create_task
was not called; keep the existing patches (create_comment_reaction,
labels_handler.label_by_user_comment, and call_test_oracle) and use the same
user_commands invocation (function user_commands in issue_comment_handler) but
add mock_create_task.assert_not_called() after the await.

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 02cc72c and 5fe14f0.

📒 Files selected for processing (2)
  • webhook_server/libs/handlers/issue_comment_handler.py
  • webhook_server/tests/test_issue_comment_handler.py

Comment thread webhook_server/tests/test_issue_comment_handler.py Outdated
Strengthen cancel-path test by also patching asyncio.create_task
and asserting it was not called.
@rnetser
Copy link
Copy Markdown
Collaborator Author

rnetser commented Mar 1, 2026

/verified

@myakove myakove merged commit 8585312 into main Mar 1, 2026
8 of 9 checks passed
@myakove myakove deleted the fix-no-ai-on-cancel branch March 1, 2026 16:09
@myakove-bot
Copy link
Copy Markdown
Collaborator

Successfully removed PR tag: ghcr.io/myk-org/github-webhook-server:pr-1007.

@myakove-bot
Copy link
Copy Markdown
Collaborator

New container for ghcr.io/myk-org/github-webhook-server:latest published

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: AI analysis runs on /approve cancel

3 participants