ci(ios-e2e): classify failure mode; skip retry on real assertion failures#328
Conversation
…ailures (#324) Adds an `is_retriable_failure()` helper that runs after each failed xcodebuild attempt. If a simulator-side crash report (*StillPoint*.ips under ~/Library/Logs/DiagnosticReports) exists, the failure is treated as infra regardless of surface symptom — this keeps the macos-26 XPC fault path retriable even when it surfaces as an "X did not appear" XCTAssertTrue timeout (the failure mode that hit PR #311 / PR #308 this week). Otherwise, an XCTest assertion signature in the log (`XCTAssertX failed`, or a method-level `error: -[<TestClass> <testMethod>] :` frame) marks the failure non-retriable and the loop exits without consuming the retry budget. Default for any unmatched failure stays retriable, preserving current behavior on infra/transient/unknown errors. Updates docs/testing/e2e-policy.md Section 1 to note the iOS lanes now enforce the classification. Closes #324 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe iOS E2E runner now creates per-attempt sentinels and classifies failures by scanning the attempt log and simulator DiagnosticReports; simulator crash diagnostics matching Changes
Sequence Diagram(s)sequenceDiagram
participant Runner as "run-ios-tests.sh"
participant Xcode as "xcodebuild (attempt)"
participant FS as "Attempt Log (artifacts)"
participant Diag as "Simulator DiagnosticReports"
Runner->>Xcode: start attempt (create .start sentinel)
Xcode-->>FS: write attempt log
Xcode-->>Runner: exit non-zero
Runner->>Diag: scan DiagnosticReports for *StillPoint*.ips newer than sentinel
alt Simulator crash report found
Diag-->>Runner: crash report matched
Runner->>Runner: classify as retriable (consume retry) and continue loop
else No crash report
Runner->>FS: grep attempt_log for XCTest assertion / test-selector frames
alt Assertion/test-selector patterns found
Runner->>Runner: classify as non-retriable -> fail fast (no retry)
else Unknown/infra patterns
Runner->>Runner: classify as retriable/unknown -> consume retry and continue
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. 📋 Issue PlannerBuilt with CodeRabbit's Coding Plans for faster development and fewer bugs. View plan used: ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/e2e/run-ios-tests.sh`:
- Around line 169-173: The current check treats any existing *StillPoint* in
diag_dir as infra even if it’s from a previous run; fix by scoping detection to
files created during this attempt: at test start create a marker file (e.g.,
touch "${TMPDIR}/stillpoint_attempt_marker") and then change the find invocation
in the block that references diag_dir to only match files newer than that marker
(use find "${diag_dir}" -type f -name '*StillPoint*' -newer
"${TMPDIR}/stillpoint_attempt_marker" -print -quit 2>/dev/null | grep -q .).
Update references to the marker variable where you create and check it so only
crash reports from the current attempt trigger the early return.
- Around line 179-181: The current grep in run-ios-tests.sh uses a regex
requiring a Module.Class (dot) form and misses lines like "error: -[TestClass
testMethod]"; update the pattern used in the grep -qE call to accept either
"Module.Class" or plain "Class" before the space and test method. Specifically,
modify the regex in the grep that references "$log" (the line containing grep
-qE 'error:
-\[[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*[[:space:]]+test[A-Za-z0-9_]+'
"$log") to allow an optional ".Module" portion (e.g., make the dot-prefixed part
optional) so it matches both "-[Module.Class testX]" and "-[Class testX]"
formats.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0c33c3db-5370-4a50-a3bc-c51845dd2de3
📒 Files selected for processing (2)
docs/testing/e2e-policy.mdscripts/e2e/run-ios-tests.sh
…ssertion regex CodeRabbit findings on PR #328: 1. Crash-report check at scripts/e2e/run-ios-tests.sh:171 used to match any *StillPoint*.ips in DiagnosticReports — including stale files from a prior attempt within the same job. Add `-newer "$log"` so detection scopes to .ips files written after the current attempt's log was created. 2. Method-level error-frame regex at line 180 required `Module.Class` (Swift form). Broaden to also accept plain `Class` (ObjC form) and require the closing `]` plus trailing `:` so the pattern is anchored. Both changes preserve the existing fail-fast vs retry behavior on real assertion failures while preventing the two specific edge cases CR flagged.
|
@coderabbitai Fixed in |
|
@coderabbitai Fixed in |
|
Tip For best results, initiate chat on the files or code changes.
|
|
Tip For best results, initiate chat on the files or code changes.
|
|
@coderabbitai full review |
|
@CodeAnt-AI review |
|
@cursor review |
✅ Actions performedFull review triggered. |
|
CodeAnt AI is running the review. |
codeant-ai
Bot
added
the
size:M
Sequence DiagramThis PR updates the iOS E2E runner to classify each failed xcodebuild attempt using simulator crash reports and XCTest assertion signatures, skipping retries on real test failures while preserving retries for infra and unknown issues. sequenceDiagram
participant CI
participant Script
participant Xcodebuild
participant DiagnosticReports
CI->>Script: Start iOS iE2E lane with retry budget
loop Each attempt
Script->>Xcodebuild: Run xcodebuild tests for lane
Xcodebuild-->>Script: Attempt fails and writes log
Script->>DiagnosticReports: Check for recent StillPoint crash report
alt Crash report found
Script->>Script: Treat as infra and retry if budget remains
else No crash report
Script->>Script: Scan log for XCTest assertion signatures
alt Assertion signature found
Script-->>CI: Mark lane failed without retry
else No assertion signature
Script->>Script: Treat as retriable and retry if budget remains
end
end
end
Generated by CodeAnt AI |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/e2e/run-ios-tests.sh`:
- Around line 167-173: The current detection uses the log file mtime ("$log")
which is updated continuously by tee/xcodebuild; instead create a
start-of-attempt marker file (e.g., attempt_marker="${TMPDIR}ios_test_start.$$"
or similar) immediately before invoking xcodebuild, touch that marker to record
the attempt start, then change the find check that references "${diag_dir}" to
use -newer "$attempt_marker" instead of -newer "$log", and finally remove the
marker after the run; update any references to "$log" in the crash-detection
block to use the new marker (look for diag_dir, the find ... -newer usage, and
the xcodebuild/tee invocation).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 6e37081a-6eb3-4591-af87-14fc023f214f
📒 Files selected for processing (1)
scripts/e2e/run-ios-tests.sh
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
|
CodeAnt AI finished running the review. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Previous fix (commit 57db5cf) used `find -newer "$log"` to scope crash detection to the current attempt. CR / CodeAnt both flagged a real bug: `tee` keeps updating the .log file's mtime throughout the xcodebuild run, so a mid-run .ips crash report could end up older than the log's final mtime and never match. Real simulator crashes that produced EXC_GUARD/XPC_EXIT_REASON_FAULT this week — exactly the failure mode the retry budget exists to absorb — would be misclassified as non-retriable assertion failures and skip the retry. Fix: in run_lane(), `: > artifacts/e2e/ios/<lane>-attempt-N.start` before invoking xcodebuild. is_retriable_failure() now takes the marker path as an optional 2nd arg (defaulted from the log path) and compares crash reports against the marker's attempt-start mtime instead of the log's end-of-attempt mtime. The retry loop passes the marker explicitly. Addresses CR PR review comment 3174801442 and CodeAnt PR review comment 3174804692.
|
@coderabbitai full review |
|
@graphite-app re-review |
|
@CodeAnt-AI review |
|
@cursor review |
✅ Actions performedFull review triggered. |
|
CodeAnt AI is running the review. |
|
A review was recently triggered for this PR. Please wait 29s before retriggering. |
|
A review was recently triggered for this PR. Please wait 26s before retriggering. |
|
A review was recently triggered for this PR. Please wait 7s before retriggering. |
1 similar comment
|
A review was recently triggered for this PR. Please wait 7s before retriggering. |
|
CodeAnt AI is running the review. |
codeant-ai
Bot
added
size:M
Sequence DiagramThis PR updates the iOS end-to-end test script to classify each failed xcodebuild attempt using simulator crash reports and XCTest assertion patterns, deciding whether to retry or fail immediately without consuming the retry budget on real assertion failures. sequenceDiagram
participant CI
participant IOSScript
participant Xcodebuild
participant Diagnostics
CI->>IOSScript: Run iOS test lane
loop Each attempt until max retries
IOSScript->>Xcodebuild: Execute tests and write log
Xcodebuild-->>IOSScript: Return failure and log
IOSScript->>Diagnostics: Check crash reports for app
IOSScript->>IOSScript: Classify failure as retriable or non-retriable
alt Retriable and retries remain
IOSScript-->>CI: Record retriable attempt and start next attempt
else Non-retriable or retries exhausted
IOSScript-->>CI: Fail lane without further retries
end
end
Generated by CodeAnt AI |
|
CodeAnt AI finished running the review. |
|
CodeAnt AI is running the review. |
codeant-ai
Bot
added
size:M
Sequence DiagramThis PR updates the iOS end-to-end test script to classify failed xcodebuild attempts using simulator crash reports and XCTest assertion signatures, deciding whether to consume the retry budget or fail fast on real test/product regressions. sequenceDiagram
participant CI
participant Script as iOS e2e script
participant Xcode as xcodebuild
participant Diag as DiagnosticReports
CI->>Script: Start iOS lane
Script->>Script: Create attempt marker for this attempt
Script->>Xcode: Run tests and write attempt log
Xcode-->>Script: Test failure with log
Script->>Diag: Check for StillPoint crash after attempt marker
alt Crash report found
Diag-->>Script: Crash detected for this attempt
Script->>Script: Mark failure retriable and schedule retry
else No crash and assertion in log
Script->>Script: Detect XCTest assertion signature
Script-->>CI: Fail lane without retry
else Other failure
Script->>Script: Treat as retriable and schedule retry
end
Generated by CodeAnt AI |
|
CodeAnt AI finished running the review. |
|
CodeAnt AI is running the review. |
codeant-ai
Bot
added
size:M
Sequence DiagramThis PR updates the iOS end to end test script to classify xcodebuild failures using crash reports and XCTest log patterns so that real assertion failures fail fast while infra and timeout shaped issues consume the limited retry budget. sequenceDiagram
participant CI
participant Script
participant Xcodebuild
participant Diagnostics
CI->>Script: Start iOS e2e lane
loop Attempts up to max retries
Script->>Xcodebuild: Run tests and write attempt log
Xcodebuild-->>Script: Test failure with log
Script->>Diagnostics: Check for StillPoint crash report after attempt start
alt Crash report or timeout shaped failure
Script-->>Script: Treat as retriable failure
Script-->>Script: Increment attempt and retry
else Assertion signature found or no retries left
Script-->>CI: Fail lane without further retries
break
end
end
Generated by CodeAnt AI |
|
CodeAnt AI finished running the review. |
|
CodeAnt AI is running the review. |
codeant-ai
Bot
added
size:M
Sequence DiagramThis PR adds a classifier to the iOS e2e runner that inspects simulator crash reports and XCTest log signatures after each failed attempt to decide whether to consume the retry budget or fail fast on real assertion failures. sequenceDiagram
participant CI
participant Script as iOS e2e script
participant Xcode as xcodebuild
participant Diagnostics as Simulator diagnostics
CI->>Script: Run iOS tests with retry budget
loop Each attempt
Script->>Xcode: Run tests for selected lane
Xcode-->>Script: Result with attempt log
alt Tests pass
Script-->>CI: Report success and stop
break
else Tests fail
Script->>Diagnostics: Check crash reports and classify failure
Script->>Script: Scan test log for timeout or assertion signatures
alt Non-retriable assertion failure
Script-->>CI: Mark lane failed without retry
break
else Retriable or unknown failure
Script-->>CI: Consume retry and start next attempt
end
end
end
Generated by CodeAnt AI |
|
CodeAnt AI finished running the review. |
|
CodeAnt AI is running the review. |
codeant-ai
Bot
added
size:M
Sequence DiagramThis PR updates the iOS end-to-end test script to classify failures using simulator crash reports and XCTest log signatures to decide whether to retry or fail immediately without consuming the retry budget. sequenceDiagram
participant CI
participant IosE2EScript as iOS e2e script
participant Xcodebuild
participant CrashReports as Simulator diagnostic reports
CI->>IosE2EScript: Run iOS lane
IosE2EScript->>Xcodebuild: Execute tests for attempt
Xcodebuild-->>IosE2EScript: Return failure and test log
IosE2EScript->>CrashReports: Check for new StillPoint crash report
alt Crash report found
IosE2EScript-->>CI: Mark as infra failure and schedule retry
else No crash report
IosE2EScript->>IosE2EScript: Scan log for XCTest assertions
alt Assertion signature found
IosE2EScript-->>CI: Fail immediately without retry
else Timeout or unknown signature
IosE2EScript-->>CI: Treat as retriable and schedule retry
end
end
Generated by CodeAnt AI |
|
CodeAnt AI finished running the review. |
1 participant
User description
Summary
xcodebuildattempt,scripts/e2e/run-ios-tests.shnow decides whether to consume the retry budget based on the attempt's log + simulator crash reports.scripts/e2e/run-ios-tests.sh:~/Library/Logs/DiagnosticReportsfor any*StillPoint*crash report. If present, treat failure as infra (retry as before). This catches macos-26 XPC faults (EXC_GUARD/XPC_EXIT_REASON_FAULT) that surface asXCTAssertTrue failed - <X> did not appeartimeouts but are actually launchd/sim-level crashes — exactly the failure shape that hit PR feat(ios): Settings username inline edit (closes #282) #311 and PR feat: timer extensions + bonus time (#90) #308 this week.XCTAssertX failed, orerror: -[<TestClass>.<TestClass> <testMethod>]method-level error frames). If matched, exit fast without consuming retry budget.docs/testing/e2e-policy.mdSection 1 with an Enforcement note so future contributors know the script enforces the rules, not just human reviewers.Diff: 2 files, +42/-0. No behavior change unless the retry budget is exceeded — single-attempt success path is identical.
Closes #324
Test plan
bash -n scripts/e2e/run-ios-tests.shpasses (syntax check)ios-e2e-smokeandios-e2e-criticallanes pass on this PR's HEAD with a single attempt (zero added cost on success path)de91966retry behavior on infra flakes —Cursor Bugbot/CodeAntreviewers don't flag scope driftdocs/testing/e2e-policy.mdSection 1 shows the new Enforcement paragraph cleanlyscripts/e2e/run-ios-tests.sh(+40) anddocs/testing/e2e-policy.md(+2) changedTrade-offs / known limits
~/Library/Logs/DiagnosticReports). Works on the macos runner because it's ephemeral; would need scoping if used on a long-lived host.XCTAssertX failedand method-level error frames. Doesn't coverXCTFail()calls without anXCTAssertprefix; those would currently fall through to "retriable default." Acceptable starting point; can tighten in a follow-up if observed..xcresultbundles — listed as out-of-scope in #324. Pure log-line + filesystem checks; future enhancement could read structured xcresult data for richer signals.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
CodeAnt-AI Description
Stop retrying iOS e2e runs on real test assertion failures
What Changed
Impact
✅ Faster detection of real iOS regressions✅ Fewer wasted retries on assertion failures✅ Clearer iOS e2e failure handling🔄 Retrigger CodeAnt AI Review
Details
💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.