Skip to content

fix(formatter): show full error message for test failures (#690)#729

Merged
aeppling merged 2 commits intortk-ai:developfrom
ousamabenyounes:fix/test-failure-error-message-truncation
Mar 21, 2026
Merged

fix(formatter): show full error message for test failures (#690)#729
aeppling merged 2 commits intortk-ai:developfrom
ousamabenyounes:fix/test-failure-error-message-truncation

Conversation

@ousamabenyounes
Copy link
Copy Markdown
Contributor

Problem

format_compact() truncated all test failure error messages to 2 lines via .take(2). Playwright errors look like:

Error: expect(locator).toHaveText(expected)   ← line 1 (visible)
                                               ← line 2 (visible, empty)
Expected: 'Submit'                             ← cut here
Received: 'Loading'                            ← cut here
Call log:                                      ← cut here
  - waiting for getByRole('button', ...)       ← cut here

Agents saw only "Error: expect(locator).toHaveText(expected)" — no diff, no call log, nothing actionable. This caused the 170 tool calls / 3.6M token loop described in #690.

Fix

Replace .take(2).join(" ") with a full line-by-line iteration of error_message. The summary line (PASS (28) FAIL (2)) stays compact. Only the per-failure detail is now fully preserved.

// Before
let error_preview: String = failure.error_message.lines().take(2).collect::<Vec<_>>().join(" ");
lines.push(format!("   {}", error_preview));

// After
for line in failure.error_message.lines() {
    lines.push(format!("   {}", line));
}

Impact

Affects all test runners using TestResult: Playwright, Vitest, Cargo test, Pytest, .NET — any test failure is now fully visible in compact mode.

Tests

4 new unit tests in formatter.rs:

  • test_compact_shows_full_error_message — asserts expected/received diff and call log are present
  • test_compact_summary_line_is_concise — summary stays on one line
  • test_compact_all_pass_is_one_line — success output stays compact
  • test_compact_single_line_error_no_trailing_noise — single-line errors still work
cargo test parser::formatter::tests
✓ 4 passed
cargo test
✓ 983 passed, 0 failed

Closes #690

@aeppling
Copy link
Copy Markdown
Contributor

Hello Ousama !

Thanks for contributing again :)

The truncation fix is of course correct, but PR 690 also asked for pass-through mechanism like others commands,

One thing missing : playwright_cmd.rs is the only test runner that doesn't call tee_and_hint, which is RTK mechanism for pass-through on failure, used by vitest, pytest, cargo, go, etc.

That was asked in #690, could you maybe fix this little as well so we resolve the full issue ?

Should be easy to add and straght-forward to merge after

@aeppling aeppling self-assigned this Mar 19, 2026
@aeppling
Copy link
Copy Markdown
Contributor

If no news from you, i'll merge what you have done, it solve a part of the issue, but i'll not be able to close the issue if not totally resolved.

Thanks again for solving issues

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Mar 20, 2026

CLA assistant check
All committers have signed the CLA.

format_compact() was truncating error messages to 2 lines via .take(2).
Playwright (and vitest, pytest, cargo test) errors contain the critical
expected/received diff and call log starting at line 3+, so agents saw
only the error type with no actionable debug information.

Fix: iterate all lines of error_message for each failure instead of
taking the first 2. Summary line stays compact; only the per-failure
detail is preserved in full.

Affects all test runners using TestResult (playwright, vitest, cargo,
pytest, dotnet) — any test failure is now fully visible.

Closes rtk-ai#690

Signed-off-by: Ousama Ben Younes <benyounes.ousama@gmail.com>
@ousamabenyounes ousamabenyounes force-pushed the fix/test-failure-error-message-truncation branch from 5312363 to c0f5043 Compare March 21, 2026 15:53
Add crate::tee::tee_and_hint call to playwright_cmd::run(), matching
the pattern used by all other test runners (vitest, pytest, cargo,
go, rspec, rake). When playwright fails, the full raw output is saved
to ~/.local/share/rtk/tee/ and a one-line hint is appended so the
LLM can read the complete output without re-running.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Ousama Ben Younes <benyounes.ousama@gmail.com>
@ousamabenyounes ousamabenyounes force-pushed the fix/test-failure-error-message-truncation branch from c0f5043 to b4ccf04 Compare March 21, 2026 15:58
@ousamabenyounes
Copy link
Copy Markdown
Contributor Author

Hey, happy to contribute!

Added tee_and_hint in playwright_cmd::run() — same pattern as vitest, pytest, cargo, go, rspec and rake. On failure, the full raw output is saved to ~/.local/share/rtk/tee/ and a one-line hint is appended to the filtered output so agents can read the complete trace without re-running.

On the test side: the tee_and_hint logic itself is covered by the 12 existing tests in tee.rs, and the playwright parsing (JSON/regex/passthrough) by the 5 existing tests in playwright_cmd.rs. The run() function isn't unit tested — consistent with how vitest, pytest and the other runners are handled in the codebase.

All 1060 tests pass. This should close #690 fully.

@aeppling
Copy link
Copy Markdown
Contributor

Hello Ousama ,

You're just missing the CLA sign, you can sign it by clicking the git check status

i'll approve & merge once signed

@ousamabenyounes
Copy link
Copy Markdown
Contributor Author

Hi, thanks for the reminder! I've just signed the CLA.

@aeppling
Copy link
Copy Markdown
Contributor

Merged, thanks for the contribution !

@aeppling aeppling merged commit dc6b026 into rtk-ai:develop Mar 21, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants