Skip to content

test: add --help scenario tests for cut, echo, head, printf, sed, sort, tail, wc#151

Merged
thieman merged 2 commits intomainfrom
thieman/add-help-flag-tests
Mar 26, 2026
Merged

test: add --help scenario tests for cut, echo, head, printf, sed, sort, tail, wc#151
thieman merged 2 commits intomainfrom
thieman/add-help-flag-tests

Conversation

@thieman
Copy link
Copy Markdown
Collaborator

@thieman thieman commented Mar 25, 2026

Problem

T-1: Missing --help scenario tests for 8 commands

RULES.md requires: "Every supported flag MUST have correctness tests."

These 8 commands had no scenario tests verifying that --help (and -h where supported) exits with the correct code and prints usage to stdout:

  • cut--help only
  • echo — no --help flag; documents that it prints the literal string
  • head--help and -h
  • printf--help only (exits 2 per implementation)
  • sed--help and -h
  • sort--help and -h
  • tail--help and -h
  • wc--help only

Fix

Added help/help_flag.yaml (and help/short_help_flag.yaml for -h variants) for each command. Tests verify:

  • Correct exit code (0 for all except printf which exits 2)
  • Usage information printed to stdout (via stdout_contains)
  • No stderr output

All tests are marked skip_assert_against_bash: true since bash handles these differently.

Test plan

  • go test ./tests/... -run TestShellScenarios passes with all 12 new scenario files

🤖 Generated with Claude Code

…t, tail, wc

These 8 commands were missing scenario tests for their --help/-h flags.
Each test verifies: exit code 0 (or 2 for printf), usage printed to stdout,
no stderr output. Also documents that echo treats --help as literal text.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@thieman
Copy link
Copy Markdown
Collaborator Author

thieman commented Mar 25, 2026

@codex review this PR

Copy link
Copy Markdown
Collaborator Author

@thieman thieman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-Review: PR #151 — Add help flag scenario tests

Scope reviewed: 12 new YAML scenario test files across 8 commands (cut, echo, head, printf, sed, sort, tail, wc).

Overall assessment: needs minor fixes — the tests correctly capture help-flag behavior and all pass. Two findings below.

Coverage Summary

Code path Scenario test Go test Status
cut --help → stdout, exit 0 cut/help/help_flag.yaml Covered
echo --help → prints literal text, exit 0 echo/help/help_flag.yaml Covered
head --help → stdout, exit 0 head/help/help_flag.yaml Covered
head -h → stdout, exit 0 head/help/short_help_flag.yaml Covered
printf --help → stdout, exit 2 printf/help/help_flag.yaml Covered
sed --help → stdout, exit 0 sed/help/help_flag.yaml Covered
sed -h → stdout, exit 0 sed/help/short_help_flag.yaml Covered
sort --help → stdout, exit 0 sort/help/help_flag.yaml Covered
sort -h → stdout, exit 0 sort/help/short_help_flag.yaml Covered
tail --help → stdout, exit 0 tail/help/help_flag.yaml Covered
tail -h → stdout, exit 0 tail/help/short_help_flag.yaml Covered
wc --help → stdout, exit 0 wc/help/help_flag.yaml Covered

Findings

# Priority File Finding
1 P2 Badge tests/scenarios/cmd/echo/help/help_flag.yaml:3 skip_assert_against_bash is set but bash behavior matches rshell
2 P3 Badge tests/scenarios/cmd/printf/help/help_flag.yaml:9 stdout_contains assertion is too loose

Positive Observations

  • All 12 scenarios correctly align with the actual builtin implementations (verified by reading source and running tests).
  • exit_code: 2 for printf --help correctly matches the implementation.
  • Commands without a -h short flag (cut, wc) correctly omit short_help_flag.yaml.
  • All files follow the project YAML conventions (|+ block scalars, expect.stderr exact match).

Comment thread tests/scenarios/cmd/echo/help/help_flag.yaml Outdated
Comment thread tests/scenarios/cmd/printf/help/help_flag.yaml
@thieman
Copy link
Copy Markdown
Collaborator Author

thieman commented Mar 25, 2026

Iteration 1 self-review result: 2 findings — 1× P2, 1× P3. No security or correctness issues. The P2 is an unnecessary skip_assert_against_bash: true in the echo scenario (bash behavior matches); the P3 is a loose stdout_contains assertion in the printf scenario.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bfee7e45e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@@ -0,0 +1,11 @@
description: echo --help is not recognised as a flag and is printed as literal text.
# skip: echo does not implement --help; bash behaves the same way
skip_assert_against_bash: true
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Re-enable bash comparison for this echo --help scenario

This test is explicitly asserting bash-equivalent behavior (echo --help is treated as a literal string), but skip_assert_against_bash: true prevents it from running in TestShellScenariosAgainstBash. I verified in bash that echo --help exits 0, writes --help to stdout, and leaves stderr empty, so the current expectations are compatible; skipping here removes useful compatibility coverage and can hide regressions.

Useful? React with 👍 / 👎.

- tests/scenarios/cmd/echo/help/help_flag.yaml: Remove
  skip_assert_against_bash: true since bash echo --help behaves
  identically to rshell (prints "--help", exits 0). The test now
  validates against bash as it should.

- tests/scenarios/cmd/printf/help/help_flag.yaml: Strengthen
  stdout_contains from ["printf"] to ["printf: usage: printf"] to
  match the actual help string and avoid loose matching.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@thieman thieman marked this pull request as ready for review March 26, 2026 14:48
@thieman thieman added this pull request to the merge queue Mar 26, 2026
Merged via the queue into main with commit d53e149 Mar 26, 2026
34 checks passed
@thieman thieman deleted the thieman/add-help-flag-tests branch March 26, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants