CLI QA: bugs and UX issues found in pred v0.3.0

[GiggleLiu]

Summary

Automated feature testing of the pred CLI (v0.3.0) found 4 bugs, 5 UX issues, and 4 cosmetic issues. The core workflow (create → reduce → solve, piping, JSON output) works well. Issues are concentrated in edge cases and discoverability.

Bugs

1. 9 problem types listed in pred create --help but cannot be created

BinPacking, BicliqueCover, CVP, ILP, MaximumSetPacking, MinimumSetCovering, PaintShop, CircuitSAT, MaximalIS/BMF appear in [possible values] and show parameter help (pred create <PROBLEM> with no flags), but their parameters are not registered as clap flags. Example:

$ pred create BinPacking --sizes 3,5,2 --capacity 7
error: unexpected argument '--sizes' found

Expected: Either remove from possible values, or show "CLI creation not yet supported" in the problem-specific help.

2. MaximalIS fuzzy match suggests itself

$ pred create MaximalIS --graph 0-1,1-2
Error: Unknown problem: MaximalIS

Did you mean: MaximalIS?

The creation handler is missing, but the name IS recognized — so the suggestion is self-referential.

3. Error messages print twice

Multiple commands (show, solve, reduce) print the error message twice, e.g.:

$ pred show NONEXISTENT
Error: Unknown problem: NONEXISTENT
...
Error: Unknown problem: NONEXISTENT

Likely stderr + stdout duplication or double-print in the error path.

4. Bare pred appends unrelated pred solve help text

Running pred with no subcommand outputs the top-level help PLUS the full pred solve --help text (ILP backend info, solver examples, etc.) — ~71 lines. The pred solve block appears to be appended erroneously.

UX Issues

5. to/from naming is counterintuitive

pred to MIS means "what reduces TO MIS" (incoming edges), but users instinctively read it as "where can MIS go TO" (outgoing edges). Multiple independent testers got the direction wrong.

Suggestion: Consider pred sources MIS / pred targets MIS, or add prominent direction labels.

6. pred show ignores variant suffix

pred show MIS/UnitDiskGraph shows ALL 7 variants, identical to pred show MIS. Since to/from DO use the variant to select a specific node, this inconsistency is surprising.

7. "No solution found" is ambiguous for brute-force

When brute-force fails on large instances (e.g., 72-variable SpinGlass from Factoring reduction), the message "No solution found" doesn't distinguish infeasible problem vs. search space exhaustion.

8. MaxMatching alias missing

The pred create --help "Flags by problem type" section references MaxMatching, but only MaximumMatching (full name) is accepted.

9. pred create <PROBLEM> (no flags) exits code 0

When showing problem-specific help instead of creating, exit code is 0. Scripts may misinterpret this as success.

Cosmetic Issues

10. --timeout and --cost defaults shown twice in help

--timeout <TIMEOUT>  Timeout in seconds ... [default: 0] [default: 0]

11. -q (quiet) flag has no observable effect

No commands produce suppressible stderr messages in normal operation.

12. --json is a no-op for solve, evaluate, inspect

These commands always output JSON regardless of the flag. Only exploration commands (list, show, path, to, from) have a distinct human-readable format.

13. --to and --via silently coexist in pred reduce

When both flags are provided, no warning is given. Behavior is unclear.

What Works Well

• Core workflow: create → reduce → solve pipeline works correctly
• Piping: 3-stage pipelines (create | reduce | solve) work perfectly
• Help system: progressive disclosure, cross-references, examples
• Aliases and case-insensitive fuzzy matching
• Path finding with overhead composition
• JSON output (--json, -o) across all exploration commands
• Shell completions (zsh, bash)
• Random instance generation with --seed reproducibility

Test Report

Full report: docs/test-reports/test-feature-20260306-cli.md