refactor(tests): extract CLI tests from main.rs into integration tests#159
Conversation
…ess testing This commit introduces a new set of integration tests for the `rmagic` CLI, leveraging the `assert_cmd` crate to ensure reliable subprocess execution and output validation. The tests cover various file types (ELF, PNG, JPEG, PDF, ZIP, GIF) and include checks for both standard and strict modes, as well as JSON output formatting. - Created `tests/cli_integration.rs` to house the new integration tests. - Implemented helper functions for generating temporary test files. - Enhanced test coverage for CLI argument parsing and output verification. These changes improve the robustness of the CLI testing framework and facilitate easier maintenance and expansion of test cases in the future. Co-authored-by: dosubot[bot] <131922026+dosubot[bot]@users.noreply.github.com> Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
This commit updates the Rust version specification in both `mise.lock` and `mise.toml` files. The `mise.lock` file now specifies Rust version `1.93.1`, while the `mise.toml` file has been changed to use the `latest` version instead of `stable`. These changes ensure that the project is aligned with the latest Rust tooling and dependencies. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
This commit reformats the `dist-workspace.toml` file to enhance readability by adjusting the spacing and organizing the `targets` list into a multi-line format. The changes do not affect the functionality but improve the clarity of the configuration. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
This commit updates several dependencies in the `Cargo.lock` file, including `anyhow` to version `1.0.102`, `bumpalo` to `3.20.2`, `getrandom` to `0.4.2`, `js-sys` to `0.3.91`, `quote` to `1.0.45`, `regex-syntax` to `0.8.10`, `syn` to `2.0.117`, and `wasm-bindgen` related packages to `0.2.114`. Additionally, the `r-efi` dependency has been added with version `6.0.0`. These updates ensure compatibility with the latest features and improvements in the respective libraries. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
|
Caution Review failedPull request was closed or merged during review Summary by CodeRabbit
WalkthroughExtracts fragile, process-wide CLI tests from Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Comment |
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 CI must passWonderful, this rule succeeded.All CI checks must pass. Release-plz PRs are exempt because they only bump versions and changelogs (code was already tested on main), and GITHUB_TOKEN-triggered force-pushes suppress CI.
🟢 Do not merge outdated PRsWonderful, this rule succeeded.Make sure PRs are within 10 commits of the base branch before merging
|
…to-integration-tests
|
Related Documentation 3 document(s) may need updating based on files changed in this PR: libMagic-rs Testing Strategy And Coverage RequirementsView Suggested Changes@@ -4,7 +4,7 @@
The libmagic-rs project implements a comprehensive, multi-layered testing strategy designed to ensure reliability, maintainability, and cross-platform compatibility for its pure-Rust replacement of libmagic. The testing approach combines traditional unit and integration testing with modern property-based testing techniques, enforcing strict coverage requirements of >85% line coverage.
-The project maintains [over 900 test cases](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/) across [22 source files with unit tests](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/) and [11 integration test files](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/), achieving [94%+ test coverage](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/codecov.yml). The testing infrastructure leverages [cargo-nextest for faster test execution](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/mise.toml#L8), [proptest for property-based testing](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/Cargo.toml#L166), and [cargo llvm-cov for coverage measurement](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/mise.toml#L7).
+The project maintains [over 900 test cases](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/) across [22 source files with unit tests](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/) and [12 integration test files](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/), achieving [94%+ test coverage](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/codecov.yml). The testing infrastructure leverages [cargo-nextest for faster test execution](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/mise.toml#L8), [proptest for property-based testing](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/Cargo.toml#L166), and [cargo llvm-cov for coverage measurement](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/mise.toml#L7).
A unique aspect of the testing strategy is the [build script testing pattern](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/build_helpers.rs), which extracts build logic into testable modules using `#[cfg(any(test, doc))]` to enable comprehensive testing of build-time code generation. All testing commands are executed through [mise exec](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/justfile#L12) to ensure consistent toolchain versions, and the [just ci-check command](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/justfile#L207) provides local CI parity for developers.
@@ -16,7 +16,7 @@
**Core library tests:**
- [src/lib.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/lib.rs) - 26 unit tests
-- [src/main.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/main.rs) - 55 unit tests (CLI functionality)
+- [src/main.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/main.rs) - Core CLI logic (tested via integration tests)
- [src/error.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/error.rs) - 24 unit tests
- [src/build_helpers.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/build_helpers.rs) - 35 unit tests
@@ -31,14 +31,14 @@
- [src/parser/ast.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/parser/ast.rs) - 36 unit tests
- [src/parser/preprocessing.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/parser/preprocessing.rs) - 32 unit tests
-The unit test count totals approximately [746 tests across all source files](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/).
+The unit test count totals approximately [691 tests across all source files](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/) (following CLI test extraction to integration tests).
### Integration Tests Directory
-The [tests/ directory](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/) contains [11 integration test files](https://github.com/EvilBit-Labs/libmagic-rs/tree/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests) with approximately 155 test functions testing complete workflows:
+The [tests/ directory](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/) contains [12 integration test files](https://github.com/EvilBit-Labs/libmagic-rs/tree/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests) testing complete workflows:
1. **[tests/integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/integration_tests.rs)** (378 lines) - Core end-to-end tests covering built-in rules, file loading, and custom magic rules, including quad type tests (`test_quad_lequad_matches_little_endian_value`, `test_quad_bequad_matches_big_endian_value`, `test_quad_signed_negative_one`, `test_quad_nested_child_rule_with_offset`)
-2. **[tests/cli_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/cli_integration_tests.rs)** (1,625 lines) - Comprehensive CLI testing including canonical libmagic compatibility, error handling, and edge cases
+2. **[tests/cli_integration.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/cli_integration.rs)** (887 lines) - CLI integration tests using `assert_cmd` for subprocess-based testing. This approach provides natural process isolation and eliminates fd manipulation. Tests cover builtin detection (ELF, PNG, JPEG, PDF, ZIP, GIF), stdin handling with truncation warnings, multiple file processing, error handling, timeout validation, output formats (text/JSON), shell completion, custom magic files, and edge cases like Unicode filenames and empty files.
3. **[tests/property_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/property_tests.rs)** (221 lines) - Property-based testing with proptest
4. **[tests/evaluator_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/evaluator_tests.rs)** - Evaluator integration tests for confidence calculation and configuration
5. **[tests/compatibility_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/compatibility_tests.rs)** - Compatibility with original libmagic implementation
@@ -111,7 +111,7 @@
### CI Coverage Workflow
The [CI workflow coverage job](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/.github/workflows/ci.yml#L82-L111):
-1. Generates coverage with [`--test-threads=1`](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/.github/workflows/ci.yml#L96) to prevent race conditions in stdin-mocking tests
+1. Generates coverage with standard `cargo test`
2. Combines reports and outputs lcov.info
3. Uploads to [Codecov](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/.github/workflows/ci.yml#L101-L107) and [Qlty](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/.github/workflows/ci.yml#L108-L111) services
@@ -403,7 +403,7 @@
| File Path | Description | Lines |
|-----------|-------------|-------|
| [tests/integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/integration_tests.rs) | Core end-to-end integration tests | 378 |
-| [tests/cli_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/cli_integration_tests.rs) | Comprehensive CLI testing | 1,625 |
+| [tests/cli_integration.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/cli_integration.rs) | CLI integration tests with assert_cmd | 887 |
| [tests/property_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/property_tests.rs) | Property-based tests with proptest | 221 |
| [tests/common/mod.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/tests/common/mod.rs) | Shared test utilities | 185 |
| [src/build_helpers.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/build_helpers.rs) | Build script testing module | 709 |✅ Accepted Testing Strategy And InfrastructureView Suggested Changes@@ -1,8 +1,8 @@
# Testing Strategy and Infrastructure
-The [libmagic-rs project maintains a comprehensive, multi-layered testing strategy with over 900 test cases across 22 source files with unit tests and 11 integration test files, achieving 94%+ test coverage](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384). The testing architecture combines unit tests, integration tests, property-based testing with proptest, snapshot testing with insta, CLI testing with assert_cmd, and performance benchmarking with criterion. [All testing is coordinated through cargo-nextest version 0.9.123-b.4 as the primary test runner](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384), with strict coverage requirements enforced through cargo llvm-cov (≥85% overall, >90% for new code).
-
-The infrastructure represents a durable testing architecture designed specifically for a magic file parser and evaluator, featuring unique patterns like build script testing through extracted modules, cross-platform CLI snapshot normalization, and property-based fuzzing of rule evaluation. [Testing tools are version-pinned through mise.toml to ensure consistency between local development and CI environments](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384).
+The [libmagic-rs project maintains a comprehensive, multi-layered testing strategy with over 900 test cases across 22 source files with unit tests and 12 integration test files, achieving 94%+ test coverage](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384). The testing architecture combines unit tests, integration tests, property-based testing with proptest, snapshot testing with insta, CLI testing with assert_cmd, and performance benchmarking with criterion. [All testing is coordinated through cargo-nextest version 0.9.123-b.4 as the primary test runner](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384), with strict coverage requirements enforced through cargo llvm-cov (≥85% overall, >90% for new code).
+
+The infrastructure represents a durable testing architecture designed specifically for a magic file parser and evaluator, featuring unique patterns like build script testing through extracted modules, subprocess-based CLI testing with assert_cmd, cross-platform CLI snapshot normalization, and property-based fuzzing of rule evaluation. [Testing tools are version-pinned through mise.toml to ensure consistency between local development and CI environments](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384).
## Test Categories and Organization
@@ -11,38 +11,40 @@
[Unit tests are located in `#[cfg(test)]` modules within source files](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384), keeping tests close to the code they validate:
- **[src/lib.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/lib.rs)** - 26 unit tests
-- **[src/main.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/main.rs)** - 55 unit tests (CLI functionality)
+- **[src/main.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/main.rs)** - 4 unit tests
- **[src/evaluator/mod.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/evaluator/mod.rs)** - 77 unit tests
- **[src/parser/grammar.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/parser/grammar.rs)** - 91 unit tests
- **[src/evaluator/operators.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e925ef6b3f2208fc8805a728ba3de55956f4447a/src/evaluator/operators.rs)** - 86 unit tests covering equality, inequality, comparison, and bitwise operations
-[Total unit test count: approximately 746 tests across all source files](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384).
+[Total unit test count: approximately 695 tests across all source files](https://app.dosu.dev/documents/62393a9d-87d8-43cd-b7c1-b117edd5a384).
### Integration Tests Directory
-[The tests/ directory contains 11 integration test files](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests) with approximately 155 test functions:
+[The tests/ directory contains 12 integration test files](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests) with approximately 205 test functions:
1. **[tests/integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/integration_tests.rs)** (468 lines) - [Core end-to-end tests covering the complete workflow: load database, evaluate files/buffers, verify output formatting and metadata](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/integration_tests.rs). Tests built-in rules, file loading, and custom magic rules. Includes quad type tests: `test_quad_lequad_matches_little_endian_value`, `test_quad_bequad_matches_big_endian_value`, `test_quad_signed_negative_one`, `test_quad_nested_child_rule_with_offset`.
-2. **[tests/cli_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration_tests.rs)** (1,625 lines) - [CLI integration tests using the canonical libmagic test suite](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration_tests.rs). Verifies command-line interface functionality by running against third_party/tests/, covering canonical test suite compatibility, multiple file processing, strict mode (`--strict`), built-in rules (`--use-builtin`), and JSON lines output.
-
-3. **[tests/property_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs)** (225 lines) - [Property-based tests verifying that evaluator never panics, buffer access is bounds-checked, metadata consistency, and serde roundtrips](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs). Includes [6 custom proptest strategies](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs#L18-L84): arb_offset_spec, arb_type_kind, arb_operator, arb_value, arb_magic_rule, arb_buffer.
-
-4. **[tests/evaluator_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/evaluator_tests.rs)** (248 lines) - [Evaluator integration tests through the public MagicDatabase API](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/evaluator_tests.rs). Tests confidence calculation, rule ordering, evaluation behavior, and validates that known file types return non-zero confidence scores.
-
-5. **[tests/compatibility_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility_tests.rs)** (360 lines) - [Compatibility tests ensuring identical results to the original libmagic](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility_tests.rs). [Test files are downloaded from the file/file repository](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility/README.md#L7) and compared against GNU file command.
-
-6. **[tests/parser_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/parser_integration_tests.rs)** (380 lines) - [End-to-end integration tests for magic file parser and database integration](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/parser_integration_tests.rs). Validates the complete flow from file/directory loading through rule evaluation.
-
-7. **[tests/directory_loading_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/directory_loading_tests.rs)** (431 lines) - [Integration tests for directory loading functionality](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/directory_loading_tests.rs). Validates `load_magic_directory()` function with various directory structures and batch loading of magic files from directories.
-
-8. **[tests/mime_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/mime_tests.rs)** (230 lines) - [MIME type mapping integration tests](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/mime_tests.rs). Tests MIME type detection through the public API including hardcoded mappings, prefix matching, and case insensitivity.
-
-9. **[tests/tags_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/tags_tests.rs)** (163 lines) - [Tag extraction integration tests](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/tags_tests.rs). Tests keyword extraction, case insensitivity, rule path tags, and custom keyword support.
-
-10. **[tests/json_integration_test.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/json_integration_test.rs)** (327 lines) - [Integration tests for JSON output functionality](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/json_integration_test.rs). Verifies that the CLI correctly integrates the JSON output formatter and produces expected JSON structure when the `--json` flag is used.
-
-11. **[tests/cli_normalization.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_normalization.rs)** (49 lines) - [Tests for CLI output normalization functionality](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_normalization.rs). Ensures cross-platform normalization helpers work correctly across different environments using snapshot testing.
+2. **[tests/cli_integration.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration.rs)** (887 lines) - [CLI integration tests using subprocess-based testing with assert_cmd](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration.rs). Provides natural process isolation and works correctly under llvm-cov for coverage reporting. Tests command-line interface functionality including multiple file processing, strict mode (`--strict`), built-in rules (`--use-builtin`), JSON output, stdin handling, error handling, timeout validation, shell completions, and custom magic files.
+
+3. **[tests/cli_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration_tests.rs)** (1,625 lines) - [CLI integration tests using the canonical libmagic test suite](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration_tests.rs). Verifies command-line interface functionality by running against third_party/tests/, covering canonical test suite compatibility and compatibility with original libmagic behavior.
+
+4. **[tests/property_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs)** (225 lines) - [Property-based tests verifying that evaluator never panics, buffer access is bounds-checked, metadata consistency, and serde roundtrips](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs). Includes [6 custom proptest strategies](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs#L18-L84): arb_offset_spec, arb_type_kind, arb_operator, arb_value, arb_magic_rule, arb_buffer.
+
+5. **[tests/evaluator_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/evaluator_tests.rs)** (248 lines) - [Evaluator integration tests through the public MagicDatabase API](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/evaluator_tests.rs). Tests confidence calculation, rule ordering, evaluation behavior, and validates that known file types return non-zero confidence scores.
+
+6. **[tests/compatibility_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility_tests.rs)** (360 lines) - [Compatibility tests ensuring identical results to the original libmagic](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility_tests.rs). [Test files are downloaded from the file/file repository](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility/README.md#L7) and compared against GNU file command.
+
+7. **[tests/parser_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/parser_integration_tests.rs)** (380 lines) - [End-to-end integration tests for magic file parser and database integration](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/parser_integration_tests.rs). Validates the complete flow from file/directory loading through rule evaluation.
+
+8. **[tests/directory_loading_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/directory_loading_tests.rs)** (431 lines) - [Integration tests for directory loading functionality](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/directory_loading_tests.rs). Validates `load_magic_directory()` function with various directory structures and batch loading of magic files from directories.
+
+9. **[tests/mime_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/mime_tests.rs)** (230 lines) - [MIME type mapping integration tests](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/mime_tests.rs). Tests MIME type detection through the public API including hardcoded mappings, prefix matching, and case insensitivity.
+
+10. **[tests/tags_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/tags_tests.rs)** (163 lines) - [Tag extraction integration tests](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/tags_tests.rs). Tests keyword extraction, case insensitivity, rule path tags, and custom keyword support.
+
+11. **[tests/json_integration_test.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/json_integration_test.rs)** (327 lines) - [Integration tests for JSON output functionality](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/json_integration_test.rs). Verifies that the CLI correctly integrates the JSON output formatter and produces expected JSON structure when the `--json` flag is used.
+
+12. **[tests/cli_normalization.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_normalization.rs)** (49 lines) - [Tests for CLI output normalization functionality](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_normalization.rs). Ensures cross-platform normalization helpers work correctly across different environments using snapshot testing.
#### Supporting Test Files
@@ -75,6 +77,29 @@
```
[The `common::normalize_cli_output()` function handles executable name conversion (rmagic.exe → rmagic), Windows path prefix removal (\\?\\), and error message filtering](https://app.dosu.dev/documents/691b7c0a-2533-474f-bdd5-11a429656050).
+
+### CLI Testing with assert_cmd
+
+The project uses subprocess-based CLI testing with assert_cmd for natural process isolation. [The tests/cli_integration.rs file](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration.rs) contains comprehensive CLI tests that run the rmagic binary as a subprocess:
+
+```rust
+use assert_cmd::Command;
+use predicates::prelude::*;
+
+#[test]
+fn test_builtin_elf_detection() {
+ let temp_dir = TempDir::new().expect("Failed to create temp dir");
+ let test_file = create_elf_file(&temp_dir);
+
+ Command::new(assert_cmd::cargo::cargo_bin!("rmagic"))
+ .args(["--use-builtin", test_file.to_str().expect("Invalid path")])
+ .assert()
+ .success()
+ .stdout(predicate::str::contains("ELF"));
+}
+```
+
+This approach provides reliable CLI testing that works correctly under llvm-cov for coverage reporting without platform-specific fd manipulation.
### Property-Based Testing with proptest
@@ -133,10 +158,12 @@
### Dev Dependencies
[Core testing dependencies in Cargo.toml](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L162-L168):
+- **[assert_cmd 2.1.2](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L164)** - Subprocess-based CLI testing with process isolation
- **[criterion 0.8.2](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L162)** - Benchmarking framework for performance measurement
- **[insta 1.46.3](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L163)** - Snapshot testing library with JSON feature enabled
+- **[nix 0.31.2](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L164)** - Unix system API bindings with fs feature for filesystem testing
+- **[predicates 3.1.4](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L167)** - Assertion predicates for command output validation
- **[proptest 1.10.0](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L166)** - Property-based testing framework
-- **[nix 0.31.2](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L164)** - Unix system API bindings with fs feature for filesystem testing
- **[regex 1.12.3](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L165)** - Regular expression support for test assertions
- **[tempfile 3.26.0](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/Cargo.toml#L167)** - Temporary file and directory creation for test fixtures
@@ -269,7 +296,8 @@
| File Path | Description | Lines |
|-----------|-------------|-------|
| [tests/integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/integration_tests.rs) | Core end-to-end integration tests | 468 |
-| [tests/cli_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration_tests.rs) | Comprehensive CLI testing with canonical libmagic test suite | 1,625 |
+| [tests/cli_integration.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration.rs) | Subprocess-based CLI integration tests using assert_cmd | 887 |
+| [tests/cli_integration_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/cli_integration_tests.rs) | Canonical libmagic test suite compatibility tests | 1,625 |
| [tests/property_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/property_tests.rs) | Property-based tests with proptest strategies | 225 |
| [tests/evaluator_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/evaluator_tests.rs) | Evaluator integration tests through public API | 248 |
| [tests/compatibility_tests.rs](https://github.com/EvilBit-Labs/libmagic-rs/blob/e8f5eb490731996b82b22ace849dd45832405cd1/tests/compatibility_tests.rs) | Compatibility validation with original libmagic | 360 |✅ Accepted testing-guidelines
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
🧪 CI InsightsHere's what we observed from your CI run for cc00082. 🟢 All jobs passed!But CI Insights is watching 👀 |
There was a problem hiding this comment.
Pull request overview
This PR extracts fragile CLI unit tests from src/main.rs into a dedicated tests/cli_integration.rs integration test file, replacing process-wide fd manipulation with assert_cmd-based subprocess testing. This addresses issue #61 which highlighted that main.rs was ~2.7x the 500-600 line guideline and used brittle dup/dup2 operations requiring a global mutex.
Changes:
- New
tests/cli_integration.rs(~887 lines) with subprocess-based CLI tests usingassert_cmd,predicates, andtempfile src/main.rsremoves fragile fd-manipulation helpers (FD_MUTEX,capture_stdout,capture_stderr,with_mocked_stdin) and several migrated test functionsCargo.tomladdsassert_cmd,predicatesas dev dependencies and bumps several dependency versions; adds unuseduserfeature to thenixdev dependencydocs/src/testing.mdupdated to document the new integration test structuremise.toml/mise.lockchanges the Rust toolchain version from"stable"to"latest"(resolved to 1.93.1)dist-workspace.tomlpurely cosmetic formatting changes
Reviewed changes
Copilot reviewed 6 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/cli_integration.rs |
New file: 60 CLI integration tests using assert_cmd for subprocess-based testing |
src/main.rs |
Removes fragile fd helpers, FD_MUTEX, and migrated test functions |
Cargo.toml |
Adds assert_cmd, predicates; adds unused user feature to nix; bumps dependency versions |
docs/src/testing.md |
Updates integration test documentation; lists only 2 of 11+ integration test files |
mise.toml |
Changes Rust version from "stable" to "latest" |
mise.lock |
Pins Rust to 1.93.1 (resolved from "latest") |
dist-workspace.toml |
Cosmetic formatting only |
Cargo.lock |
Lockfile updates for new/bumped dependencies |
| **Integration Tests**: Located in `tests/` directory: | ||
|
|
||
| - `tests/cli_integration.rs` - CLI subprocess tests using `assert_cmd` | ||
| - `tests/property_tests.rs` - Property-based tests using `proptest` |
There was a problem hiding this comment.
The "Integration Tests" bullet list in docs/src/testing.md only mentions tests/cli_integration.rs and tests/property_tests.rs, but the tests/ directory contains many other integration test files: tests/cli_integration_tests.rs, tests/compatibility_tests.rs, tests/evaluator_tests.rs, tests/integration_tests.rs, tests/json_integration_test.rs, tests/mime_tests.rs, tests/parser_integration_tests.rs, tests/tags_tests.rs, and tests/directory_loading_tests.rs. The documentation gives the impression that only two integration test files exist, which is misleading.
| rmagic_cmd() | ||
| .args([ | ||
| "--magic-file", | ||
| "nonexistent.magic", | ||
| test_file.to_str().expect("Invalid path"), | ||
| ]) | ||
| .assert() | ||
| .failure() | ||
| .code(4) | ||
| .stderr(predicate::str::contains("Magic file")); |
There was a problem hiding this comment.
The test test_error_magic_file_not_found passes "nonexistent.magic" as a relative path to --magic-file. However, a file named nonexistent.magic actually exists in the repository root (it contains valid magic rules). Since integration test subprocesses typically inherit the workspace root as their working directory, this relative path will resolve to an existing valid magic file, causing the binary to succeed (exit code 0) instead of failing with exit code 4 as the test expects.
The fix is to use a path that is guaranteed not to exist, such as a path inside a TempDir that was not created (e.g., temp_dir.path().join("nonexistent.magic")), similar to how test_multiple_files_partial_failure_non_strict constructs temp_dir.path().join("nonexistent.bin").
…view findings - Delete orphaned test infrastructure: cli_integration_tests.rs (1733 lines), cli_normalization.rs, common/mod.rs, and 6 stale snapshot files - Consolidate 6 format-specific file creator helpers into constants + create_data_file - Replace duplicate tests with table-driven patterns per AGENTS.md convention - Strengthen PDF/GIF detection assertions (assert format name or "data") - Replace from_utf8_lossy with from_utf8().expect() in JSON test - Restore 7 missing unit-level argument parsing tests to main.rs - Migrate pre-existing main.rs tests from manual cleanup to tempfile::TempDir - Remove unused nix "user" feature from Cargo.toml - Fix stale cli_integration_tests.rs references in builtin_rules.rs, Claude instincts/skills, plan docs, and testing.md - Fix testing.md example to match actual rmagic_cmd() helper pattern Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Fix test_error_magic_file_not_found using relative path that resolves to an existing file; use temp_dir path instead - Expand integration test file list in testing.md to include all test files in tests/ directory Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
| [dev-dependencies] | ||
| assert_cmd = "2.1.2" | ||
| criterion = "0.8.2" | ||
| insta = { version = "1.46.3", features = ["json"] } |
There was a problem hiding this comment.
The insta dev-dependency is still listed in Cargo.toml but is no longer used anywhere in the codebase. The tests/cli_integration_tests.rs and tests/cli_normalization.rs files that used assert_snapshot! have been deleted. The insta dependency should be removed from [dev-dependencies] to keep the build clean and avoid an unused dependency warning/audit finding.
| insta = { version = "1.46.3", features = ["json"] } |
| let assertion = rmagic_cmd() | ||
| .args(["--use-builtin", "-"]) | ||
| .write_stdin(*input) | ||
| .assert() | ||
| .success() | ||
| .stdout(predicate::str::contains("stdin:")); | ||
|
|
||
| if let Some(substr) = expected_substr { | ||
| assertion.stdout(predicate::str::contains(*substr)); | ||
| } | ||
|
|
||
| // Satisfy the borrow checker - label is used for debugging context | ||
| let _ = label; |
There was a problem hiding this comment.
The label variable in test_stdin_format_detection is bound in the destructuring pattern but immediately discarded with let _ = label; and a comment saying it is "used for debugging context". However, it is never actually passed to any assertion message. If a test case fails, there is no indication of which of the four cases triggered the failure, making debugging harder. The label should be included in the assertion message using .with_context(|| label.to_string()) or by refactoring into a helper that accepts a label and forwards it to the assertion, or alternatively by splitting this into separate named test functions.
| let assertion = rmagic_cmd() | |
| .args(["--use-builtin", "-"]) | |
| .write_stdin(*input) | |
| .assert() | |
| .success() | |
| .stdout(predicate::str::contains("stdin:")); | |
| if let Some(substr) = expected_substr { | |
| assertion.stdout(predicate::str::contains(*substr)); | |
| } | |
| // Satisfy the borrow checker - label is used for debugging context | |
| let _ = label; | |
| let output = rmagic_cmd() | |
| .args(["--use-builtin", "-"]) | |
| .write_stdin(*input) | |
| .output() | |
| .expect("failed to run rmagic"); | |
| assert!( | |
| output.status.success(), | |
| "stdin case `{label}` failed: process exited with status {:?}", | |
| output.status.code() | |
| ); | |
| let stdout = String::from_utf8_lossy(&output.stdout); | |
| assert!( | |
| stdout.contains("stdin:"), | |
| "stdin case `{label}`: expected stdout to contain \"stdin:\", got: {stdout:?}" | |
| ); | |
| if let Some(substr) = expected_substr { | |
| assert!( | |
| stdout.contains(substr), | |
| "stdin case `{label}`: expected stdout to contain {substr:?}, got: {stdout:?}" | |
| ); | |
| } |
…support find_rmagic_binary() only searched target/release/ and target/debug/, but cargo llvm-cov builds to target/llvm-cov-target/. Use assert_cmd's cargo_bin! macro as the primary lookup to find the binary regardless of target directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
## 🤖 New release * `libmagic-rs`: 0.4.2 -> 0.4.3 (✓ API compatible changes) <details><summary><i><b>Changelog</b></i></summary><p> <blockquote> ## [0.4.3] - 2026-03-07 ### Refactor - **tests**: Extract CLI tests from main.rs into integration tests ([#159](#159)) <!-- generated by git-cliff --> </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary
src/main.rs(~430 lines removed) into a dedicatedtests/cli_integration.rsintegration test file (~887 lines)assert_cmdcrate for proper subprocess-based CLI testing instead of calling internal functions directlyassert_cmd,predicates, andtempfileas dev dependencies for robust test infrastructuredocs/src/testing.mdto reflect the new integration test structureTest Plan
cargo test)Closes #61
🤖 Generated with Claude Code