Skip to content

12 add magic file format detection subtask 143#25

Merged
unclesp1d3r merged 16 commits into
mainfrom
12-add-magic-file-format-detection-subtask-143
Jan 23, 2026
Merged

12 add magic file format detection subtask 143#25
unclesp1d3r merged 16 commits into
mainfrom
12-add-magic-file-format-detection-subtask-143

Conversation

@unclesp1d3r
Copy link
Copy Markdown
Member

This pull request introduces major documentation and configuration updates to support the Phase 1 MVP of libmagic-rs. The changes focus on contributor onboarding, project planning, CLI enhancements, and build-time support for built-in magic rules. These improvements clarify project architecture, establish development standards, and outline the technical approach for key upcoming features.

Documentation and Contributor Experience:

  • Added a comprehensive CONTRIBUTING.md guide detailing code standards, testing, documentation, branching, and error handling for all contributors.
  • Created an epic brief (docs/plan/phase_1_mvp/specs/Epic_Brief__libmagic-rs_Phase_1_MVP.md) outlining the goals, context, scope, and value proposition for the Phase 1 MVP, including compatibility, safety, and documentation requirements.

Technical Planning for Core Features:

  • Added a technical specification for build-time compilation of built-in magic rules (docs/plan/phase_1_mvp/tickets/Built-in_Rules__Build-Time_Compilation_&_Fallback.md), describing the use of a build script, fallback rules, and new APIs for rule loading.
  • Documented CLI enhancements (docs/plan/phase_1_mvp/tickets/CLI_Enhancements__Multiple_Files,_Stdin,_Magic_Discovery.md) including support for multiple files, stdin, improved magic file discovery, new flags (--use-builtin, --strict), and robust error/output handling. (docs/plan/phase_1_mvp/tickets/CLI_Enhancements__Multiple_Files,_Stdin,_Magic_Discovery.mdR1-R241)

Development Environment Improvements:

  • Updated .vscode/settings.json to configure ruff and Python interpreter paths for consistent linting and formatting in the development environment.

Miscellaneous:

  • Minor cleanup in dist-workspace.toml for release configuration.

These changes lay the groundwork for the next development phase, ensuring clarity, maintainability, and a strong foundation for new features and contributor engagement.

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Add apprentice_magic_strength algorithm port from libmagic
- Implement strength modifier parsing with !:strength syntax
- Calculate default strength based on rule specificity
- Store strength in MagicRule structure for evaluation
- Update rule sorting by strength during evaluation
- Add unit tests for strength parsing and calculation
- Integrate strength into confidence scoring and rule ordering

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
@unclesp1d3r unclesp1d3r linked an issue Jan 23, 2026 that may be closed by this pull request
@unclesp1d3r unclesp1d3r self-assigned this Jan 23, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jan 23, 2026

Caution

Review failed

Failed to post review comments

Summary by CodeRabbit

  • Documentation

    • Added comprehensive contributing guidelines and project architecture documentation
    • Expanded getting started guide with examples and public API usage patterns
    • Updated project status to "Active Development" with new capability overviews
    • Enhanced architecture documentation with visual diagrams
  • Chores

    • Updated Rust toolchain to 1.91.0 and bumped dependencies (clap, memmap2, serde_json, insta, tempfile, etc.)
    • Refactored GitHub Actions workflows to use centralized tooling setup via mise-action
    • Added mise.toml tool configuration
    • Reorganized justfile to consolidate build commands

✏️ Tip: You can customize this high-level summary in your review settings.

Walkthrough

Adds extensive documentation and planning for a Phase 1 MVP, a new CONTRIBUTING guide, tooling/config updates (VSCode, mise.toml, rust-toolchain, Cargo), CI/workflow consolidation to jdx/mise-action@v3, justfile reorganizations, and minor repo infra edits. Mostly docs and infra/config changes; no broad API-breaking code changes reported.

Changes

Cohort / File(s) Summary
Editor & Tooling settings
\.vscode/settings.json, mise.toml, rust-toolchain.toml
Adds VSCode settings for ruff and Python interpreter under .vscode/mise-tools/...; introduces mise.toml tool pins; bumps toolchain to 1.91.0 (formatting only on components list).
Contribution guide
CONTRIBUTING.md
Adds a new, comprehensive contributing guide (CoC, setup, development workflow, testing, PR template, Rust style/safety guidance).
Phase 1 MVP specs & tickets
docs/plan/phase_1_mvp/specs/*, docs/plan/phase_1_mvp/tickets/*
Adds many new design/spec and ticket documents covering Epic, Technical Plan, Core Flows, built-in rules, CLI enhancements, parser integration, evaluation enhancements (confidence/MIME/tags/metadata), strength calculation, testing infra, and documentation plans.
Documentation & research
docs/src/*, docs/research/*, docs/book.toml, docs/plan/*
Large doc additions and revisions: new research and architecture docs, Mermaid diagrams replacing ASCII art, expanded Getting Started/Introduction and examples, mdBook config reflow and preprocessors adjusted.
CI / GitHub workflows
.github/workflows/*
(e.g., \.github/workflows/audit.yml, ci.yml, codeql.yml, compatibility.yml, docs.yml, security.yml, copilot-setup-steps.yml)
Replaces actions/checkout v5→v6 and consolidates per-job Rust/tool setup into jdx/mise-action@v3 (install/cache/github_token), centralizing installs and caching across workflows; some rust-toolchain/setup steps removed.
Parser / Evaluation API (planning notes)
docs/plan/...
Documents proposed public API additions and behavior (e.g., MagicDatabase builder methods, EvaluationResult/Metadata/MatchResult, MimeMapper, TagExtractor, MagicFileFormat/detection, builtin rules build-time flow). Changes appear as design/spec text rather than implemented code in this diff.
Build / repo automation
justfile, .pre-commit-config.yaml, .gitignore, Cargo.toml
Consolidates justfile recipes into private wrappers and adjusts public entries, updates pre-commit hooks, adds .gitignore entry undefined/, and bumps various dependencies in Cargo.toml.
mdBook/site config
docs/book.toml
Reflows and aligns mdBook keys, removes or disables some preprocessors, and adjusts HTML output config formatting (cosmetic).
Small infra files
.serena/project.yml, .serena/.gitignore, dist-workspace.toml
Adds Serenа project config and .serena ignore, minor whitespace/formatting tweaks in dist-workspace.toml.

Sequence Diagram(s)

(omitted — changes are primarily documentation and infra; no new multi-component control flow added that requires sequence visualization)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I hop through specs and tidy up the lair,
Docs stacked like carrots, workflows trimmed with care,
CONTRIBUTING stitched, tools pinned in neat rows,
Tickets sketch paths where future logic grows,
A small rabbit cheers — MVP seeds sown fair.

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 inconclusive)
Check name Status Explanation Resolution
Title check ❓ Inconclusive The title '12 add magic file format detection subtask 143' is partially related but overly broad and vague. It references a ticket/subtask number rather than clearly describing the main change. Revise the title to clearly describe the primary change, such as 'Add Phase 1 MVP planning and contributing documentation' or 'Add CONTRIBUTING guide and Phase 1 MVP specifications.' Avoid vague issue references as the primary description.
✅ Passed checks (2 passed)
Check name Status Explanation
Description check ✅ Passed The description is comprehensive and directly related to the changeset, providing clear context for documentation updates, CLI enhancements, and configuration changes that align with the files and objectives listed.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch 12-add-magic-file-format-detection-subtask-143

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot added enhancement New feature or request evaluator Rule evaluation engine and logic testing Test infrastructure and coverage labels Jan 23, 2026
unclesp1d3r and others added 6 commits January 22, 2026 19:30
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
chore(mise): update Rust and add new tools for formatting and auditing

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Move Rust components (llvm-tools-preview) to mise.toml
- Add mdformat with plugins via mise pipx backend
- DRY up GoReleaser and coverage recipes with private helpers
- Remove unused recipes (cd-root, help, test-justfile, test-fs)
- Simplify format-docs using mdformat's built-in --exclude
- Update dependencies

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@coderabbitai coderabbitai Bot added the output Result formatting and output generation label Jan 23, 2026
unclesp1d3r and others added 4 commits January 22, 2026 20:29
- Ensure 'quality' job is a dependency for 'test'
- Ensure 'test' job is a dependency for 'test-cross-platform'

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
No docs use YAML front matter, so this preprocessor is unnecessary
and was causing CI issues.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
@unclesp1d3r unclesp1d3r enabled auto-merge (squash) January 23, 2026 01:49
@unclesp1d3r unclesp1d3r disabled auto-merge January 23, 2026 02:10
@unclesp1d3r unclesp1d3r merged commit 81b2ee1 into main Jan 23, 2026
18 of 19 checks passed
@unclesp1d3r unclesp1d3r deleted the 12-add-magic-file-format-detection-subtask-143 branch January 23, 2026 02:10
unclesp1d3r added a commit that referenced this pull request Jan 23, 2026
* feat(docs): add contribution guidelines and update architecture diagrams

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* style(toml): format spacing and alignment in dist-workspace.toml

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(vscode): update ruff configuration paths in settings

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(parser): implement libmagic strength calculation and parsing

- Add apprentice_magic_strength algorithm port from libmagic
- Implement strength modifier parsing with !:strength syntax
- Calculate default strength based on rule specificity
- Store strength in MagicRule structure for evaluation
- Update rule sorting by strength during evaluation
- Add unit tests for strength parsing and calculation
- Integrate strength into confidence scoring and rule ordering

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(docs): update documentation for phase 1 MVP enhancements

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* style(docs): format spacing and alignment in book.toml

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(tools): add tools configuration for project dependencies

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: upgrade checkout action to v6 and add mise-action

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(ci): replace rustfmt action with just fmt-check command

chore(mise): update Rust and add new tools for formatting and auditing

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* refactor(justfile): streamline cross-platform helper commands

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore: simplify justfile and mise.toml tooling

- Move Rust components (llvm-tools-preview) to mise.toml
- Add mdformat with plugins via mise pipx backend
- DRY up GoReleaser and coverage recipes with private helpers
- Remove unused recipes (cd-root, help, test-justfile, test-fs)
- Simplify format-docs using mdformat's built-in --exclude
- Update dependencies

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ci: update job dependencies for test and coverage workflows

- Ensure 'quality' job is a dependency for 'test'
- Ensure 'test' job is a dependency for 'test-cross-platform'

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(deps): update Rust version to 1.91.0 in configuration

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* fix: remove unused mdbook-yml-header preprocessor

No docs use YAML front matter, so this preprocessor is unnecessary
and was causing CI issues.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add .gitignore and project.yml for configuration

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(docs): update repository URL in documentation files

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

---------

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
unclesp1d3r added a commit that referenced this pull request Jan 24, 2026
…er support (#26)

* chore(settings): enable collapse walkthrough in reviews

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(deps): update dependabot configuration for clarity

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(devcontainer): add initial configuration for Rust development

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* 12 add magic file format detection subtask 143 (#25)

* feat(docs): add contribution guidelines and update architecture diagrams

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* style(toml): format spacing and alignment in dist-workspace.toml

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(vscode): update ruff configuration paths in settings

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(parser): implement libmagic strength calculation and parsing

- Add apprentice_magic_strength algorithm port from libmagic
- Implement strength modifier parsing with !:strength syntax
- Calculate default strength based on rule specificity
- Store strength in MagicRule structure for evaluation
- Update rule sorting by strength during evaluation
- Add unit tests for strength parsing and calculation
- Integrate strength into confidence scoring and rule ordering

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(docs): update documentation for phase 1 MVP enhancements

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* style(docs): format spacing and alignment in book.toml

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(tools): add tools configuration for project dependencies

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: upgrade checkout action to v6 and add mise-action

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(ci): replace rustfmt action with just fmt-check command

chore(mise): update Rust and add new tools for formatting and auditing

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* refactor(justfile): streamline cross-platform helper commands

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore: simplify justfile and mise.toml tooling

- Move Rust components (llvm-tools-preview) to mise.toml
- Add mdformat with plugins via mise pipx backend
- DRY up GoReleaser and coverage recipes with private helpers
- Remove unused recipes (cd-root, help, test-justfile, test-fs)
- Simplify format-docs using mdformat's built-in --exclude
- Update dependencies

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ci: update job dependencies for test and coverage workflows

- Ensure 'quality' job is a dependency for 'test'
- Ensure 'test' job is a dependency for 'test-cross-platform'

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(deps): update Rust version to 1.91.0 in configuration

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* fix: remove unused mdbook-yml-header preprocessor

No docs use YAML front matter, so this preprocessor is unnecessary
and was causing CI issues.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add .gitignore and project.yml for configuration

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(docs): update repository URL in documentation files

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

---------

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* feat(parser): enhance magic file parsing and format detection

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* feat(parser): add magic file format detection support

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* docs(agents): add guidelines for emoji usage in code and docs

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* test(cli): add snapshot for canonical CLI test failures

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* docs: update commit message style guidelines for clarity and consistency

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* docs(readme): update project status and feature overview

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci(release): update GitHub Actions to use latest versions of actions

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: replace Rustfmt check command with action

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: add rustfmt and clippy components to toolchain setup

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: update Rust toolchain components for CI workflow

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* style(mise): remove extra space in rust components list

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(deps): remove default component from Rust toolchain configuration

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: add dependency conditions for quality checks

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* chore(ci): disable compatibility tests in workflow

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* fix(evaluator): replace panic with Result in decrement_recursion_depth

Convert `decrement_recursion_depth()` from a panicking function to one
that returns `Result<(), LibmagicError>`. This addresses the critical
issue where a programming error in recursion tracking would crash the
entire application instead of returning a recoverable error.

Changes:
- Add `InternalError` variant to `EvaluationError` for internal logic errors
- Add `internal_error()` constructor method
- Update `decrement_recursion_depth()` to return `Result`
- Update all call sites to use `?` operator or `.unwrap()` in tests
- Convert `#[should_panic]` test to check for error result instead

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(docs): correct documentation inaccuracies and improve error handling

Address high-priority issues from PR review:

Documentation fixes:
- Remove non-existent `start_time` field from EvaluationContext docs
- Fix MatchResult value type: `Value` not `Vec<u8>`
- Update security claims in load_magic_file() to be accurate
- Remove outdated "placeholder" warnings from getting-started.md
- Fix test count claims: 650+ tests (was inconsistent 745+/200+)

Error handling improvements:
- Return error when ALL files in directory fail to parse
- Track parse failures and provide detailed error messages
- Add tests for all-files-fail and partial-failure scenarios

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(ci): use cargo fmt directly instead of rustfmt action

The actions-rust-lang/rustfmt@v1 action uses its own rustfmt binary
which doesn't properly respect the mise-installed toolchain or the
project's rustfmt.toml configuration (edition = "2024").

Replace with a direct `cargo fmt --check` command which uses the
mise-installed rust toolchain consistently.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore(deps): add cargo-fmt to rust components in mise.toml

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: replace cargo fmt and clippy commands with just lint-rust

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* fix(ci): simplify job dependencies for test and coverage

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: disable caching for mise-action in CI workflow

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* ci: update rust-toolchain components and enable caching

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

* fix(ci): correct rust-toolchain components syntax in CI config

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>

---------

Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request evaluator Rule evaluation engine and logic output Result formatting and output generation testing Test infrastructure and coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add magic file format detection (subtask 14.3)

1 participant