Built-in Rules: Build-Time Compilation & Fallback

[unclesp1d3r]

Built-in Rules: Build-Time Compilation & Fallback

Overview

Implement build-time compilation of built-in magic rules using build.rs, providing fallback rules for common file types (ELF, PE, ZIP, JPEG, PNG, PDF, GIF) that work without external magic files. This enables --use-builtin and --create-magic functionality.

Scope

In Scope:

• Create file:src/builtin_rules.magic with common file type patterns
• Build script (build.rs) to parse and compile rules at build time
• Build-time validation with clear error messages
• MagicDatabase::with_builtin_rules() API

Out of Scope:

• Comprehensive magic rules (only common types)
• Runtime rule compilation
• Dynamic rule loading

Technical Approach

1. Magic File Source

Create file:src/builtin_rules.magic:

# Built-in fallback rules for common file types

# ELF executables
0       string  \x7fELF         ELF
>4      byte    1               32-bit
>4      byte    2               64-bit
>5      byte    1               LSB
>5      byte    2               MSB

# PE executables
0       string  MZ              DOS/Windows executable
>0x3c   lelong  <0x40000000
>>0x3c  lelong  >0
>>>0x3c lelong  x               PE

# ZIP archives
0       string  PK\003\004      ZIP archive

# JPEG images
0       string  \xff\xd8\xff    JPEG image

# PNG images
0       string  \x89PNG\r\n\x1a\n   PNG image

# PDF documents
0       string  %PDF-           PDF document

# GIF images
0       string  GIF8            GIF image

2. Build Script

Create file:build.rs:

use std::fs;
use std::path::Path;

fn main() {
    let magic_text = include_str!("src/builtin_rules.magic");

    // Parse at build time
    match parse_magic_text(magic_text) {
        Ok(rules) => {
            // Generate Rust code with AST structures
            let code = generate_builtin_rules_code(&rules);
            let out_dir = std::env::var("OUT_DIR").unwrap();
            let dest_path = Path::new(&out_dir).join("builtin_rules.rs");
            fs::write(&dest_path, code).unwrap();
        }
        Err(e) => {
            eprintln!("ERROR: Failed to parse built-in magic rules");
            eprintln!("File: src/builtin_rules.magic");
            eprintln!("Error: {}", e);
            eprintln!("\nBuilt-in rules must be valid magic file syntax.");
            eprintln!("Please fix the syntax errors and rebuild.");
            std::process::exit(1);
        }
    }
}

3. Built-in Rules Module

Create file:src/builtin_rules.rs:

include!(concat!(env!("OUT_DIR"), "/builtin_rules.rs"));

pub fn get_builtin_rules() -> Vec<MagicRule> {
    BUILTIN_RULES.clone()
}

4. MagicDatabase API

Add to file:src/lib.rs:

impl MagicDatabase {
    pub fn with_builtin_rules() -> Self {
        Self {
            rules: builtin_rules::get_builtin_rules(),
            config: EvaluationConfig::default(),
        }
    }
}

Acceptance Criteria

• builtin_rules.magic contains rules for common file types
• build.rs parses magic file at build time
• Build fails with clear error if magic file is invalid
• Generated code compiles without warnings
• MagicDatabase::with_builtin_rules() returns working database
• Built-in rules correctly identify ELF, PE, ZIP, JPEG, PNG, PDF, GIF
• --use-builtin flag works end-to-end
• Rustdoc added for all public APIs
• Unit tests for built-in rules
• Integration test with --use-builtin
• Build script test (verify error handling)

Dependencies

None - can be implemented independently

Related Specs

• spec:75a688c2-0ac4-489a-a35d-6e824c94c153/269e848a-258d-4cd4-99b1-386bd400a109 (Technical Plan - Built-in Rules Module)
• spec:75a688c2-0ac4-489a-a35d-6e824c94c153/36539700-862d-4fdf-9c79-3c36390f6aa8 (Core Flows - Flow 3)

Files to Create

• file:src/builtin_rules.magic - Magic file source
• file:build.rs - Build script
• file:src/builtin_rules.rs - Module for built-in rules

Files to Modify

• file:src/lib.rs - Add with_builtin_rules() method
• file:Cargo.toml - Add build script configuration if needed

[traycerai]

Plan

Observations

The codebase has a well-structured parser that can convert magic files to MagicRule AST with full serde support. Currently, MagicDatabase::load_from_file() is a stub returning empty rules. The project uses comprehensive error handling and all AST structures are serializable. The parser module provides parse_text_magic_file() for parsing magic file syntax at runtime.

Approach

Implement build-time compilation of built-in magic rules for common file types (ELF, PE, ZIP, JPEG, PNG, PDF, GIF) using a build.rs script that parses a source magic file during compilation and generates Rust code containing the compiled rules. This provides fallback rules that work without external magic files while keeping the implementation focused and maintainable.

Implementation Steps

1. Create Built-in Magic File Source

Create file:src/builtin_rules.magic with common file type patterns:

File Content:

• ELF executables (32-bit, 64-bit, LSB, MSB variants)
• PE executables (DOS/Windows)
• ZIP archives
• JPEG images
• PNG images
• PDF documents
• GIF images

Format:

• Use standard magic file syntax with offset, type, and test value
• Include descriptive output strings for each file type
• Add comments explaining each rule group
• Keep rules focused on common file types only (out of scope: comprehensive rules)

2. Create Build Script

Create file:build.rs in the project root:

Parse Magic File at Build Time:

• Use std::fs::read_to_string() to read src/builtin_rules.magic
• Call libmagic_rs::parser::parse_text_magic_file() to parse the magic file into Vec<MagicRule>
• Implement error handling that prints clear error messages to stderr:
  • "ERROR: Failed to parse built-in magic rules"
  • "File: src/builtin_rules.magic"
  • Actual error details
  • Guidance: "Built-in rules must be valid magic file syntax. Please fix the syntax errors and rebuild."
• Call std::process::exit(1) on parse failure to fail the build

Generate Rust Code:

• Create a function generate_builtin_rules_code(rules: &[MagicRule]) -> String that generates Rust code
• Output format: pub const BUILTIN_RULES: &[MagicRule] = &[...]; with serialized rule data
• Write generated code to $OUT_DIR/builtin_rules.rs
• Add println!("cargo:rerun-if-changed=src/builtin_rules.magic"); to rebuild when source changes

3. Create Built-in Rules Module

Create file:src/builtin_rules.rs:

Module Structure:

• Include the generated code: include!(concat!(env!("OUT_DIR"), "/builtin_rules.rs"));
• Export a public function pub fn get_builtin_rules() -> Vec<MagicRule> that returns BUILTIN_RULES.to_vec()
• Add module-level documentation explaining the purpose and contents

4. Add MagicDatabase API

Update file:src/lib.rs in the MagicDatabase implementation:

New Constructor:

• Add pub fn with_builtin_rules() -> Self that:
  • Calls builtin_rules::get_builtin_rules() to get the rules
  • Returns a MagicDatabase instance with those rules
  • Uses EvaluationConfig::default() for configuration
  • Include doc comments explaining this is for built-in fallback rules

Module Declaration:

• Add mod builtin_rules; to make the module available
• Ensure the module is properly integrated into the library structure

5. Update CLI Integration

Modify file:src/main.rs to support --use-builtin flag:

Add CLI Flag:

• Add --use-builtin boolean flag to the argument parser
• Document the flag: "Use built-in fallback rules instead of system magic files"

Update Magic File Loading Logic:

• In the file loading section, check if --use-builtin is set
• If true, call MagicDatabase::with_builtin_rules() to use built-in rules
• If false, use existing file loading logic (system files or --magic-file override)
• Ensure error messages clearly indicate which rules are being used

6. Add Documentation

Rustdoc Comments:

• Add comprehensive doc comments to MagicDatabase::with_builtin_rules()
• Document the builtin_rules module with examples
• Include examples showing how to use built-in rules

Update README:

• Document the --use-builtin flag and when to use it
• Explain that built-in rules provide fallback for common file types
• Note that custom magic files can still be used with --magic-file

7. Testing Strategy

Unit Tests:

• Add tests in file:src/lib.rs for MagicDatabase::with_builtin_rules()
• Verify that the returned database contains rules
• Test that built-in rules correctly identify each supported file type (ELF, PE, ZIP, JPEG, PNG, PDF, GIF)

Build Script Tests:

• Create file:tests/build_script.rs to verify:
  • Build script successfully parses src/builtin_rules.magic
  • Generated code compiles without warnings
  • BUILTIN_RULES constant is accessible and non-empty

Integration Tests:

• Create file:tests/builtin_rules_integration.rs to test:
  • CLI works with --use-builtin flag
  • Built-in rules correctly identify test files for each supported type
  • Error handling when magic file is invalid (build-time)

Error Handling Tests:

• Verify build fails with clear error message if src/builtin_rules.magic has syntax errors
• Test that error message guides users to fix the syntax

8. Acceptance Criteria Verification

Checklist:

• builtin_rules.magic contains rules for ELF, PE, ZIP, JPEG, PNG, PDF, GIF
• build.rs parses magic file at build time using parse_text_magic_file()
• Build fails with clear error message if magic file is invalid
• Generated code in $OUT_DIR/builtin_rules.rs compiles without warnings
• MagicDatabase::with_builtin_rules() returns working database
• Built-in rules correctly identify all supported file types
• --use-builtin CLI flag works end-to-end
• Rustdoc added for with_builtin_rules() and builtin_rules module
• Unit tests verify rule functionality
• Integration test with --use-builtin flag passes
• Build script error handling test verifies error messages

Sequence Diagram

sequenceDiagram
    participant Build as build.rs
    participant Parser as Magic Parser
    participant CodeGen as Code Generator
    participant Binary as Compiled Binary
    participant Runtime as MagicDatabase
    participant User as User Code

    Note over Build,CodeGen: Build Time
    Build->>Build: Read src/builtin_rules.magic
    Build->>Parser: parse_text_magic_file(content)
    alt Parse Success
        Parser-->>Build: Vec<MagicRule>
        Build->>CodeGen: generate_builtin_rules_code(rules)
        CodeGen-->>Build: Rust code string
        Build->>Build: Write to OUT_DIR/builtin_rules.rs
        Build->>Binary: Include generated code
    else Parse Error
        Parser-->>Build: ParseError
        Build->>Build: Print error to stderr
        Build->>Build: exit(1) - Build fails
    end

    Note over Runtime,User: Runtime
    User->>Runtime: MagicDatabase::with_builtin_rules()
    Runtime->>Runtime: include!(builtin_rules.rs)
    Runtime->>Runtime: get_builtin_rules()
    Runtime-->>User: MagicDatabase with built-in rules

    alt CLI with --use-builtin
        User->>Runtime: with_builtin_rules()
        Runtime-->>User: Database ready
    else CLI with --magic-file
        User->>Runtime: load_from_file(path)
        Runtime-->>User: Database with custom rules
    end

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured parser that can convert magic files to `MagicRule` AST with full serde support. Currently, `MagicDatabase::load_from_file()` is a stub returning empty rules. The project uses comprehensive error handling and all AST structures are serializable. The parser module provides `parse_text_magic_file()` for parsing magic file syntax at runtime.

## Approach

Implement build-time compilation of built-in magic rules for common file types (ELF, PE, ZIP, JPEG, PNG, PDF, GIF) using a `build.rs` script that parses a source magic file during compilation and generates Rust code containing the compiled rules. This provides fallback rules that work without external magic files while keeping the implementation focused and maintainable.

## Implementation Steps

### 1. Create Built-in Magic File Source

Create `file:src/builtin_rules.magic` with common file type patterns:

**File Content:**
- ELF executables (32-bit, 64-bit, LSB, MSB variants)
- PE executables (DOS/Windows)
- ZIP archives
- JPEG images
- PNG images
- PDF documents
- GIF images

**Format:**
- Use standard magic file syntax with offset, type, and test value
- Include descriptive output strings for each file type
- Add comments explaining each rule group
- Keep rules focused on common file types only (out of scope: comprehensive rules)

### 2. Create Build Script

Create `file:build.rs` in the project root:

**Parse Magic File at Build Time:**
- Use `std::fs::read_to_string()` to read `src/builtin_rules.magic`
- Call `libmagic_rs::parser::parse_text_magic_file()` to parse the magic file into `Vec<MagicRule>`
- Implement error handling that prints clear error messages to stderr:
  - "ERROR: Failed to parse built-in magic rules"
  - "File: src/builtin_rules.magic"
  - Actual error details
  - Guidance: "Built-in rules must be valid magic file syntax. Please fix the syntax errors and rebuild."
- Call `std::process::exit(1)` on parse failure to fail the build

**Generate Rust Code:**
- Create a function `generate_builtin_rules_code(rules: &[MagicRule]) -> String` that generates Rust code
- Output format: `pub const BUILTIN_RULES: &[MagicRule] = &[...];` with serialized rule data
- Write generated code to `$OUT_DIR/builtin_rules.rs`
- Add `println!("cargo:rerun-if-changed=src/builtin_rules.magic");` to rebuild when source changes

### 3. Create Built-in Rules Module

Create `file:src/builtin_rules.rs`:

**Module Structure:**
- Include the generated code: `include!(concat!(env!("OUT_DIR"), "/builtin_rules.rs"));`
- Export a public function `pub fn get_builtin_rules() -> Vec<MagicRule>` that returns `BUILTIN_RULES.to_vec()`
- Add module-level documentation explaining the purpose and contents

### 4. Add MagicDatabase API

Update `file:src/lib.rs` in the `MagicDatabase` implementation:

**New Constructor:**
- Add `pub fn with_builtin_rules() -> Self` that:
  - Calls `builtin_rules::get_builtin_rules()` to get the rules
  - Returns a `MagicDatabase` instance with those rules
  - Uses `EvaluationConfig::default()` for configuration
  - Include doc comments explaining this is for built-in fallback rules

**Module Declaration:**
- Add `mod builtin_rules;` to make the module available
- Ensure the module is properly integrated into the library structure

### 5. Update CLI Integration

Modify `file:src/main.rs` to support `--use-builtin` flag:

**Add CLI Flag:**
- Add `--use-builtin` boolean flag to the argument parser
- Document the flag: "Use built-in fallback rules instead of system magic files"

**Update Magic File Loading Logic:**
- In the file loading section, check if `--use-builtin` is set
- If true, call `MagicDatabase::with_builtin_rules()` to use built-in rules
- If false, use existing file loading logic (system files or `--magic-file` override)
- Ensure error messages clearly indicate which rules are being used

### 6. Add Documentation

**Rustdoc Comments:**
- Add comprehensive doc comments to `MagicDatabase::with_builtin_rules()`
- Document the `builtin_rules` module with examples
- Include examples showing how to use built-in rules

**Update README:**
- Document the `--use-builtin` flag and when to use it
- Explain that built-in rules provide fallback for common file types
- Note that custom magic files can still be used with `--magic-file`

### 7. Testing Strategy

**Unit Tests:**
- Add tests in `file:src/lib.rs` for `MagicDatabase::with_builtin_rules()`
- Verify that the returned database contains rules
- Test that built-in rules correctly identify each supported file type (ELF, PE, ZIP, JPEG, PNG, PDF, GIF)

**Build Script Tests:**
- Create `file:tests/build_script.rs` to verify:
  - Build script successfully parses `src/builtin_rules.magic`
  - Generated code compiles without warnings
  - `BUILTIN_RULES` constant is accessible and non-empty

**Integration Tests:**
- Create `file:tests/builtin_rules_integration.rs` to test:
  - CLI works with `--use-builtin` flag
  - Built-in rules correctly identify test files for each supported type
  - Error handling when magic file is invalid (build-time)

**Error Handling Tests:**
- Verify build fails with clear error message if `src/builtin_rules.magic` has syntax errors
- Test that error message guides users to fix the syntax

### 8. Acceptance Criteria Verification

**Checklist:**
- [ ] `builtin_rules.magic` contains rules for ELF, PE, ZIP, JPEG, PNG, PDF, GIF
- [ ] `build.rs` parses magic file at build time using `parse_text_magic_file()`
- [ ] Build fails with clear error message if magic file is invalid
- [ ] Generated code in `$OUT_DIR/builtin_rules.rs` compiles without warnings
- [ ] `MagicDatabase::with_builtin_rules()` returns working database
- [ ] Built-in rules correctly identify all supported file types
- [ ] `--use-builtin` CLI flag works end-to-end
- [ ] Rustdoc added for `with_builtin_rules()` and `builtin_rules` module
- [ ] Unit tests verify rule functionality
- [ ] Integration test with `--use-builtin` flag passes
- [ ] Build script error handling test verifies error messages

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Build as build.rs
    participant Parser as Magic Parser
    participant CodeGen as Code Generator
    participant Binary as Compiled Binary
    participant Runtime as MagicDatabase
    participant User as User Code

    Note over Build,CodeGen: Build Time
    Build->>Build: Read src/builtin_rules.magic
    Build->>Parser: parse_text_magic_file(content)
    alt Parse Success
        Parser-->>Build: Vec<MagicRule>
        Build->>CodeGen: generate_builtin_rules_code(rules)
        CodeGen-->>Build: Rust code string
        Build->>Build: Write to OUT_DIR/builtin_rules.rs
        Build->>Binary: Include generated code
    else Parse Error
        Parser-->>Build: ParseError
        Build->>Build: Print error to stderr
        Build->>Build: exit(1) - Build fails
    end

    Note over Runtime,User: Runtime
    User->>Runtime: MagicDatabase::with_builtin_rules()
    Runtime->>Runtime: include!(builtin_rules.rs)
    Runtime->>Runtime: get_builtin_rules()
    Runtime-->>User: MagicDatabase with built-in rules

    alt CLI with --use-builtin
        User->>Runtime: with_builtin_rules()
        Runtime-->>User: Database ready
    else CLI with --magic-file
        User->>Runtime: load_from_file(path)
        Runtime-->>User: Database with custom rules
    end
```

---

Execution Information

Branch: main
Commit: 50b370c

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

Previous Plan

Observations

The codebase has a well-structured parser that can convert magic files to MagicRule AST with full serde support. Currently, MagicDatabase::load_from_file() is a stub returning empty rules. The CLI attempts to locate system magic files with fallback to third_party/magic.mgc. The project uses memmap2 for I/O and has comprehensive error handling. All AST structures are serializable, making them suitable for compile-time embedding.

Approach

Implement build-time compilation of magic rules by creating a build.rs script that parses magic files during compilation, serializes them using bincode, and embeds the binary data into the executable. Modify MagicDatabase to support loading from both embedded rules (as primary) and external files (as override), providing a self-contained binary that doesn't require system magic files while still allowing customization.

Implementation Steps

1. Add Build Dependencies

Update file:Cargo.toml to include build dependencies:

• Add [build-dependencies] section with bincode = "1.3" for efficient binary serialization
• Ensure serde is available in build dependencies for serialization during build
• The existing serde and serde_json dependencies are already suitable for runtime deserialization

2. Create Build Script

Create file:build.rs in the project root:

Parse Magic Files at Build Time:

• Use std::fs::read_to_string() to read third_party/magic.mgc or text magic files
• Import and use libmagic_rs::parser::parse_text_magic_file() to parse magic content into Vec<MagicRule>
• Handle both text .magic files and binary .mgc files (skip binary for now, focus on text)
• Implement error handling for missing or malformed magic files with helpful build-time error messages

Serialize Parsed Rules:

• Use bincode::serialize() to convert Vec<MagicRule> to binary format
• Write serialized data to $OUT_DIR/embedded_rules.bin
• Generate a Rust source file $OUT_DIR/embedded_rules.rs containing:
  • pub const EMBEDDED_RULES_DATA: &[u8] = include_bytes!("embedded_rules.bin");
  • Metadata constants like EMBEDDED_RULES_COUNT, EMBEDDED_RULES_VERSION

Build Configuration:

• Add println!("cargo:rerun-if-changed=third_party/"); to rebuild when magic files change
• Consider adding a feature flag embedded-rules (enabled by default) to control this behavior
• Print build-time statistics (number of rules compiled, file size)

3. Modify MagicDatabase Structure

Update file:src/lib.rs in the MagicDatabase implementation:

Add Embedded Rules Support:

• Include the generated module: include!(concat!(env!("OUT_DIR"), "/embedded_rules.rs"));
• Add a new constructor MagicDatabase::from_embedded() -> Result<Self> that:
  • Deserializes EMBEDDED_RULES_DATA using bincode::deserialize()
  • Returns a MagicDatabase with the embedded rules
  • Handles deserialization errors gracefully

Implement Fallback Logic:

• Modify load_from_file() to actually parse the file using parser::parse_text_magic_file()
• Add MagicDatabase::new() -> Result<Self> as the primary constructor that:
  • First attempts to load from embedded rules using from_embedded()
  • Returns the embedded database if successful
  • Falls back to error if embedded rules fail (since external file loading is separate)

Add Override Mechanism:

• Create MagicDatabase::with_custom_file<P: AsRef<Path>>(path: P) -> Result<Self> that:
  • Loads rules from the specified file path
  • Overrides embedded rules completely
  • Useful for testing or custom magic databases

4. Update CLI Integration

Modify file:src/main.rs to use the new loading mechanism:

Update Magic File Loading:

• In run_analysis(), change the loading logic:
  • If --magic-file is specified, use MagicDatabase::with_custom_file(path)
  • Otherwise, use MagicDatabase::from_embedded() to use built-in rules
  • Remove the complex platform-specific file search logic from default_magic_file_path() or make it optional
  • Keep the fallback creation logic for CI/CD environments as a last resort

Improve Error Messages:

• Update error messages to indicate whether embedded or external rules are being used
• Add a --version or --info flag that shows embedded rules metadata (count, version)

5. Add Feature Flag Support

Update file:Cargo.toml to add optional feature:

Define Feature:

• Add [features] section with default = ["embedded-rules"]
• Add embedded-rules = [] feature definition
• This allows users to disable embedded rules with --no-default-features

Conditional Compilation:

• In file:build.rs, use #[cfg(feature = "embedded-rules")] to conditionally compile rules
• In file:src/lib.rs, use #[cfg(feature = "embedded-rules")] around embedded rules code
• Provide a fallback implementation when feature is disabled that requires external files

6. Update Documentation

Update README:

• Document that the binary includes built-in magic rules
• Explain how to use custom magic files with --magic-file
• Note the embedded-rules feature flag for custom builds

Add Architecture Documentation:

• Update file:docs/src/architecture.md to include build-time compilation in the data flow
• Add a section explaining the embedded rules mechanism
• Document the fallback hierarchy: embedded → custom file → error

Code Documentation:

• Add comprehensive doc comments to MagicDatabase::from_embedded()
• Document the build script behavior in file:build.rs
• Add examples showing both embedded and custom file usage

7. Testing Strategy

Build Script Tests:

• Create file:tests/build_integration.rs to verify embedded rules are accessible
• Test that EMBEDDED_RULES_DATA is non-empty
• Verify deserialization succeeds

Unit Tests:

• Add tests in file:src/lib.rs for MagicDatabase::from_embedded()
• Test fallback behavior when embedded rules are available
• Test custom file override functionality

Integration Tests:

• Verify CLI works without any external magic files
• Test that --magic-file properly overrides embedded rules
• Ensure error messages are appropriate for each scenario

8. Migration Path

Backward Compatibility:

• Existing code using MagicDatabase::load_from_file() continues to work
• The new from_embedded() method is additive, not breaking
• CLI behavior improves but remains compatible

Deployment Benefits:

• Binary becomes self-contained and portable
• No dependency on system magic files
• Faster startup (no file I/O for default case)
• Consistent behavior across platforms

Sequence Diagram

sequenceDiagram
    participant Build as build.rs
    participant Parser as Magic Parser
    participant Bincode as Bincode
    participant Binary as Compiled Binary
    participant Runtime as MagicDatabase
    participant User as User Code

    Note over Build,Bincode: Build Time
    Build->>Build: Read third_party/*.magic
    Build->>Parser: parse_text_magic_file(content)
    Parser-->>Build: Vec<MagicRule>
    Build->>Bincode: serialize(rules)
    Bincode-->>Build: Binary data
    Build->>Build: Write to OUT_DIR/embedded_rules.bin
    Build->>Build: Generate embedded_rules.rs
    Build->>Binary: Include embedded data

    Note over Runtime,User: Runtime
    User->>Runtime: MagicDatabase::from_embedded()
    Runtime->>Runtime: include!(embedded_rules.rs)
    Runtime->>Bincode: deserialize(EMBEDDED_RULES_DATA)
    Bincode-->>Runtime: Vec<MagicRule>
    Runtime-->>User: MagicDatabase with rules

    alt Custom Magic File
        User->>Runtime: with_custom_file(path)
        Runtime->>Parser: parse_text_magic_file(file)
        Parser-->>Runtime: Vec<MagicRule>
        Runtime-->>User: MagicDatabase with custom rules
    end

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured parser that can convert magic files to `MagicRule` AST with full serde support. Currently, `MagicDatabase::load_from_file()` is a stub returning empty rules. The CLI attempts to locate system magic files with fallback to `third_party/magic.mgc`. The project uses `memmap2` for I/O and has comprehensive error handling. All AST structures are serializable, making them suitable for compile-time embedding.

## Approach

Implement build-time compilation of magic rules by creating a `build.rs` script that parses magic files during compilation, serializes them using `bincode`, and embeds the binary data into the executable. Modify `MagicDatabase` to support loading from both embedded rules (as primary) and external files (as override), providing a self-contained binary that doesn't require system magic files while still allowing customization.

## Implementation Steps

### 1. Add Build Dependencies

Update `file:Cargo.toml` to include build dependencies:

- Add `[build-dependencies]` section with `bincode = "1.3"` for efficient binary serialization
- Ensure `serde` is available in build dependencies for serialization during build
- The existing `serde` and `serde_json` dependencies are already suitable for runtime deserialization

### 2. Create Build Script

Create `file:build.rs` in the project root:

**Parse Magic Files at Build Time:**
- Use `std::fs::read_to_string()` to read `third_party/magic.mgc` or text magic files
- Import and use `libmagic_rs::parser::parse_text_magic_file()` to parse magic content into `Vec<MagicRule>`
- Handle both text `.magic` files and binary `.mgc` files (skip binary for now, focus on text)
- Implement error handling for missing or malformed magic files with helpful build-time error messages

**Serialize Parsed Rules:**
- Use `bincode::serialize()` to convert `Vec<MagicRule>` to binary format
- Write serialized data to `$OUT_DIR/embedded_rules.bin`
- Generate a Rust source file `$OUT_DIR/embedded_rules.rs` containing:
  - `pub const EMBEDDED_RULES_DATA: &[u8] = include_bytes!("embedded_rules.bin");`
  - Metadata constants like `EMBEDDED_RULES_COUNT`, `EMBEDDED_RULES_VERSION`

**Build Configuration:**
- Add `println!("cargo:rerun-if-changed=third_party/");` to rebuild when magic files change
- Consider adding a feature flag `embedded-rules` (enabled by default) to control this behavior
- Print build-time statistics (number of rules compiled, file size)

### 3. Modify MagicDatabase Structure

Update `file:src/lib.rs` in the `MagicDatabase` implementation:

**Add Embedded Rules Support:**
- Include the generated module: `include!(concat!(env!("OUT_DIR"), "/embedded_rules.rs"));`
- Add a new constructor `MagicDatabase::from_embedded() -> Result<Self>` that:
  - Deserializes `EMBEDDED_RULES_DATA` using `bincode::deserialize()`
  - Returns a `MagicDatabase` with the embedded rules
  - Handles deserialization errors gracefully

**Implement Fallback Logic:**
- Modify `load_from_file()` to actually parse the file using `parser::parse_text_magic_file()`
- Add `MagicDatabase::new() -> Result<Self>` as the primary constructor that:
  - First attempts to load from embedded rules using `from_embedded()`
  - Returns the embedded database if successful
  - Falls back to error if embedded rules fail (since external file loading is separate)

**Add Override Mechanism:**
- Create `MagicDatabase::with_custom_file<P: AsRef<Path>>(path: P) -> Result<Self>` that:
  - Loads rules from the specified file path
  - Overrides embedded rules completely
  - Useful for testing or custom magic databases

### 4. Update CLI Integration

Modify `file:src/main.rs` to use the new loading mechanism:

**Update Magic File Loading:**
- In `run_analysis()`, change the loading logic:
  - If `--magic-file` is specified, use `MagicDatabase::with_custom_file(path)`
  - Otherwise, use `MagicDatabase::from_embedded()` to use built-in rules
  - Remove the complex platform-specific file search logic from `default_magic_file_path()` or make it optional
  - Keep the fallback creation logic for CI/CD environments as a last resort

**Improve Error Messages:**
- Update error messages to indicate whether embedded or external rules are being used
- Add a `--version` or `--info` flag that shows embedded rules metadata (count, version)

### 5. Add Feature Flag Support

Update `file:Cargo.toml` to add optional feature:

**Define Feature:**
- Add `[features]` section with `default = ["embedded-rules"]`
- Add `embedded-rules = []` feature definition
- This allows users to disable embedded rules with `--no-default-features`

**Conditional Compilation:**
- In `file:build.rs`, use `#[cfg(feature = "embedded-rules")]` to conditionally compile rules
- In `file:src/lib.rs`, use `#[cfg(feature = "embedded-rules")]` around embedded rules code
- Provide a fallback implementation when feature is disabled that requires external files

### 6. Update Documentation

**Update README:**
- Document that the binary includes built-in magic rules
- Explain how to use custom magic files with `--magic-file`
- Note the `embedded-rules` feature flag for custom builds

**Add Architecture Documentation:**
- Update `file:docs/src/architecture.md` to include build-time compilation in the data flow
- Add a section explaining the embedded rules mechanism
- Document the fallback hierarchy: embedded → custom file → error

**Code Documentation:**
- Add comprehensive doc comments to `MagicDatabase::from_embedded()`
- Document the build script behavior in `file:build.rs`
- Add examples showing both embedded and custom file usage

### 7. Testing Strategy

**Build Script Tests:**
- Create `file:tests/build_integration.rs` to verify embedded rules are accessible
- Test that `EMBEDDED_RULES_DATA` is non-empty
- Verify deserialization succeeds

**Unit Tests:**
- Add tests in `file:src/lib.rs` for `MagicDatabase::from_embedded()`
- Test fallback behavior when embedded rules are available
- Test custom file override functionality

**Integration Tests:**
- Verify CLI works without any external magic files
- Test that `--magic-file` properly overrides embedded rules
- Ensure error messages are appropriate for each scenario

### 8. Migration Path

**Backward Compatibility:**
- Existing code using `MagicDatabase::load_from_file()` continues to work
- The new `from_embedded()` method is additive, not breaking
- CLI behavior improves but remains compatible

**Deployment Benefits:**
- Binary becomes self-contained and portable
- No dependency on system magic files
- Faster startup (no file I/O for default case)
- Consistent behavior across platforms

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Build as build.rs
    participant Parser as Magic Parser
    participant Bincode as Bincode
    participant Binary as Compiled Binary
    participant Runtime as MagicDatabase
    participant User as User Code

    Note over Build,Bincode: Build Time
    Build->>Build: Read third_party/*.magic
    Build->>Parser: parse_text_magic_file(content)
    Parser-->>Build: Vec<MagicRule>
    Build->>Bincode: serialize(rules)
    Bincode-->>Build: Binary data
    Build->>Build: Write to OUT_DIR/embedded_rules.bin
    Build->>Build: Generate embedded_rules.rs
    Build->>Binary: Include embedded data

    Note over Runtime,User: Runtime
    User->>Runtime: MagicDatabase::from_embedded()
    Runtime->>Runtime: include!(embedded_rules.rs)
    Runtime->>Bincode: deserialize(EMBEDDED_RULES_DATA)
    Bincode-->>Runtime: Vec<MagicRule>
    Runtime-->>User: MagicDatabase with rules

    alt Custom Magic File
        User->>Runtime: with_custom_file(path)
        Runtime->>Parser: parse_text_magic_file(file)
        Parser-->>Runtime: Vec<MagicRule>
        Runtime-->>User: MagicDatabase with custom rules
    end
```

---

Execution Information

Branch: main
Commit: 50b370c

New Plan

Observations

The codebase has a well-structured parser that can convert magic files to MagicRule AST with full serde support. Currently, MagicDatabase::load_from_file() is a stub returning empty rules. The project uses comprehensive error handling and all AST structures are serializable. The parser module provides parse_text_magic_file() for parsing magic file syntax at runtime.

Approach

Implement build-time compilation of built-in magic rules for common file types (ELF, PE, ZIP, JPEG, PNG, PDF, GIF) using a build.rs script that parses a source magic file during compilation and generates Rust code containing the compiled rules. This provides fallback rules that work without external magic files while keeping the implementation focused and maintainable.

Implementation Steps

1. Create Built-in Magic File Source

Create file:src/builtin_rules.magic with common file type patterns:

File Content:

• ELF executables (32-bit, 64-bit, LSB, MSB variants)
• PE executables (DOS/Windows)
• ZIP archives
• JPEG images
• PNG images
• PDF documents
• GIF images

Format:

• Use standard magic file syntax with offset, type, and test value
• Include descriptive output strings for each file type
• Add comments explaining each rule group
• Keep rules focused on common file types only (out of scope: comprehensive rules)

2. Create Build Script

Create file:build.rs in the project root:

Parse Magic File at Build Time:

• Use std::fs::read_to_string() to read src/builtin_rules.magic
• Call libmagic_rs::parser::parse_text_magic_file() to parse the magic file into Vec<MagicRule>
• Implement error handling that prints clear error messages to stderr:
  • "ERROR: Failed to parse built-in magic rules"
  • "File: src/builtin_rules.magic"
  • Actual error details
  • Guidance: "Built-in rules must be valid magic file syntax. Please fix the syntax errors and rebuild."
• Call std::process::exit(1) on parse failure to fail the build

Generate Rust Code:

• Create a function generate_builtin_rules_code(rules: &[MagicRule]) -> String that generates Rust code
• Output format: pub const BUILTIN_RULES: &[MagicRule] = &[...]; with serialized rule data
• Write generated code to $OUT_DIR/builtin_rules.rs
• Add println!("cargo:rerun-if-changed=src/builtin_rules.magic"); to rebuild when source changes

3. Create Built-in Rules Module

Create file:src/builtin_rules.rs:

Module Structure:

• Include the generated code: include!(concat!(env!("OUT_DIR"), "/builtin_rules.rs"));
• Export a public function pub fn get_builtin_rules() -> Vec<MagicRule> that returns BUILTIN_RULES.to_vec()
• Add module-level documentation explaining the purpose and contents

4. Add MagicDatabase API

Update file:src/lib.rs in the MagicDatabase implementation:

New Constructor:

• Add pub fn with_builtin_rules() -> Self that:
  • Calls builtin_rules::get_builtin_rules() to get the rules
  • Returns a MagicDatabase instance with those rules
  • Uses EvaluationConfig::default() for configuration
  • Include doc comments explaining this is for built-in fallback rules

Module Declaration:

• Add mod builtin_rules; to make the module available
• Ensure the module is properly integrated into the library structure

5. Update CLI Integration

Modify file:src/main.rs to support --use-builtin flag:

Add CLI Flag:

• Add --use-builtin boolean flag to the argument parser
• Document the flag: "Use built-in fallback rules instead of system magic files"

Update Magic File Loading Logic:

• In the file loading section, check if --use-builtin is set
• If true, call MagicDatabase::with_builtin_rules() to use built-in rules
• If false, use existing file loading logic (system files or --magic-file override)
• Ensure error messages clearly indicate which rules are being used

6. Add Documentation

Rustdoc Comments:

• Add comprehensive doc comments to MagicDatabase::with_builtin_rules()
• Document the builtin_rules module with examples
• Include examples showing how to use built-in rules

Update README:

• Document the --use-builtin flag and when to use it
• Explain that built-in rules provide fallback for common file types
• Note that custom magic files can still be used with --magic-file

7. Testing Strategy

Unit Tests:

• Add tests in file:src/lib.rs for MagicDatabase::with_builtin_rules()
• Verify that the returned database contains rules
• Test that built-in rules correctly identify each supported file type (ELF, PE, ZIP, JPEG, PNG, PDF, GIF)

Build Script Tests:

• Create file:tests/build_script.rs to verify:
  • Build script successfully parses src/builtin_rules.magic
  • Generated code compiles without warnings
  • BUILTIN_RULES constant is accessible and non-empty

Integration Tests:

• Create file:tests/builtin_rules_integration.rs to test:
  • CLI works with --use-builtin flag
  • Built-in rules correctly identify test files for each supported type
  • Error handling when magic file is invalid (build-time)

Error Handling Tests:

• Verify build fails with clear error message if src/builtin_rules.magic has syntax errors
• Test that error message guides users to fix the syntax

8. Acceptance Criteria Verification

Checklist:

• builtin_rules.magic contains rules for ELF, PE, ZIP, JPEG, PNG, PDF, GIF
• build.rs parses magic file at build time using parse_text_magic_file()
• Build fails with clear error message if magic file is invalid
• Generated code in $OUT_DIR/builtin_rules.rs compiles without warnings
• MagicDatabase::with_builtin_rules() returns working database
• Built-in rules correctly identify all supported file types
• --use-builtin CLI flag works end-to-end
• Rustdoc added for with_builtin_rules() and builtin_rules module
• Unit tests verify rule functionality
• Integration test with --use-builtin flag passes
• Build script error handling test verifies error messages

Sequence Diagram

sequenceDiagram
    participant Build as build.rs
    participant Parser as Magic Parser
    participant CodeGen as Code Generator
    participant Binary as Compiled Binary
    participant Runtime as MagicDatabase
    participant User as User Code

    Note over Build,CodeGen: Build Time
    Build->>Build: Read src/builtin_rules.magic
    Build->>Parser: parse_text_magic_file(content)
    alt Parse Success
        Parser-->>Build: Vec<MagicRule>
        Build->>CodeGen: generate_builtin_rules_code(rules)
        CodeGen-->>Build: Rust code string
        Build->>Build: Write to OUT_DIR/builtin_rules.rs
        Build->>Binary: Include generated code
    else Parse Error
        Parser-->>Build: ParseError
        Build->>Build: Print error to stderr
        Build->>Build: exit(1) - Build fails
    end

    Note over Runtime,User: Runtime
    User->>Runtime: MagicDatabase::with_builtin_rules()
    Runtime->>Runtime: include!(builtin_rules.rs)
    Runtime->>Runtime: get_builtin_rules()
    Runtime-->>User: MagicDatabase with built-in rules

    alt CLI with --use-builtin
        User->>Runtime: with_builtin_rules()
        Runtime-->>User: Database ready
    else CLI with --magic-file
        User->>Runtime: load_from_file(path)
        Runtime-->>User: Database with custom rules
    end

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured parser that can convert magic files to `MagicRule` AST with full serde support. Currently, `MagicDatabase::load_from_file()` is a stub returning empty rules. The project uses comprehensive error handling and all AST structures are serializable. The parser module provides `parse_text_magic_file()` for parsing magic file syntax at runtime.

## Approach

Implement build-time compilation of built-in magic rules for common file types (ELF, PE, ZIP, JPEG, PNG, PDF, GIF) using a `build.rs` script that parses a source magic file during compilation and generates Rust code containing the compiled rules. This provides fallback rules that work without external magic files while keeping the implementation focused and maintainable.

## Implementation Steps

### 1. Create Built-in Magic File Source

Create `file:src/builtin_rules.magic` with common file type patterns:

**File Content:**
- ELF executables (32-bit, 64-bit, LSB, MSB variants)
- PE executables (DOS/Windows)
- ZIP archives
- JPEG images
- PNG images
- PDF documents
- GIF images

**Format:**
- Use standard magic file syntax with offset, type, and test value
- Include descriptive output strings for each file type
- Add comments explaining each rule group
- Keep rules focused on common file types only (out of scope: comprehensive rules)

### 2. Create Build Script

Create `file:build.rs` in the project root:

**Parse Magic File at Build Time:**
- Use `std::fs::read_to_string()` to read `src/builtin_rules.magic`
- Call `libmagic_rs::parser::parse_text_magic_file()` to parse the magic file into `Vec<MagicRule>`
- Implement error handling that prints clear error messages to stderr:
  - "ERROR: Failed to parse built-in magic rules"
  - "File: src/builtin_rules.magic"
  - Actual error details
  - Guidance: "Built-in rules must be valid magic file syntax. Please fix the syntax errors and rebuild."
- Call `std::process::exit(1)` on parse failure to fail the build

**Generate Rust Code:**
- Create a function `generate_builtin_rules_code(rules: &[MagicRule]) -> String` that generates Rust code
- Output format: `pub const BUILTIN_RULES: &[MagicRule] = &[...];` with serialized rule data
- Write generated code to `$OUT_DIR/builtin_rules.rs`
- Add `println!("cargo:rerun-if-changed=src/builtin_rules.magic");` to rebuild when source changes

### 3. Create Built-in Rules Module

Create `file:src/builtin_rules.rs`:

**Module Structure:**
- Include the generated code: `include!(concat!(env!("OUT_DIR"), "/builtin_rules.rs"));`
- Export a public function `pub fn get_builtin_rules() -> Vec<MagicRule>` that returns `BUILTIN_RULES.to_vec()`
- Add module-level documentation explaining the purpose and contents

### 4. Add MagicDatabase API

Update `file:src/lib.rs` in the `MagicDatabase` implementation:

**New Constructor:**
- Add `pub fn with_builtin_rules() -> Self` that:
  - Calls `builtin_rules::get_builtin_rules()` to get the rules
  - Returns a `MagicDatabase` instance with those rules
  - Uses `EvaluationConfig::default()` for configuration
  - Include doc comments explaining this is for built-in fallback rules

**Module Declaration:**
- Add `mod builtin_rules;` to make the module available
- Ensure the module is properly integrated into the library structure

### 5. Update CLI Integration

Modify `file:src/main.rs` to support `--use-builtin` flag:

**Add CLI Flag:**
- Add `--use-builtin` boolean flag to the argument parser
- Document the flag: "Use built-in fallback rules instead of system magic files"

**Update Magic File Loading Logic:**
- In the file loading section, check if `--use-builtin` is set
- If true, call `MagicDatabase::with_builtin_rules()` to use built-in rules
- If false, use existing file loading logic (system files or `--magic-file` override)
- Ensure error messages clearly indicate which rules are being used

### 6. Add Documentation

**Rustdoc Comments:**
- Add comprehensive doc comments to `MagicDatabase::with_builtin_rules()`
- Document the `builtin_rules` module with examples
- Include examples showing how to use built-in rules

**Update README:**
- Document the `--use-builtin` flag and when to use it
- Explain that built-in rules provide fallback for common file types
- Note that custom magic files can still be used with `--magic-file`

### 7. Testing Strategy

**Unit Tests:**
- Add tests in `file:src/lib.rs` for `MagicDatabase::with_builtin_rules()`
- Verify that the returned database contains rules
- Test that built-in rules correctly identify each supported file type (ELF, PE, ZIP, JPEG, PNG, PDF, GIF)

**Build Script Tests:**
- Create `file:tests/build_script.rs` to verify:
  - Build script successfully parses `src/builtin_rules.magic`
  - Generated code compiles without warnings
  - `BUILTIN_RULES` constant is accessible and non-empty

**Integration Tests:**
- Create `file:tests/builtin_rules_integration.rs` to test:
  - CLI works with `--use-builtin` flag
  - Built-in rules correctly identify test files for each supported type
  - Error handling when magic file is invalid (build-time)

**Error Handling Tests:**
- Verify build fails with clear error message if `src/builtin_rules.magic` has syntax errors
- Test that error message guides users to fix the syntax

### 8. Acceptance Criteria Verification

**Checklist:**
- [ ] `builtin_rules.magic` contains rules for ELF, PE, ZIP, JPEG, PNG, PDF, GIF
- [ ] `build.rs` parses magic file at build time using `parse_text_magic_file()`
- [ ] Build fails with clear error message if magic file is invalid
- [ ] Generated code in `$OUT_DIR/builtin_rules.rs` compiles without warnings
- [ ] `MagicDatabase::with_builtin_rules()` returns working database
- [ ] Built-in rules correctly identify all supported file types
- [ ] `--use-builtin` CLI flag works end-to-end
- [ ] Rustdoc added for `with_builtin_rules()` and `builtin_rules` module
- [ ] Unit tests verify rule functionality
- [ ] Integration test with `--use-builtin` flag passes
- [ ] Build script error handling test verifies error messages

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Build as build.rs
    participant Parser as Magic Parser
    participant CodeGen as Code Generator
    participant Binary as Compiled Binary
    participant Runtime as MagicDatabase
    participant User as User Code

    Note over Build,CodeGen: Build Time
    Build->>Build: Read src/builtin_rules.magic
    Build->>Parser: parse_text_magic_file(content)
    alt Parse Success
        Parser-->>Build: Vec<MagicRule>
        Build->>CodeGen: generate_builtin_rules_code(rules)
        CodeGen-->>Build: Rust code string
        Build->>Build: Write to OUT_DIR/builtin_rules.rs
        Build->>Binary: Include generated code
    else Parse Error
        Parser-->>Build: ParseError
        Build->>Build: Print error to stderr
        Build->>Build: exit(1) - Build fails
    end

    Note over Runtime,User: Runtime
    User->>Runtime: MagicDatabase::with_builtin_rules()
    Runtime->>Runtime: include!(builtin_rules.rs)
    Runtime->>Runtime: get_builtin_rules()
    Runtime-->>User: MagicDatabase with built-in rules

    alt CLI with --use-builtin
        User->>Runtime: with_builtin_rules()
        Runtime-->>User: Database ready
    else CLI with --magic-file
        User->>Runtime: load_from_file(path)
        Runtime-->>User: Database with custom rules
    end
```

---

Execution Information

Branch: main
Commit: 50b370c

[unclesp1d3r]

@coderabbitai Assign the proper labels and other metadata to this issue

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# Get current issue details
echo "=== Current Issue Details ==="
gh issue view 19 --repo EvilBit-Labs/libmagic-rs --json title,body,labels,assignees,state,milestone

echo -e "\n=== Available Labels in Repository ==="
gh label list --repo EvilBit-Labs/libmagic-rs --limit 100

echo -e "\n=== Repository Milestones ==="
gh api repos/EvilBit-Labs/libmagic-rs/milestones --jq '.[] | "\(.number): \(.title) (\(.state))"'

Length of output: 5585

---

🏁 Script executed:

#!/bin/bash

# Add labels to issue `#19`
echo "=== Adding Labels to Issue `#19` ==="
gh issue edit 19 --repo EvilBit-Labs/libmagic-rs \
  --add-label "enhancement" \
  --add-label "parser" \
  --add-label "cli" \
  --add-label "rust"

# Assign issue to unclesp1d3r
echo -e "\n=== Assigning Issue to unclesp1d3r ==="
gh issue edit 19 --repo EvilBit-Labs/libmagic-rs \
  --add-assignee unclesp1d3r

# Verify the changes
echo -e "\n=== Updated Issue Details ==="
gh issue view 19 --repo EvilBit-Labs/libmagic-rs --json title,labels,assignees --jq '{title: .title, labels: [.labels[].name], assignees: [.assignees[].login]}'

Length of output: 355

---

@unclesp1d3r I've successfully updated issue #19 with the following metadata:

Labels Added:

• enhancement - This is a new feature request
• parser - Involves parsing magic files at build time using parse_text_magic_file()
• cli - Includes CLI integration with the --use-builtin flag
• rust - Uses Rust-specific features like build.rs and compile-time code generation

Assignee:

• Assigned to @unclesp1d3r

The issue is now properly categorized and ready for implementation. The labels reflect the key areas this feature touches: parsing, CLI functionality, and Rust build system integration.

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata. Examine the Epic (#24) and evaluate which, if any, issue is this associated with. Use docs/plan/**/*.md for reference.

[coderabbitai]

Coding Plan

Approach

Implement build-time compilation of built-in magic rules that provides fallback file type detection for common formats (ELF, PE, ZIP, JPEG, PNG, PDF, GIF) without requiring external magic files. This replaces the current stub implementation of --use-builtin with functional rules.

Observations

The project is a Rust implementation of libmagic file type detection. The parser module (src/parser/) provides comprehensive magic file parsing using nom combinators, producing MagicRule AST structures. MagicDatabase in src/lib.rs already has with_builtin_rules() and with_builtin_rules_and_config() methods that currently return empty rule sets. The --use-builtin CLI flag exists in src/main.rs but uses this stub implementation. All AST types (MagicRule, OffsetSpec, TypeKind, Operator, Value, Endianness) have Clone, Debug, Serialize, and Deserialize derives. No build.rs or [build-dependencies] section currently exists.

Assumptions

Assumption 1: Parser implementation approach for build.rs

Options Considered:

1. Include parser source files via #[path = ...] attributes and add nom as a build dependency - requires managing transitive dependencies and risks breakage when parser changes
2. Extract parser into a separate crate usable by both library and build script - adds significant complexity for a simple use case
3. Write a minimal, dedicated parser in build.rs that handles only the subset of magic syntax used in the builtin rules file

Chosen Option: 3

Rationale: A dedicated minimal parser in build.rs is self-contained, easier to audit, avoids build dependency complexity, and only needs to handle the simple syntax used in the controlled builtin rules file (absolute offsets, string/byte/lelong types, escape sequences, hierarchy markers).

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Build Infrastructure

This phase establishes the build-time compilation pipeline: creating the source magic file, configuring Cargo for build scripts, and implementing the build script with parsing and code generation.

Task 1: Create Built-in Magic Rules Source File

Define the magic rules for common file types in a dedicated magic file that will be parsed at build time.

• Create src/builtin_rules.magic with rules for: ELF (with 32/64-bit and endianness children), PE/MZ executables, ZIP archives, JPEG images, PNG images, PDF documents, GIF images
• Use standard magic file syntax: comments with #, level markers (>, >>) for hierarchy, absolute offsets (decimal and hex), types (string, byte, lelong), escape sequences (\x7f, \x89, etc.)
• Keep rules minimal and well-documented with inline comments explaining each file signature

Task 2: Configure Build Script in Cargo.toml

Add the necessary build script configuration to enable build-time rule compilation.

• Add build = "build.rs" to the [package] section in Cargo.toml
• No additional build dependencies are required since we're writing a self-contained minimal parser

Task 3: Implement Build Script with Minimal Parser

Create the build script that parses the magic file and generates Rust code at build time.

• Create build.rs at the project root
• Implement a minimal parser that handles only the syntax used in builtin_rules.magic: comment lines, empty lines, absolute offsets (decimal/hex), level markers (>, >>), supported types (string, byte, lelong), escape sequences in values, and message extraction
• Implement code generation functions that produce valid Rust code constructing MagicRule, OffsetSpec, TypeKind, Operator, Value, and Endianness structures
• Generate a function builtin_rules() -> Vec<MagicRule> that constructs the parsed rules
• Write generated code to $OUT_DIR/builtin_rules_generated.rs
• Implement clear error handling that prints the file name, line number, and error description, then exits with code 1 on parse failure
• Add cargo:rerun-if-changed=src/builtin_rules.magic for incremental build support

🤖 Prompt for AI agents

This phase establishes the build-time compilation pipeline for built-in magic
rules. The goal is to create the source magic file with common file type
signatures, configure Cargo to run a build script, and implement a minimal
parser in build.rs that generates Rust code at build time.

**Task 1: Create `src/builtin_rules.magic`**
- Create a new file `src/builtin_rules.magic` containing magic rules for common
file types
- Include rules for: ELF (with 32/64-bit and endianness children), PE/MZ
executables, ZIP archives, JPEG images, PNG images, PDF documents, GIF images
- Use standard magic file syntax:
  - Comments with `#`
  - Level markers (`>`, `>>`) for hierarchical rules
  - Absolute offsets in decimal and hex format
  - Types: `string`, `byte`, `lelong`
  - Escape sequences like `\x7f`, `\x89`
- Add inline comments explaining each file signature
- Keep rules minimal and focused on common formats

**Task 2: Configure `Cargo.toml`**
- Add `build = "build.rs"` to the `[package]` section
- No build dependencies needed (self-contained parser)

**Task 3: Create `build.rs`**
- Create `build.rs` at the project root
- Implement minimal parser functions that handle:
  - Comment lines (starting with `#`)
  - Empty lines
  - Absolute offsets (decimal and hex formats)
  - Level markers (`>`, `>>`) for rule hierarchy
  - Supported types: `string`, `byte`, `lelong`
  - Escape sequences in values
  - Message extraction
- Implement code generation functions that:
- Produce Rust code constructing `MagicRule`, `OffsetSpec`, `TypeKind`,
`Operator`, `Value`, and `Endianness` structures
- Generate a function `builtin_rules() -> Vec<MagicRule>` that returns the
parsed rules
- Read `src/builtin_rules.magic`, parse it, and generate code
- Write output to `$OUT_DIR/builtin_rules_generated.rs`
- Error handling:
  - Print file name, line number, and error description on parse failure
  - Exit with code 1 on error
- Add `cargo:rerun-if-changed=src/builtin_rules.magic` for incremental builds

Phase 2: Library Integration

This phase connects the generated rules to the library's public API, replacing the stub implementation with functional built-in rules.

Task 1: Create Built-in Rules Module

Create the module that exposes the generated rules to the rest of the library.

• Create src/builtin_rules.rs that includes the generated code via include!(concat!(env!("OUT_DIR"), "/builtin_rules_generated.rs"))
• Export a public get_builtin_rules() -> Vec<MagicRule> function that calls the generated function
• Add module documentation explaining that rules are compiled at build time from builtin_rules.magic

Task 2: Update MagicDatabase API

Modify the existing stub implementations to use the actual built-in rules.

• In src/lib.rs, add mod builtin_rules; declaration
• Update MagicDatabase::with_builtin_rules() to call builtin_rules::get_builtin_rules() instead of returning Vec::new()
• Update MagicDatabase::with_builtin_rules_and_config() similarly
• Add Rustdoc to both methods explaining they provide fallback rules for common file types and listing the supported formats
• Ensure the source_path field is set to None to indicate built-in rules

🤖 Prompt for AI agents

This phase connects the build-time generated rules to the library's public API.
The goal is to create a module that includes the generated code and update the
`MagicDatabase` stub methods to use the actual built-in rules.

**Task 1: Create `src/builtin_rules.rs`**
- Create a new file `src/builtin_rules.rs`
- Include the generated code using `include!(concat!(env!("OUT_DIR"),
"/builtin_rules_generated.rs"))`
- Export a public function `get_builtin_rules() -> Vec<MagicRule>` that calls
the generated `builtin_rules()` function
- Add module-level documentation:
  - Explain that rules are compiled at build time from `builtin_rules.magic`
  - Mention the supported file formats

**Task 2: Update `src/lib.rs`**
- Add `mod builtin_rules;` declaration
- Update `MagicDatabase::with_builtin_rules()` method:
  - Replace the current stub implementation (returning `Vec::new()`)
  - Call `builtin_rules::get_builtin_rules()` to get the actual rules
  - Ensure `source_path` field is set to `None`
- Update `MagicDatabase::with_builtin_rules_and_config()` method similarly:
  - Replace stub with call to `builtin_rules::get_builtin_rules()`
  - Ensure `source_path` field is set to `None`
- Add or update Rustdoc for both methods:
  - Explain they provide fallback rules for common file types
  - List supported formats: ELF, PE, ZIP, JPEG, PNG, PDF, GIF

Phase 3: Testing and Documentation

This phase validates the implementation through comprehensive tests and ensures proper documentation.

Task 1: Unit Tests for Built-in Rules

Add unit tests verifying the built-in rules are correctly parsed and structured.

• In src/builtin_rules.rs, add tests verifying get_builtin_rules() returns a non-empty vector
• Test that expected top-level rules exist (ELF, MZ, ZIP, JPEG, PNG, PDF, GIF)
• Test that ELF rule has expected child rules for architecture detection
• Test rule field values (offsets, types, operators, values) match expected patterns

Task 2: Integration Tests for File Detection

Add integration tests that verify built-in rules correctly identify test files.

• Create tests in tests/builtin_rules_tests.rs
• Test MagicDatabase::with_builtin_rules() correctly identifies: ELF binary (use existing test file helpers), ZIP archive, JPEG image, PNG image, PDF document, GIF image
• Test that unknown files return "data" as fallback
• Use the existing test file creation helpers (create_elf_test_file, create_zip_test_file, etc.) from the test utilities

Task 3: CLI Integration Test for --use-builtin

Add integration test verifying the --use-builtin flag works end-to-end.

• In tests/cli_integration_tests.rs, add test that invokes the CLI with --use-builtin flag
• Test file identification output for known file types
• Verify the flag takes precedence over --magic-file as documented

Task 4: Build Script Error Handling Test

Document and verify build script error behavior.

• Add a comment in build.rs explaining how to test error handling (temporarily introduce syntax error in magic file)
• Ensure error messages include: "ERROR: Failed to parse built-in magic rules", file path, line number, specific error description, and suggestion to fix syntax

🤖 Prompt for AI agents

This phase validates the implementation through comprehensive testing. The goal
is to add unit tests for the generated rules structure, integration tests for
file detection, CLI tests for the `--use-builtin` flag, and document error
handling behavior.

**Task 1: Add unit tests in `src/builtin_rules.rs`**
- Add a `#[cfg(test)]` module
- Test that `get_builtin_rules()` returns a non-empty vector
- Test that expected top-level rules exist for: ELF, MZ, ZIP, JPEG, PNG, PDF,
GIF
- Test that the ELF rule has expected child rules for architecture detection
- Test specific rule field values:
  - Offsets match expected values
  - Types (TypeKind) are correct
  - Operators are as expected
  - Values match the file signatures

**Task 2: Create `tests/builtin_rules_tests.rs`**
- Create integration tests for file detection using built-in rules
- Test `MagicDatabase::with_builtin_rules()` correctly identifies:
  - ELF binary (use existing test file helper)
  - ZIP archive
  - JPEG image
  - PNG image
  - PDF document
  - GIF image
- Test that unknown/unrecognized files return "data" as fallback
- Use existing test file creation helpers from test utilities:
`create_elf_test_file`, `create_zip_test_file`, etc.

**Task 3: Add CLI test in `tests/cli_integration_tests.rs`**
- Add integration test that invokes the CLI with `--use-builtin` flag
- Test file identification output for known file types
- Verify the flag works correctly and takes precedence over `--magic-file` as
documented

**Task 4: Document error handling in `build.rs`**
- Add detailed comments explaining how to test error handling:
  - Suggest temporarily introducing a syntax error in `src/builtin_rules.magic`
- Verify error messages include:
  - "ERROR: Failed to parse built-in magic rules"
  - File path (`src/builtin_rules.magic`)
  - Line number where error occurred
  - Specific error description
  - Suggestion to fix syntax

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

This phase establishes the build-time compilation pipeline for built-in magic
rules. The goal is to create the source magic file with common file type
signatures, configure Cargo to run a build script, and implement a minimal
parser in build.rs that generates Rust code at build time.

**Task 1: Create `src/builtin_rules.magic`**
- Create a new file `src/builtin_rules.magic` containing magic rules for common
file types
- Include rules for: ELF (with 32/64-bit and endianness children), PE/MZ
executables, ZIP archives, JPEG images, PNG images, PDF documents, GIF images
- Use standard magic file syntax:
  - Comments with `#`
  - Level markers (`>`, `>>`) for hierarchical rules
  - Absolute offsets in decimal and hex format
  - Types: `string`, `byte`, `lelong`
  - Escape sequences like `\x7f`, `\x89`
- Add inline comments explaining each file signature
- Keep rules minimal and focused on common formats

**Task 2: Configure `Cargo.toml`**
- Add `build = "build.rs"` to the `[package]` section
- No build dependencies needed (self-contained parser)

**Task 3: Create `build.rs`**
- Create `build.rs` at the project root
- Implement minimal parser functions that handle:
  - Comment lines (starting with `#`)
  - Empty lines
  - Absolute offsets (decimal and hex formats)
  - Level markers (`>`, `>>`) for rule hierarchy
  - Supported types: `string`, `byte`, `lelong`
  - Escape sequences in values
  - Message extraction
- Implement code generation functions that:
- Produce Rust code constructing `MagicRule`, `OffsetSpec`, `TypeKind`,
`Operator`, `Value`, and `Endianness` structures
- Generate a function `builtin_rules() -> Vec<MagicRule>` that returns the
parsed rules
- Read `src/builtin_rules.magic`, parse it, and generate code
- Write output to `$OUT_DIR/builtin_rules_generated.rs`
- Error handling:
  - Print file name, line number, and error description on parse failure
  - Exit with code 1 on error
- Add `cargo:rerun-if-changed=src/builtin_rules.magic` for incremental builds
===============================================================================

Task: 2

This phase connects the build-time generated rules to the library's public API.
The goal is to create a module that includes the generated code and update the
`MagicDatabase` stub methods to use the actual built-in rules.

**Task 1: Create `src/builtin_rules.rs`**
- Create a new file `src/builtin_rules.rs`
- Include the generated code using `include!(concat!(env!("OUT_DIR"),
"/builtin_rules_generated.rs"))`
- Export a public function `get_builtin_rules() -> Vec<MagicRule>` that calls
the generated `builtin_rules()` function
- Add module-level documentation:
  - Explain that rules are compiled at build time from `builtin_rules.magic`
  - Mention the supported file formats

**Task 2: Update `src/lib.rs`**
- Add `mod builtin_rules;` declaration
- Update `MagicDatabase::with_builtin_rules()` method:
  - Replace the current stub implementation (returning `Vec::new()`)
  - Call `builtin_rules::get_builtin_rules()` to get the actual rules
  - Ensure `source_path` field is set to `None`
- Update `MagicDatabase::with_builtin_rules_and_config()` method similarly:
  - Replace stub with call to `builtin_rules::get_builtin_rules()`
  - Ensure `source_path` field is set to `None`
- Add or update Rustdoc for both methods:
  - Explain they provide fallback rules for common file types
  - List supported formats: ELF, PE, ZIP, JPEG, PNG, PDF, GIF
===============================================================================

Task: 3

This phase validates the implementation through comprehensive testing. The goal
is to add unit tests for the generated rules structure, integration tests for
file detection, CLI tests for the `--use-builtin` flag, and document error
handling behavior.

**Task 1: Add unit tests in `src/builtin_rules.rs`**
- Add a `#[cfg(test)]` module
- Test that `get_builtin_rules()` returns a non-empty vector
- Test that expected top-level rules exist for: ELF, MZ, ZIP, JPEG, PNG, PDF,
GIF
- Test that the ELF rule has expected child rules for architecture detection
- Test specific rule field values:
  - Offsets match expected values
  - Types (TypeKind) are correct
  - Operators are as expected
  - Values match the file signatures

**Task 2: Create `tests/builtin_rules_tests.rs`**
- Create integration tests for file detection using built-in rules
- Test `MagicDatabase::with_builtin_rules()` correctly identifies:
  - ELF binary (use existing test file helper)
  - ZIP archive
  - JPEG image
  - PNG image
  - PDF document
  - GIF image
- Test that unknown/unrecognized files return "data" as fallback
- Use existing test file creation helpers from test utilities:
`create_elf_test_file`, `create_zip_test_file`, etc.

**Task 3: Add CLI test in `tests/cli_integration_tests.rs`**
- Add integration test that invokes the CLI with `--use-builtin` flag
- Test file identification output for known file types
- Verify the flag works correctly and takes precedence over `--magic-file` as
documented

**Task 4: Document error handling in `build.rs`**
- Add detailed comments explaining how to test error handling:
  - Suggest temporarily introducing a syntax error in `src/builtin_rules.magic`
- Verify error messages include:
  - "ERROR: Failed to parse built-in magic rules"
  - File path (`src/builtin_rules.magic`)
  - Line number where error occurred
  - Specific error description
  - Suggestion to fix syntax

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!