Strength Calculation: libmagic Algorithm & !:strength Parsing

[unclesp1d3r]

Strength Calculation: libmagic Algorithm & !:strength Parsing

Overview

Implement libmagic's apprentice_magic_strength algorithm with !:strength modifier parsing to achieve high GNU file compatibility. Strength affects rule priority and matching behavior, critical for 95%+ test corpus compatibility.

Scope

In Scope:

• Port apprentice_magic_strength algorithm from libmagic
• Parse !:strength modifiers from magic files
• Calculate default strength based on rule specificity
• Store strength in MagicRule structure
• Use strength for rule ordering and confidence scoring

Out of Scope:

• Advanced strength heuristics
• Machine learning-based strength
• Performance optimization

Technical Approach

1. Strength Modifier Parsing

Update file:src/parser/ast.rs:

pub enum StrengthModifier {
    Add(i32),      // !:strength +10
    Subtract(i32), // !:strength -5
    Multiply(i32), // !:strength *2
    Divide(i32),   // !:strength /2
    Set(i32),      // !:strength =50
}

pub struct MagicRule {
    // ... existing fields ...
    pub strength_modifier: Option<StrengthModifier>,
}

2. Strength Parsing

Add to file:src/parser/grammar.rs:

pub fn parse_strength_modifier(line: &str) -> Option<StrengthModifier> {
    // Parse !:strength OP VALUE syntax
    // Examples:
    //   !:strength +10
    //   !:strength -5
    //   !:strength *2
    //   !:strength /2
    //   !:strength =50
}

3. Default Strength Calculation

Add to file:src/parser/mod.rs:

pub fn calculate_default_strength(rule: &MagicRule) -> i32 {
    // Port libmagic's apprentice_magic_strength algorithm
    // Factors:
    // - Type specificity (string > byte)
    // - Operator specificity (= > &)
    // - Offset type (absolute > indirect)
    // - Value length (longer strings = higher strength)

    let mut strength = 0;

    // Type contribution
    strength += match rule.type_kind {
        TypeKind::String => 20,
        TypeKind::Long => 15,
        TypeKind::Short => 10,
        TypeKind::Byte => 5,
        _ => 10,
    };

    // Operator contribution
    strength += match rule.operator {
        Operator::Equal => 10,
        Operator::NotEqual => 5,
        Operator::And => 3,
        _ => 5,
    };

    // Offset contribution
    strength += match rule.offset {
        OffsetSpec::Absolute(_) => 10,
        OffsetSpec::Indirect { .. } => 5,
        OffsetSpec::Relative(_) => 3,
    };

    // Value length contribution (for strings)
    if let TypeKind::String = rule.type_kind {
        strength += (rule.value.len() as i32).min(20);
    }

    strength
}

4. Strength Application

Update file:src/parser/mod.rs:

pub fn apply_strength_modifier(base_strength: i32, modifier: &StrengthModifier) -> i32 {
    match modifier {
        StrengthModifier::Add(n) => base_strength + n,
        StrengthModifier::Subtract(n) => base_strength - n,
        StrengthModifier::Multiply(n) => base_strength * n,
        StrengthModifier::Divide(n) => base_strength / n,
        StrengthModifier::Set(n) => *n,
    }
}

5. Integration with Evaluation

Update file:src/evaluator/mod.rs:

// Use strength for rule ordering
pub fn sort_rules_by_strength(rules: &mut [MagicRule]) {
    rules.sort_by(|a, b| {
        let strength_a = calculate_default_strength(a);
        let strength_b = calculate_default_strength(b);
        strength_b.cmp(&strength_a) // Higher strength first
    });
}

Acceptance Criteria

• !:strength modifiers parsed correctly
• Default strength calculated for all rule types
• Strength modifiers applied correctly
• Rules sorted by strength during evaluation
• Strength affects confidence scoring
• Unit tests for strength parsing
• Unit tests for default strength calculation
• Unit tests for strength modifiers
• Integration test comparing with GNU file output
• Compatibility test shows improved match rate

Dependencies

• Depends on: ticket:75a688c2-0ac4-489a-a35d-6e824c94c153/c554e409-ae60-407f-9596-64c5b03a9b92 (Parser Integration)

Related Specs

• spec:75a688c2-0ac4-489a-a35d-6e824c94c153/269e848a-258d-4cd4-99b1-386bd400a109 (Technical Plan - Strength Calculation)
• spec:75a688c2-0ac4-489a-a35d-6e824c94c153/3ce0475b-153d-487f-bc0d-47d0a8f6708a (Epic Brief - 95%+ compatibility target)

Files to Modify

• file:src/parser/ast.rs - Add StrengthModifier enum, update MagicRule
• file:src/parser/grammar.rs - Add strength parsing
• file:src/parser/mod.rs - Add strength calculation functions
• file:src/evaluator/mod.rs - Use strength for rule ordering
• file:tests/ - Add strength calculation tests

[traycerai]

Plan

Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (file:src/parser/grammar.rs, file:src/parser/ast.rs) and evaluator infrastructure (file:src/evaluator/mod.rs). The MagicRule structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's apprentice_magic_strength algorithm with !:strength modifier parsing. The baseline strength starts at 0, with contributions from type specificity, operator specificity, offset type, and value length.

Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for !:strength directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

Implementation Steps

1. Extend AST Structures for Strength Support

File: file:src/parser/ast.rs

Add StrengthModifier enum to represent the five strength operations:

• Add(i32) for !:strength +N
• Subtract(i32) for !:strength -N
• Multiply(i32) for !:strength *N
• Divide(i32) for !:strength /N
• Set(i32) for !:strength =N

Update MagicRule struct to include:

• strength_modifier: Option<StrengthModifier> - stores parsed !:strength directive
• calculated_strength: Option<i32> - stores final computed strength value

Add impl blocks for StrengthModifier with validation methods ensuring values are within 0-255 range.

2. Implement Strength Modifier Parsing

File: file:src/parser/grammar.rs

Create parse_strength_modifier() function using nom combinators:

• Match the literal !:strength prefix
• Parse operator: one of +, -, *, /, =
• Parse numeric value (0-255 range)
• Return StrengthModifier enum variant
• Handle whitespace around operators

Add unit tests covering:

• All five operator types
• Edge cases (0, 255, out-of-range values)
• Whitespace variations
• Invalid syntax error cases

3. Integrate Strength Parsing into Rule Parser

File: file:src/parser/mod.rs

Update parse_text_magic_file() to handle strength directives:

• Track the last parsed rule during preprocessing
• When a line starts with !:strength, parse using parse_strength_modifier()
• Attach modifier to the last parsed rule in the hierarchy
• Validate that !:strength doesn't appear before any rules
• Preserve line number information for error reporting

Create helper function attach_strength_modifier_to_last_rule():

• Traverses rule hierarchy to find the last rule at appropriate level
• Attaches strength modifier to that rule
• Returns error if no previous rule exists

4. Implement Default Strength Calculation Algorithm

File: file:src/parser/mod.rs

Create calculate_default_strength(rule: &MagicRule) -> i32 function implementing libmagic's algorithm:

Type Contribution:

• TypeKind::String → +20 points
• TypeKind::Long → +15 points
• TypeKind::Short → +10 points
• TypeKind::Byte → +5 points

Operator Contribution:

• Operator::Equal → +10 points
• Operator::NotEqual → +5 points
• Operator::BitwiseAnd / Operator::BitwiseAndMask → +3 points

Offset Contribution:

• OffsetSpec::Absolute → +10 points
• OffsetSpec::Indirect → +5 points
• OffsetSpec::Relative → +3 points
• OffsetSpec::FromEnd → +3 points

Value Length Contribution (for strings):

• Add min(value_length, 20) for string types

Start with baseline of 0 and sum all contributions. Clamp final result to valid range (0-255).

5. Implement Strength Modifier Application

File: file:src/parser/mod.rs

Create apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32:

• Match on modifier variant
• Apply operation: Add → base + n, Subtract → base - n, Multiply → base * n, Divide → base / n, Set → n
• Clamp result to valid range (0-255)
• Handle division by zero gracefully (return base unchanged)
• Validate modifier values are within 0-255 range

Add comprehensive unit tests for:

• Each operation type
• Boundary conditions (overflow, underflow)
• Clamping behavior
• Division by zero handling

6. Integrate Strength Calculation into Parser Pipeline

File: file:src/parser/mod.rs

Create calculate_rule_strengths() post-processing function:

• After build_rule_hierarchy() completes, traverse entire rule tree
• For each rule, calculate default strength using calculate_default_strength()
• If strength_modifier is present, apply using apply_strength_modifier()
• Store final value in calculated_strength field
• Recursively process child rules (each calculates independently)

Integrate into parse_text_magic_file():

• Call calculate_rule_strengths() after build_rule_hierarchy()
• Ensure all rules have strength values before returning

7. Use Strength for Rule Ordering in Evaluator

File: file:src/evaluator/mod.rs

Create sort_rules_by_strength(rules: &mut [MagicRule]):

• Sort rules by calculated_strength in descending order (higher strength first)
• Preserve original order for rules with equal strength (stable sort)
• Apply recursively to child rules before returning

Integrate into evaluate_rules():

• Sort sibling rules by strength before iterating through them
• When multiple rules match at same level, strength determines priority
• Higher strength matches are returned first
• Strength affects confidence scoring in results

8. Expose Strength in Evaluation Results

File: file:src/evaluator/mod.rs

Update MatchResult struct to include strength: i32 field.

Populate strength value during rule evaluation:

• In evaluate_single_rule(), retrieve calculated_strength from matched rule
• Pass strength to MatchResult constructor
• Ensure strength is available for all match results

File: file:src/output/json.rs and file:src/output/text.rs

Include strength in output:

• JSON output: add "strength" field alongside existing fields
• Text output: optionally display strength with -l flag compatibility

9. Add Comprehensive Testing

File: file:tests/strength_tests.rs (new file)

Create unit tests for:

• Strength modifier parsing (all operators, edge cases, whitespace)
• Default strength calculation (all type/operator/offset combinations)
• Strength modifier application (all operations, clamping, division by zero)
• Rule sorting by strength (stable sort, equal strength preservation)
• Strength calculation with child rules (independent calculation)
• Integration test with real magic files containing !:strength directives

Add integration tests:

• Parse magic files with !:strength modifiers
• Verify strength values are calculated correctly
• Compare rule ordering with and without strength
• Test compatibility with GNU file output using file -l command

Add edge case tests:

• Strength modifiers on rules with different types
• Multiple strength modifiers (last one wins)
• Out-of-range strength values (clamping behavior)
• Division by zero in strength calculation

10. Update Documentation

File: file:docs/src/magic-format.md or file:README.md

Add comprehensive section documenting !:strength directive:

• Syntax: !:strength OP VALUE where OP is one of +, -, *, /, =
• Semantics: modifies strength of preceding rule
• Examples of each operator with expected results
• Explanation of default strength calculation algorithm
• Best practices for using strength modifiers
• Impact on rule priority and matching behavior

File: src/parser/ast.rs

Add rustdoc comments to StrengthModifier enum:

• Document each variant with examples
• Explain valid value ranges (0-255)
• Describe clamping behavior

File: src/parser/mod.rs

Add rustdoc comments to strength functions:

• calculate_default_strength(): explain algorithm, type/operator/offset contributions
• apply_strength_modifier(): explain operations, clamping, division by zero handling
• calculate_rule_strengths(): explain tree traversal and recursive calculation
• Provide usage examples for each function
• Reference libmagic's apprentice_magic_strength algorithm

Acceptance Criteria Mapping

• !:strength modifiers parsed correctly → Steps 2, 3, 9
• Default strength calculated for all rule types → Steps 4, 9
• Strength modifiers applied correctly → Steps 5, 9
• Rules sorted by strength during evaluation → Steps 7, 9
• Strength affects confidence scoring → Steps 8, 9
• Unit tests for strength parsing → Step 9
• Unit tests for default strength calculation → Step 9
• Unit tests for strength modifiers → Step 9
• Integration test comparing with GNU file output → Step 9
• Compatibility test shows improved match rate → Step 9

Sequence Diagram

Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (`file:src/parser/grammar.rs`, `file:src/parser/ast.rs`) and evaluator infrastructure (`file:src/evaluator/mod.rs`). The `MagicRule` structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's `apprentice_magic_strength` algorithm with `!:strength` modifier parsing. The baseline strength starts at 0, with contributions from type specificity, operator specificity, offset type, and value length.

## Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for `!:strength` directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

## Implementation Steps

### 1. Extend AST Structures for Strength Support

**File: `file:src/parser/ast.rs`**

Add `StrengthModifier` enum to represent the five strength operations:
- `Add(i32)` for `!:strength +N`
- `Subtract(i32)` for `!:strength -N`
- `Multiply(i32)` for `!:strength *N`
- `Divide(i32)` for `!:strength /N`
- `Set(i32)` for `!:strength =N`

Update `MagicRule` struct to include:
- `strength_modifier: Option<StrengthModifier>` - stores parsed `!:strength` directive
- `calculated_strength: Option<i32>` - stores final computed strength value

Add `impl` blocks for `StrengthModifier` with validation methods ensuring values are within 0-255 range.

### 2. Implement Strength Modifier Parsing

**File: `file:src/parser/grammar.rs`**

Create `parse_strength_modifier()` function using nom combinators:
- Match the literal `!:strength` prefix
- Parse operator: one of `+`, `-`, `*`, `/`, `=`
- Parse numeric value (0-255 range)
- Return `StrengthModifier` enum variant
- Handle whitespace around operators

Add unit tests covering:
- All five operator types
- Edge cases (0, 255, out-of-range values)
- Whitespace variations
- Invalid syntax error cases

### 3. Integrate Strength Parsing into Rule Parser

**File: `file:src/parser/mod.rs`**

Update `parse_text_magic_file()` to handle strength directives:
- Track the last parsed rule during preprocessing
- When a line starts with `!:strength`, parse using `parse_strength_modifier()`
- Attach modifier to the last parsed rule in the hierarchy
- Validate that `!:strength` doesn't appear before any rules
- Preserve line number information for error reporting

Create helper function `attach_strength_modifier_to_last_rule()`:
- Traverses rule hierarchy to find the last rule at appropriate level
- Attaches strength modifier to that rule
- Returns error if no previous rule exists

### 4. Implement Default Strength Calculation Algorithm

**File: `file:src/parser/mod.rs`**

Create `calculate_default_strength(rule: &MagicRule) -> i32` function implementing libmagic's algorithm:

**Type Contribution:**
- `TypeKind::String` → +20 points
- `TypeKind::Long` → +15 points
- `TypeKind::Short` → +10 points
- `TypeKind::Byte` → +5 points

**Operator Contribution:**
- `Operator::Equal` → +10 points
- `Operator::NotEqual` → +5 points
- `Operator::BitwiseAnd` / `Operator::BitwiseAndMask` → +3 points

**Offset Contribution:**
- `OffsetSpec::Absolute` → +10 points
- `OffsetSpec::Indirect` → +5 points
- `OffsetSpec::Relative` → +3 points
- `OffsetSpec::FromEnd` → +3 points

**Value Length Contribution (for strings):**
- Add `min(value_length, 20)` for string types

Start with baseline of 0 and sum all contributions. Clamp final result to valid range (0-255).

### 5. Implement Strength Modifier Application

**File: `file:src/parser/mod.rs`**

Create `apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32`:
- Match on modifier variant
- Apply operation: `Add` → `base + n`, `Subtract` → `base - n`, `Multiply` → `base * n`, `Divide` → `base / n`, `Set` → `n`
- Clamp result to valid range (0-255)
- Handle division by zero gracefully (return base unchanged)
- Validate modifier values are within 0-255 range

Add comprehensive unit tests for:
- Each operation type
- Boundary conditions (overflow, underflow)
- Clamping behavior
- Division by zero handling

### 6. Integrate Strength Calculation into Parser Pipeline

**File: `file:src/parser/mod.rs`**

Create `calculate_rule_strengths()` post-processing function:
- After `build_rule_hierarchy()` completes, traverse entire rule tree
- For each rule, calculate default strength using `calculate_default_strength()`
- If `strength_modifier` is present, apply using `apply_strength_modifier()`
- Store final value in `calculated_strength` field
- Recursively process child rules (each calculates independently)

Integrate into `parse_text_magic_file()`:
- Call `calculate_rule_strengths()` after `build_rule_hierarchy()`
- Ensure all rules have strength values before returning

### 7. Use Strength for Rule Ordering in Evaluator

**File: `file:src/evaluator/mod.rs`**

Create `sort_rules_by_strength(rules: &mut [MagicRule])`:
- Sort rules by `calculated_strength` in descending order (higher strength first)
- Preserve original order for rules with equal strength (stable sort)
- Apply recursively to child rules before returning

Integrate into `evaluate_rules()`:
- Sort sibling rules by strength before iterating through them
- When multiple rules match at same level, strength determines priority
- Higher strength matches are returned first
- Strength affects confidence scoring in results

### 8. Expose Strength in Evaluation Results

**File: `file:src/evaluator/mod.rs`**

Update `MatchResult` struct to include `strength: i32` field.

Populate strength value during rule evaluation:
- In `evaluate_single_rule()`, retrieve `calculated_strength` from matched rule
- Pass strength to `MatchResult` constructor
- Ensure strength is available for all match results

**File: `file:src/output/json.rs`** and **`file:src/output/text.rs`**

Include strength in output:
- JSON output: add `"strength"` field alongside existing fields
- Text output: optionally display strength with `-l` flag compatibility

### 9. Add Comprehensive Testing

**File: `file:tests/strength_tests.rs`** (new file)

Create unit tests for:
- Strength modifier parsing (all operators, edge cases, whitespace)
- Default strength calculation (all type/operator/offset combinations)
- Strength modifier application (all operations, clamping, division by zero)
- Rule sorting by strength (stable sort, equal strength preservation)
- Strength calculation with child rules (independent calculation)
- Integration test with real magic files containing `!:strength` directives

Add integration tests:
- Parse magic files with `!:strength` modifiers
- Verify strength values are calculated correctly
- Compare rule ordering with and without strength
- Test compatibility with GNU file output using `file -l` command

Add edge case tests:
- Strength modifiers on rules with different types
- Multiple strength modifiers (last one wins)
- Out-of-range strength values (clamping behavior)
- Division by zero in strength calculation

### 10. Update Documentation

**File: `file:docs/src/magic-format.md`** or **`file:README.md`**

Add comprehensive section documenting `!:strength` directive:
- Syntax: `!:strength OP VALUE` where OP is one of `+`, `-`, `*`, `/`, `=`
- Semantics: modifies strength of preceding rule
- Examples of each operator with expected results
- Explanation of default strength calculation algorithm
- Best practices for using strength modifiers
- Impact on rule priority and matching behavior

**File: `src/parser/ast.rs`**

Add rustdoc comments to `StrengthModifier` enum:
- Document each variant with examples
- Explain valid value ranges (0-255)
- Describe clamping behavior

**File: `src/parser/mod.rs`**

Add rustdoc comments to strength functions:
- `calculate_default_strength()`: explain algorithm, type/operator/offset contributions
- `apply_strength_modifier()`: explain operations, clamping, division by zero handling
- `calculate_rule_strengths()`: explain tree traversal and recursive calculation
- Provide usage examples for each function
- Reference libmagic's `apprentice_magic_strength` algorithm

## Acceptance Criteria Mapping

- `!:strength` modifiers parsed correctly → Steps 2, 3, 9
- Default strength calculated for all rule types → Steps 4, 9
- Strength modifiers applied correctly → Steps 5, 9
- Rules sorted by strength during evaluation → Steps 7, 9
- Strength affects confidence scoring → Steps 8, 9
- Unit tests for strength parsing → Step 9
- Unit tests for default strength calculation → Step 9
- Unit tests for strength modifiers → Step 9
- Integration test comparing with GNU file output → Step 9
- Compatibility test shows improved match rate → Step 9

## Sequence Diagram

```
Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field
```

---

Execution Information

Branch: main
Commit: 50b370c

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

Previous Plan

Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (file:src/parser/grammar.rs, file:src/parser/ast.rs) and evaluator infrastructure (file:src/evaluator/mod.rs). The MagicRule structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's apprentice_magic_strength algorithm with !:strength modifier parsing. The baseline strength starts at 20, with contributions from type specificity, operator specificity, offset type, and value length.

Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for !:strength directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

Implementation Steps

1. Extend AST Structures for Strength Support

File: file:src/parser/ast.rs

Add StrengthModifier enum to represent the five strength operations:

• Add(i32) for !:strength +N
• Subtract(i32) for !:strength -N
• Multiply(i32) for !:strength *N
• Divide(i32) for !:strength /N
• Set(i32) for !:strength =N

Update MagicRule struct to include:

• strength_modifier: Option<StrengthModifier> - stores parsed !:strength directive
• calculated_strength: Option<i32> - stores final computed strength value

Add impl blocks for StrengthModifier with validation methods ensuring values are within 0-255 range.

2. Implement Strength Modifier Parsing

File: file:src/parser/grammar.rs

Create parse_strength_modifier() function using nom combinators:

• Match the literal !:strength prefix
• Parse operator: one of +, -, *, /, =
• Parse numeric value (0-255 range)
• Return StrengthModifier enum variant
• Handle whitespace around operators

Add unit tests covering:

• All five operator types
• Edge cases (0, 255, out-of-range values)
• Whitespace variations
• Invalid syntax error cases

3. Integrate Strength Parsing into Rule Parser

File: file:src/parser/mod.rs

Update parse_magic_rule_line() to detect and parse !:strength lines:

• Check if line starts with !:strength
• If yes, parse using parse_strength_modifier() and attach to previous rule
• If no, proceed with normal rule parsing

Modify parse_text_magic_file() to handle strength directives:

• Track the last parsed rule
• When !:strength is encountered, attach modifier to last rule
• Validate that !:strength doesn't appear before any rules

4. Implement Default Strength Calculation Algorithm

File: file:src/parser/mod.rs

Create calculate_default_strength(rule: &MagicRule) -> i32 function implementing libmagic's algorithm:

Type Contribution:

• TypeKind::String → +20 points
• TypeKind::Long → +15 points
• TypeKind::Short → +10 points
• TypeKind::Byte → +5 points

Operator Contribution:

• Operator::Equal → +10 points
• Operator::NotEqual → +5 points
• Operator::BitwiseAnd / Operator::BitwiseAndMask → +3 points

Offset Contribution:

• OffsetSpec::Absolute → +10 points
• OffsetSpec::Indirect → +5 points
• OffsetSpec::Relative → +3 points
• OffsetSpec::FromEnd → +3 points

Value Length Contribution (for strings):

• Add min(value_length, 20) for string types

Start with baseline of 0 and sum all contributions.

5. Implement Strength Modifier Application

File: file:src/parser/mod.rs

Create apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32:

• Match on modifier variant
• Apply operation: Add → base + n, Subtract → base - n, Multiply → base * n, Divide → base / n, Set → n
• Clamp result to valid range (0-255)
• Handle division by zero gracefully

Add comprehensive unit tests for:

• Each operation type
• Boundary conditions (overflow, underflow)
• Clamping behavior

6. Integrate Strength Calculation into Parser Pipeline

File: file:src/parser/mod.rs

Update build_rule_hierarchy() or create post-processing step:

• After parsing all rules, iterate through rule tree
• For each rule, calculate default strength using calculate_default_strength()
• If strength_modifier is present, apply using apply_strength_modifier()
• Store final value in calculated_strength field

Ensure child rules inherit or calculate their own strength independently.

7. Use Strength for Rule Ordering in Evaluator

File: file:src/evaluator/mod.rs

Create sort_rules_by_strength(rules: &mut [MagicRule]):

• Sort rules by calculated_strength in descending order (higher strength first)
• Preserve original order for rules with equal strength
• Apply recursively to child rules

Integrate into evaluate_rules():

• Sort rules before evaluation when stop_at_first_match is false
• Use strength to determine which match to return when multiple matches occur

8. Expose Strength in Evaluation Results

File: file:src/evaluator/mod.rs

Update MatchResult struct to include strength: i32 field.

Populate strength value during rule evaluation in evaluate_single_rule().

File: file:src/output/json.rs

Include strength in JSON output as "strength" field alongside existing fields.

9. Add Comprehensive Testing

File: file:tests/strength_tests.rs (new file)

Create unit tests for:

• Strength modifier parsing (all operators, edge cases)
• Default strength calculation (all type/operator/offset combinations)
• Strength modifier application (all operations, clamping)
• Rule sorting by strength
• Integration test comparing with GNU file output using file -l command

Add property-based tests using proptest crate to verify:

• Strength values always in valid range
• Sorting is stable and correct
• Modifier application is associative where applicable

10. Update Documentation

File: file:docs/src/magic-format.md

Add section documenting !:strength directive:

• Syntax and semantics
• Examples of each operator
• Explanation of default strength calculation
• Best practices for using strength modifiers

File: file:src/parser/ast.rs, file:src/parser/mod.rs

Add rustdoc comments to all new types and functions:

• Explain strength calculation algorithm
• Provide usage examples
• Document valid ranges and error conditions

sequenceDiagram
    participant Parser as Parser Module
    participant AST as MagicRule AST
    participant Calc as Strength Calculator
    participant Eval as Evaluator
    
    Parser->>Parser: parse_magic_rule_line()
    Parser->>Parser: detect !:strength directive
    Parser->>Parser: parse_strength_modifier()
    Parser->>AST: attach StrengthModifier
    
    Parser->>Calc: calculate_default_strength(rule)
    Calc->>Calc: sum type + operator + offset + value contributions
    Calc-->>Parser: base_strength
    
    alt has strength_modifier
        Parser->>Calc: apply_strength_modifier(base, modifier)
        Calc-->>Parser: final_strength
    end
    
    Parser->>AST: set calculated_strength
    
    Eval->>Eval: sort_rules_by_strength()
    Eval->>Eval: evaluate rules in strength order
    Eval->>AST: include strength in MatchResult

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (`file:src/parser/grammar.rs`, `file:src/parser/ast.rs`) and evaluator infrastructure (`file:src/evaluator/mod.rs`). The `MagicRule` structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's `apprentice_magic_strength` algorithm with `!:strength` modifier parsing. The baseline strength starts at 20, with contributions from type specificity, operator specificity, offset type, and value length.

## Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for `!:strength` directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

## Implementation Steps

### 1. Extend AST Structures for Strength Support

**File: `file:src/parser/ast.rs`**

Add `StrengthModifier` enum to represent the five strength operations:
- `Add(i32)` for `!:strength +N`
- `Subtract(i32)` for `!:strength -N`
- `Multiply(i32)` for `!:strength *N`
- `Divide(i32)` for `!:strength /N`
- `Set(i32)` for `!:strength =N`

Update `MagicRule` struct to include:
- `strength_modifier: Option<StrengthModifier>` - stores parsed `!:strength` directive
- `calculated_strength: Option<i32>` - stores final computed strength value

Add `impl` blocks for `StrengthModifier` with validation methods ensuring values are within 0-255 range.

### 2. Implement Strength Modifier Parsing

**File: `file:src/parser/grammar.rs`**

Create `parse_strength_modifier()` function using nom combinators:
- Match the literal `!:strength` prefix
- Parse operator: one of `+`, `-`, `*`, `/`, `=`
- Parse numeric value (0-255 range)
- Return `StrengthModifier` enum variant
- Handle whitespace around operators

Add unit tests covering:
- All five operator types
- Edge cases (0, 255, out-of-range values)
- Whitespace variations
- Invalid syntax error cases

### 3. Integrate Strength Parsing into Rule Parser

**File: `file:src/parser/mod.rs`**

Update `parse_magic_rule_line()` to detect and parse `!:strength` lines:
- Check if line starts with `!:strength`
- If yes, parse using `parse_strength_modifier()` and attach to previous rule
- If no, proceed with normal rule parsing

Modify `parse_text_magic_file()` to handle strength directives:
- Track the last parsed rule
- When `!:strength` is encountered, attach modifier to last rule
- Validate that `!:strength` doesn't appear before any rules

### 4. Implement Default Strength Calculation Algorithm

**File: `file:src/parser/mod.rs`**

Create `calculate_default_strength(rule: &MagicRule) -> i32` function implementing libmagic's algorithm:

**Type Contribution:**
- `TypeKind::String` → +20 points
- `TypeKind::Long` → +15 points
- `TypeKind::Short` → +10 points
- `TypeKind::Byte` → +5 points

**Operator Contribution:**
- `Operator::Equal` → +10 points
- `Operator::NotEqual` → +5 points
- `Operator::BitwiseAnd` / `Operator::BitwiseAndMask` → +3 points

**Offset Contribution:**
- `OffsetSpec::Absolute` → +10 points
- `OffsetSpec::Indirect` → +5 points
- `OffsetSpec::Relative` → +3 points
- `OffsetSpec::FromEnd` → +3 points

**Value Length Contribution (for strings):**
- Add `min(value_length, 20)` for string types

Start with baseline of 0 and sum all contributions.

### 5. Implement Strength Modifier Application

**File: `file:src/parser/mod.rs`**

Create `apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32`:
- Match on modifier variant
- Apply operation: `Add` → `base + n`, `Subtract` → `base - n`, `Multiply` → `base * n`, `Divide` → `base / n`, `Set` → `n`
- Clamp result to valid range (0-255)
- Handle division by zero gracefully

Add comprehensive unit tests for:
- Each operation type
- Boundary conditions (overflow, underflow)
- Clamping behavior

### 6. Integrate Strength Calculation into Parser Pipeline

**File: `file:src/parser/mod.rs`**

Update `build_rule_hierarchy()` or create post-processing step:
- After parsing all rules, iterate through rule tree
- For each rule, calculate default strength using `calculate_default_strength()`
- If `strength_modifier` is present, apply using `apply_strength_modifier()`
- Store final value in `calculated_strength` field

Ensure child rules inherit or calculate their own strength independently.

### 7. Use Strength for Rule Ordering in Evaluator

**File: `file:src/evaluator/mod.rs`**

Create `sort_rules_by_strength(rules: &mut [MagicRule])`:
- Sort rules by `calculated_strength` in descending order (higher strength first)
- Preserve original order for rules with equal strength
- Apply recursively to child rules

Integrate into `evaluate_rules()`:
- Sort rules before evaluation when `stop_at_first_match` is false
- Use strength to determine which match to return when multiple matches occur

### 8. Expose Strength in Evaluation Results

**File: `file:src/evaluator/mod.rs`**

Update `MatchResult` struct to include `strength: i32` field.

Populate strength value during rule evaluation in `evaluate_single_rule()`.

**File: `file:src/output/json.rs`**

Include strength in JSON output as `"strength"` field alongside existing fields.

### 9. Add Comprehensive Testing

**File: `file:tests/strength_tests.rs`** (new file)

Create unit tests for:
- Strength modifier parsing (all operators, edge cases)
- Default strength calculation (all type/operator/offset combinations)
- Strength modifier application (all operations, clamping)
- Rule sorting by strength
- Integration test comparing with GNU file output using `file -l` command

Add property-based tests using `proptest` crate to verify:
- Strength values always in valid range
- Sorting is stable and correct
- Modifier application is associative where applicable

### 10. Update Documentation

**File: `file:docs/src/magic-format.md`**

Add section documenting `!:strength` directive:
- Syntax and semantics
- Examples of each operator
- Explanation of default strength calculation
- Best practices for using strength modifiers

**File: `file:src/parser/ast.rs`, `file:src/parser/mod.rs`**

Add rustdoc comments to all new types and functions:
- Explain strength calculation algorithm
- Provide usage examples
- Document valid ranges and error conditions

```mermaid
sequenceDiagram
    participant Parser as Parser Module
    participant AST as MagicRule AST
    participant Calc as Strength Calculator
    participant Eval as Evaluator
    
    Parser->>Parser: parse_magic_rule_line()
    Parser->>Parser: detect !:strength directive
    Parser->>Parser: parse_strength_modifier()
    Parser->>AST: attach StrengthModifier
    
    Parser->>Calc: calculate_default_strength(rule)
    Calc->>Calc: sum type + operator + offset + value contributions
    Calc-->>Parser: base_strength
    
    alt has strength_modifier
        Parser->>Calc: apply_strength_modifier(base, modifier)
        Calc-->>Parser: final_strength
    end
    
    Parser->>AST: set calculated_strength
    
    Eval->>Eval: sort_rules_by_strength()
    Eval->>Eval: evaluate rules in strength order
    Eval->>AST: include strength in MatchResult
```

---

Execution Information

Branch: main
Commit: 50b370c

New Plan

Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (file:src/parser/grammar.rs, file:src/parser/ast.rs) and evaluator infrastructure (file:src/evaluator/mod.rs). The MagicRule structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's apprentice_magic_strength algorithm with !:strength modifier parsing. The baseline strength starts at 20, with contributions from type specificity, operator specificity, offset type, and value length.

Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for !:strength directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

Implementation Steps

1. Extend AST Structures for Strength Support

File: file:src/parser/ast.rs

Add StrengthModifier enum to represent the five strength operations:

• Add(i32) for !:strength +N
• Subtract(i32) for !:strength -N
• Multiply(i32) for !:strength *N
• Divide(i32) for !:strength /N
• Set(i32) for !:strength =N

Update MagicRule struct to include:

• strength_modifier: Option<StrengthModifier> - stores parsed !:strength directive
• calculated_strength: Option<i32> - stores final computed strength value

Add impl blocks for StrengthModifier with validation methods ensuring values are within 0-255 range.

2. Implement Strength Modifier Parsing

File: file:src/parser/grammar.rs

Create parse_strength_modifier() function using nom combinators:

• Match the literal !:strength prefix
• Parse operator: one of +, -, *, /, =
• Parse numeric value (0-255 range)
• Return StrengthModifier enum variant
• Handle whitespace around operators

Add unit tests covering:

• All five operator types
• Edge cases (0, 255, out-of-range values)
• Whitespace variations
• Invalid syntax error cases

3. Integrate Strength Parsing into Rule Parser

File: file:src/parser/mod.rs

Update parse_magic_rule_line() to detect and parse !:strength lines:

• Check if line starts with !:strength
• If yes, parse using parse_strength_modifier() and attach to previous rule
• If no, proceed with normal rule parsing

Modify parse_text_magic_file() to handle strength directives:

• Track the last parsed rule
• When !:strength is encountered, attach modifier to last rule
• Validate that !:strength doesn't appear before any rules

4. Implement Default Strength Calculation Algorithm

File: file:src/parser/mod.rs

Create calculate_default_strength(rule: &MagicRule) -> i32 function implementing libmagic's algorithm:

Type Contribution:

• TypeKind::String → +20 points
• TypeKind::Long → +15 points
• TypeKind::Short → +10 points
• TypeKind::Byte → +5 points

Operator Contribution:

• Operator::Equal → +10 points
• Operator::NotEqual → +5 points
• Operator::BitwiseAnd / Operator::BitwiseAndMask → +3 points

Offset Contribution:

• OffsetSpec::Absolute → +10 points
• OffsetSpec::Indirect → +5 points
• OffsetSpec::Relative → +3 points
• OffsetSpec::FromEnd → +3 points

Value Length Contribution (for strings):

• Add min(value_length, 20) for string types

Start with baseline of 0 and sum all contributions.

5. Implement Strength Modifier Application

File: file:src/parser/mod.rs

Create apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32:

• Match on modifier variant
• Apply operation: Add → base + n, Subtract → base - n, Multiply → base * n, Divide → base / n, Set → n
• Clamp result to valid range (0-255)
• Handle division by zero gracefully

Add comprehensive unit tests for:

• Each operation type
• Boundary conditions (overflow, underflow)
• Clamping behavior

6. Integrate Strength Calculation into Parser Pipeline

File: file:src/parser/mod.rs

Update build_rule_hierarchy() or create post-processing step:

• After parsing all rules, iterate through rule tree
• For each rule, calculate default strength using calculate_default_strength()
• If strength_modifier is present, apply using apply_strength_modifier()
• Store final value in calculated_strength field

Ensure child rules inherit or calculate their own strength independently.

7. Use Strength for Rule Ordering in Evaluator

File: file:src/evaluator/mod.rs

Create sort_rules_by_strength(rules: &mut [MagicRule]):

• Sort rules by calculated_strength in descending order (higher strength first)
• Preserve original order for rules with equal strength
• Apply recursively to child rules

Integrate into evaluate_rules():

• Sort rules before evaluation when stop_at_first_match is false
• Use strength to determine which match to return when multiple matches occur

8. Expose Strength in Evaluation Results

File: file:src/evaluator/mod.rs

Update MatchResult struct to include strength: i32 field.

Populate strength value during rule evaluation in evaluate_single_rule().

File: file:src/output/json.rs

Include strength in JSON output as "strength" field alongside existing fields.

9. Add Comprehensive Testing

File: file:tests/strength_tests.rs (new file)

Create unit tests for:

• Strength modifier parsing (all operators, edge cases)
• Default strength calculation (all type/operator/offset combinations)
• Strength modifier application (all operations, clamping)
• Rule sorting by strength
• Integration test comparing with GNU file output using file -l command

Add property-based tests using proptest crate to verify:

• Strength values always in valid range
• Sorting is stable and correct
• Modifier application is associative where applicable

10. Update Documentation

File: file:docs/src/magic-format.md

Add section documenting !:strength directive:

• Syntax and semantics
• Examples of each operator
• Explanation of default strength calculation
• Best practices for using strength modifiers

File: src/parser/ast.rs, src/parser/mod.rs

Add rustdoc comments to all new types and functions:

• Explain strength calculation algorithm with examples
• Provide usage examples for each modifier type
• Document valid ranges (0-255) and error conditions
• Reference libmagic's apprentice_magic_strength algorithm

Acceptance Criteria Mapping

• !:strength modifiers parsed correctly → Steps 2, 3, 9
• Default strength calculated for all rule types → Steps 4, 9
• Strength modifiers applied correctly → Steps 5, 9
• Rules sorted by strength during evaluation → Steps 7, 9
• Strength affects confidence scoring → Steps 8, 9
• Unit tests for strength parsing → Step 9
• Unit tests for default strength calculation → Step 9
• Unit tests for strength modifiers → Step 9
• Integration test comparing with GNU file output → Step 9
• Compatibility test shows improved match rate → Step 9

Sequence Diagram

Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (`file:src/parser/grammar.rs`, `file:src/parser/ast.rs`) and evaluator infrastructure (`file:src/evaluator/mod.rs`). The `MagicRule` structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's `apprentice_magic_strength` algorithm with `!:strength` modifier parsing. The baseline strength starts at 20, with contributions from type specificity, operator specificity, offset type, and value length.

## Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for `!:strength` directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

## Implementation Steps

### 1. Extend AST Structures for Strength Support

**File: `file:src/parser/ast.rs`**

Add `StrengthModifier` enum to represent the five strength operations:
- `Add(i32)` for `!:strength +N`
- `Subtract(i32)` for `!:strength -N`
- `Multiply(i32)` for `!:strength *N`
- `Divide(i32)` for `!:strength /N`
- `Set(i32)` for `!:strength =N`

Update `MagicRule` struct to include:
- `strength_modifier: Option<StrengthModifier>` - stores parsed `!:strength` directive
- `calculated_strength: Option<i32>` - stores final computed strength value

Add `impl` blocks for `StrengthModifier` with validation methods ensuring values are within 0-255 range.

### 2. Implement Strength Modifier Parsing

**File: `file:src/parser/grammar.rs`**

Create `parse_strength_modifier()` function using nom combinators:
- Match the literal `!:strength` prefix
- Parse operator: one of `+`, `-`, `*`, `/`, `=`
- Parse numeric value (0-255 range)
- Return `StrengthModifier` enum variant
- Handle whitespace around operators

Add unit tests covering:
- All five operator types
- Edge cases (0, 255, out-of-range values)
- Whitespace variations
- Invalid syntax error cases

### 3. Integrate Strength Parsing into Rule Parser

**File: `file:src/parser/mod.rs`**

Update `parse_magic_rule_line()` to detect and parse `!:strength` lines:
- Check if line starts with `!:strength`
- If yes, parse using `parse_strength_modifier()` and attach to previous rule
- If no, proceed with normal rule parsing

Modify `parse_text_magic_file()` to handle strength directives:
- Track the last parsed rule
- When `!:strength` is encountered, attach modifier to last rule
- Validate that `!:strength` doesn't appear before any rules

### 4. Implement Default Strength Calculation Algorithm

**File: `file:src/parser/mod.rs`**

Create `calculate_default_strength(rule: &MagicRule) -> i32` function implementing libmagic's algorithm:

**Type Contribution:**
- `TypeKind::String` → +20 points
- `TypeKind::Long` → +15 points
- `TypeKind::Short` → +10 points
- `TypeKind::Byte` → +5 points

**Operator Contribution:**
- `Operator::Equal` → +10 points
- `Operator::NotEqual` → +5 points
- `Operator::BitwiseAnd` / `Operator::BitwiseAndMask` → +3 points

**Offset Contribution:**
- `OffsetSpec::Absolute` → +10 points
- `OffsetSpec::Indirect` → +5 points
- `OffsetSpec::Relative` → +3 points
- `OffsetSpec::FromEnd` → +3 points

**Value Length Contribution (for strings):**
- Add `min(value_length, 20)` for string types

Start with baseline of 0 and sum all contributions.

### 5. Implement Strength Modifier Application

**File: `file:src/parser/mod.rs`**

Create `apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32`:
- Match on modifier variant
- Apply operation: `Add` → `base + n`, `Subtract` → `base - n`, `Multiply` → `base * n`, `Divide` → `base / n`, `Set` → `n`
- Clamp result to valid range (0-255)
- Handle division by zero gracefully

Add comprehensive unit tests for:
- Each operation type
- Boundary conditions (overflow, underflow)
- Clamping behavior

### 6. Integrate Strength Calculation into Parser Pipeline

**File: `file:src/parser/mod.rs`**

Update `build_rule_hierarchy()` or create post-processing step:
- After parsing all rules, iterate through rule tree
- For each rule, calculate default strength using `calculate_default_strength()`
- If `strength_modifier` is present, apply using `apply_strength_modifier()`
- Store final value in `calculated_strength` field

Ensure child rules inherit or calculate their own strength independently.

### 7. Use Strength for Rule Ordering in Evaluator

**File: `file:src/evaluator/mod.rs`**

Create `sort_rules_by_strength(rules: &mut [MagicRule])`:
- Sort rules by `calculated_strength` in descending order (higher strength first)
- Preserve original order for rules with equal strength
- Apply recursively to child rules

Integrate into `evaluate_rules()`:
- Sort rules before evaluation when `stop_at_first_match` is false
- Use strength to determine which match to return when multiple matches occur

### 8. Expose Strength in Evaluation Results

**File: `file:src/evaluator/mod.rs`**

Update `MatchResult` struct to include `strength: i32` field.

Populate strength value during rule evaluation in `evaluate_single_rule()`.

**File: `file:src/output/json.rs`**

Include strength in JSON output as `"strength"` field alongside existing fields.

### 9. Add Comprehensive Testing

**File: `file:tests/strength_tests.rs`** (new file)

Create unit tests for:
- Strength modifier parsing (all operators, edge cases)
- Default strength calculation (all type/operator/offset combinations)
- Strength modifier application (all operations, clamping)
- Rule sorting by strength
- Integration test comparing with GNU file output using `file -l` command

Add property-based tests using `proptest` crate to verify:
- Strength values always in valid range
- Sorting is stable and correct
- Modifier application is associative where applicable

### 10. Update Documentation

**File: `file:docs/src/magic-format.md`**

Add section documenting `!:strength` directive:
- Syntax and semantics
- Examples of each operator
- Explanation of default strength calculation
- Best practices for using strength modifiers

**File: `src/parser/ast.rs`, `src/parser/mod.rs`**

Add rustdoc comments to all new types and functions:
- Explain strength calculation algorithm with examples
- Provide usage examples for each modifier type
- Document valid ranges (0-255) and error conditions
- Reference libmagic's `apprentice_magic_strength` algorithm

## Acceptance Criteria Mapping

- `!:strength` modifiers parsed correctly → Steps 2, 3, 9
- Default strength calculated for all rule types → Steps 4, 9
- Strength modifiers applied correctly → Steps 5, 9
- Rules sorted by strength during evaluation → Steps 7, 9
- Strength affects confidence scoring → Steps 8, 9
- Unit tests for strength parsing → Step 9
- Unit tests for default strength calculation → Step 9
- Unit tests for strength modifiers → Step 9
- Integration test comparing with GNU file output → Step 9
- Compatibility test shows improved match rate → Step 9

## Sequence Diagram

```
Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field
```

---

Execution Information

Branch: main
Commit: 50b370c

[unclesp1d3r]

@coderabbitai Assign the proper labels and other metadata to this issue

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# First, let's see what labels are available in the repository
echo "=== Available labels in repository ==="
gh label list --repo EvilBit-Labs/libmagic-rs --limit 100

echo -e "\n=== Current issue `#21` details ==="
gh issue view 21 --repo EvilBit-Labs/libmagic-rs --json title,labels,assignees,milestone,projectItems

echo -e "\n=== Checking if milestones exist ==="
gh api repos/EvilBit-Labs/libmagic-rs/milestones --jq '.[] | "\(.number): \(.title)"'

Length of output: 1281

---

🏁 Script executed:

#!/bin/bash

# Assign labels to issue `#21`
gh issue edit 21 --repo EvilBit-Labs/libmagic-rs \
  --add-label "enhancement" \
  --add-label "parser" \
  --add-label "evaluator" \
  --add-label "compatibility"

# Assign to unclesp1d3r
gh issue edit 21 --repo EvilBit-Labs/libmagic-rs \
  --add-assignee "unclesp1d3r"

# Set milestone to Phase 1 MVP
gh issue edit 21 --repo EvilBit-Labs/libmagic-rs \
  --milestone "Phase 1 MVP"

# Show the updated issue
echo -e "\n=== Updated issue `#21` ==="
gh issue view 21 --repo EvilBit-Labs/libmagic-rs --json title,labels,assignees,milestone --jq '.'

Length of output: 1034

---

@unclesp1d3r, I've successfully assigned the appropriate labels and metadata to issue #21:

Labels assigned:

• enhancement - New feature request
• parser - Involves magic file parsing components
• evaluator - Affects rule evaluation engine
• compatibility - Critical for libmagic compatibility

Other metadata:

• Assignee: unclesp1d3r
• Milestone: Phase 1 MVP

The issue is now properly categorized and ready for implementation. The labels reflect that this is a cross-cutting enhancement affecting both the parser (for !:strength directive parsing) and evaluator (for strength-based rule ordering), with significant impact on libmagic compatibility.

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

Previous Plan

Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (file:src/parser/grammar.rs, file:src/parser/ast.rs) and evaluator infrastructure (file:src/evaluator/mod.rs). The MagicRule structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's apprentice_magic_strength algorithm with !:strength modifier parsing. The baseline strength starts at 20, with contributions from type specificity, operator specificity, offset type, and value length.

Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for !:strength directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

Implementation Steps

1. Extend AST Structures for Strength Support

File: file:src/parser/ast.rs

Add StrengthModifier enum to represent the five strength operations:

• Add(i32) for !:strength +N
• Subtract(i32) for !:strength -N
• Multiply(i32) for !:strength *N
• Divide(i32) for !:strength /N
• Set(i32) for !:strength =N

Update MagicRule struct to include:

• strength_modifier: Option<StrengthModifier> - stores parsed !:strength directive
• calculated_strength: Option<i32> - stores final computed strength value

Add impl blocks for StrengthModifier with validation methods ensuring values are within 0-255 range.

2. Implement Strength Modifier Parsing

File: file:src/parser/grammar.rs

Create parse_strength_modifier() function using nom combinators:

• Match the literal !:strength prefix
• Parse operator: one of +, -, *, /, =
• Parse numeric value (0-255 range)
• Return StrengthModifier enum variant
• Handle whitespace around operators

Add unit tests covering:

• All five operator types
• Edge cases (0, 255, out-of-range values)
• Whitespace variations
• Invalid syntax error cases

3. Integrate Strength Parsing into Rule Parser

File: file:src/parser/mod.rs

Update parse_magic_rule_line() to detect and parse !:strength lines:

• Check if line starts with !:strength
• If yes, parse using parse_strength_modifier() and attach to previous rule
• If no, proceed with normal rule parsing

Modify parse_text_magic_file() to handle strength directives:

• Track the last parsed rule
• When !:strength is encountered, attach modifier to last rule
• Validate that !:strength doesn't appear before any rules

4. Implement Default Strength Calculation Algorithm

File: file:src/parser/mod.rs

Create calculate_default_strength(rule: &MagicRule) -> i32 function implementing libmagic's algorithm:

Type Contribution:

• TypeKind::String → +20 points
• TypeKind::Long → +15 points
• TypeKind::Short → +10 points
• TypeKind::Byte → +5 points

Operator Contribution:

• Operator::Equal → +10 points
• Operator::NotEqual → +5 points
• Operator::BitwiseAnd / Operator::BitwiseAndMask → +3 points

Offset Contribution:

• OffsetSpec::Absolute → +10 points
• OffsetSpec::Indirect → +5 points
• OffsetSpec::Relative → +3 points
• OffsetSpec::FromEnd → +3 points

Value Length Contribution (for strings):

• Add min(value_length, 20) for string types

Start with baseline of 0 and sum all contributions.

5. Implement Strength Modifier Application

File: file:src/parser/mod.rs

Create apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32:

• Match on modifier variant
• Apply operation: Add → base + n, Subtract → base - n, Multiply → base * n, Divide → base / n, Set → n
• Clamp result to valid range (0-255)
• Handle division by zero gracefully

Add comprehensive unit tests for:

• Each operation type
• Boundary conditions (overflow, underflow)
• Clamping behavior

6. Integrate Strength Calculation into Parser Pipeline

File: file:src/parser/mod.rs

Update build_rule_hierarchy() or create post-processing step:

• After parsing all rules, iterate through rule tree
• For each rule, calculate default strength using calculate_default_strength()
• If strength_modifier is present, apply using apply_strength_modifier()
• Store final value in calculated_strength field

Ensure child rules inherit or calculate their own strength independently.

7. Use Strength for Rule Ordering in Evaluator

File: file:src/evaluator/mod.rs

Create sort_rules_by_strength(rules: &mut [MagicRule]):

• Sort rules by calculated_strength in descending order (higher strength first)
• Preserve original order for rules with equal strength
• Apply recursively to child rules

Integrate into evaluate_rules():

• Sort rules before evaluation when stop_at_first_match is false
• Use strength to determine which match to return when multiple matches occur

8. Expose Strength in Evaluation Results

File: file:src/evaluator/mod.rs

Update MatchResult struct to include strength: i32 field.

Populate strength value during rule evaluation in evaluate_single_rule().

File: file:src/output/json.rs

Include strength in JSON output as "strength" field alongside existing fields.

9. Add Comprehensive Testing

File: file:tests/strength_tests.rs (new file)

Create unit tests for:

• Strength modifier parsing (all operators, edge cases)
• Default strength calculation (all type/operator/offset combinations)
• Strength modifier application (all operations, clamping)
• Rule sorting by strength
• Integration test comparing with GNU file output using file -l command

Add property-based tests using proptest crate to verify:

• Strength values always in valid range
• Sorting is stable and correct
• Modifier application is associative where applicable

10. Update Documentation

File: file:docs/src/magic-format.md

Add section documenting !:strength directive:

• Syntax and semantics
• Examples of each operator
• Explanation of default strength calculation
• Best practices for using strength modifiers

File: src/parser/ast.rs, src/parser/mod.rs

Add rustdoc comments to all new types and functions:

• Explain strength calculation algorithm with examples
• Provide usage examples for each modifier type
• Document valid ranges (0-255) and error conditions
• Reference libmagic's apprentice_magic_strength algorithm

Acceptance Criteria Mapping

• !:strength modifiers parsed correctly → Steps 2, 3, 9
• Default strength calculated for all rule types → Steps 4, 9
• Strength modifiers applied correctly → Steps 5, 9
• Rules sorted by strength during evaluation → Steps 7, 9
• Strength affects confidence scoring → Steps 8, 9
• Unit tests for strength parsing → Step 9
• Unit tests for default strength calculation → Step 9
• Unit tests for strength modifiers → Step 9
• Integration test comparing with GNU file output → Step 9
• Compatibility test shows improved match rate → Step 9

Sequence Diagram

Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (`file:src/parser/grammar.rs`, `file:src/parser/ast.rs`) and evaluator infrastructure (`file:src/evaluator/mod.rs`). The `MagicRule` structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's `apprentice_magic_strength` algorithm with `!:strength` modifier parsing. The baseline strength starts at 20, with contributions from type specificity, operator specificity, offset type, and value length.

## Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for `!:strength` directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

## Implementation Steps

### 1. Extend AST Structures for Strength Support

**File: `file:src/parser/ast.rs`**

Add `StrengthModifier` enum to represent the five strength operations:
- `Add(i32)` for `!:strength +N`
- `Subtract(i32)` for `!:strength -N`
- `Multiply(i32)` for `!:strength *N`
- `Divide(i32)` for `!:strength /N`
- `Set(i32)` for `!:strength =N`

Update `MagicRule` struct to include:
- `strength_modifier: Option<StrengthModifier>` - stores parsed `!:strength` directive
- `calculated_strength: Option<i32>` - stores final computed strength value

Add `impl` blocks for `StrengthModifier` with validation methods ensuring values are within 0-255 range.

### 2. Implement Strength Modifier Parsing

**File: `file:src/parser/grammar.rs`**

Create `parse_strength_modifier()` function using nom combinators:
- Match the literal `!:strength` prefix
- Parse operator: one of `+`, `-`, `*`, `/`, `=`
- Parse numeric value (0-255 range)
- Return `StrengthModifier` enum variant
- Handle whitespace around operators

Add unit tests covering:
- All five operator types
- Edge cases (0, 255, out-of-range values)
- Whitespace variations
- Invalid syntax error cases

### 3. Integrate Strength Parsing into Rule Parser

**File: `file:src/parser/mod.rs`**

Update `parse_magic_rule_line()` to detect and parse `!:strength` lines:
- Check if line starts with `!:strength`
- If yes, parse using `parse_strength_modifier()` and attach to previous rule
- If no, proceed with normal rule parsing

Modify `parse_text_magic_file()` to handle strength directives:
- Track the last parsed rule
- When `!:strength` is encountered, attach modifier to last rule
- Validate that `!:strength` doesn't appear before any rules

### 4. Implement Default Strength Calculation Algorithm

**File: `file:src/parser/mod.rs`**

Create `calculate_default_strength(rule: &MagicRule) -> i32` function implementing libmagic's algorithm:

**Type Contribution:**
- `TypeKind::String` → +20 points
- `TypeKind::Long` → +15 points
- `TypeKind::Short` → +10 points
- `TypeKind::Byte` → +5 points

**Operator Contribution:**
- `Operator::Equal` → +10 points
- `Operator::NotEqual` → +5 points
- `Operator::BitwiseAnd` / `Operator::BitwiseAndMask` → +3 points

**Offset Contribution:**
- `OffsetSpec::Absolute` → +10 points
- `OffsetSpec::Indirect` → +5 points
- `OffsetSpec::Relative` → +3 points
- `OffsetSpec::FromEnd` → +3 points

**Value Length Contribution (for strings):**
- Add `min(value_length, 20)` for string types

Start with baseline of 0 and sum all contributions.

### 5. Implement Strength Modifier Application

**File: `file:src/parser/mod.rs`**

Create `apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32`:
- Match on modifier variant
- Apply operation: `Add` → `base + n`, `Subtract` → `base - n`, `Multiply` → `base * n`, `Divide` → `base / n`, `Set` → `n`
- Clamp result to valid range (0-255)
- Handle division by zero gracefully

Add comprehensive unit tests for:
- Each operation type
- Boundary conditions (overflow, underflow)
- Clamping behavior

### 6. Integrate Strength Calculation into Parser Pipeline

**File: `file:src/parser/mod.rs`**

Update `build_rule_hierarchy()` or create post-processing step:
- After parsing all rules, iterate through rule tree
- For each rule, calculate default strength using `calculate_default_strength()`
- If `strength_modifier` is present, apply using `apply_strength_modifier()`
- Store final value in `calculated_strength` field

Ensure child rules inherit or calculate their own strength independently.

### 7. Use Strength for Rule Ordering in Evaluator

**File: `file:src/evaluator/mod.rs`**

Create `sort_rules_by_strength(rules: &mut [MagicRule])`:
- Sort rules by `calculated_strength` in descending order (higher strength first)
- Preserve original order for rules with equal strength
- Apply recursively to child rules

Integrate into `evaluate_rules()`:
- Sort rules before evaluation when `stop_at_first_match` is false
- Use strength to determine which match to return when multiple matches occur

### 8. Expose Strength in Evaluation Results

**File: `file:src/evaluator/mod.rs`**

Update `MatchResult` struct to include `strength: i32` field.

Populate strength value during rule evaluation in `evaluate_single_rule()`.

**File: `file:src/output/json.rs`**

Include strength in JSON output as `"strength"` field alongside existing fields.

### 9. Add Comprehensive Testing

**File: `file:tests/strength_tests.rs`** (new file)

Create unit tests for:
- Strength modifier parsing (all operators, edge cases)
- Default strength calculation (all type/operator/offset combinations)
- Strength modifier application (all operations, clamping)
- Rule sorting by strength
- Integration test comparing with GNU file output using `file -l` command

Add property-based tests using `proptest` crate to verify:
- Strength values always in valid range
- Sorting is stable and correct
- Modifier application is associative where applicable

### 10. Update Documentation

**File: `file:docs/src/magic-format.md`**

Add section documenting `!:strength` directive:
- Syntax and semantics
- Examples of each operator
- Explanation of default strength calculation
- Best practices for using strength modifiers

**File: `src/parser/ast.rs`, `src/parser/mod.rs`**

Add rustdoc comments to all new types and functions:
- Explain strength calculation algorithm with examples
- Provide usage examples for each modifier type
- Document valid ranges (0-255) and error conditions
- Reference libmagic's `apprentice_magic_strength` algorithm

## Acceptance Criteria Mapping

- `!:strength` modifiers parsed correctly → Steps 2, 3, 9
- Default strength calculated for all rule types → Steps 4, 9
- Strength modifiers applied correctly → Steps 5, 9
- Rules sorted by strength during evaluation → Steps 7, 9
- Strength affects confidence scoring → Steps 8, 9
- Unit tests for strength parsing → Step 9
- Unit tests for default strength calculation → Step 9
- Unit tests for strength modifiers → Step 9
- Integration test comparing with GNU file output → Step 9
- Compatibility test shows improved match rate → Step 9

## Sequence Diagram

```
Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field
```

---

Execution Information

Branch: main
Commit: 50b370c

New Plan

Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (file:src/parser/grammar.rs, file:src/parser/ast.rs) and evaluator infrastructure (file:src/evaluator/mod.rs). The MagicRule structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's apprentice_magic_strength algorithm with !:strength modifier parsing. The baseline strength starts at 0, with contributions from type specificity, operator specificity, offset type, and value length.

Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for !:strength directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

Implementation Steps

1. Extend AST Structures for Strength Support

File: file:src/parser/ast.rs

Add StrengthModifier enum to represent the five strength operations:

• Add(i32) for !:strength +N
• Subtract(i32) for !:strength -N
• Multiply(i32) for !:strength *N
• Divide(i32) for !:strength /N
• Set(i32) for !:strength =N

Update MagicRule struct to include:

• strength_modifier: Option<StrengthModifier> - stores parsed !:strength directive
• calculated_strength: Option<i32> - stores final computed strength value

Add impl blocks for StrengthModifier with validation methods ensuring values are within 0-255 range.

2. Implement Strength Modifier Parsing

File: file:src/parser/grammar.rs

Create parse_strength_modifier() function using nom combinators:

• Match the literal !:strength prefix
• Parse operator: one of +, -, *, /, =
• Parse numeric value (0-255 range)
• Return StrengthModifier enum variant
• Handle whitespace around operators

Add unit tests covering:

• All five operator types
• Edge cases (0, 255, out-of-range values)
• Whitespace variations
• Invalid syntax error cases

3. Integrate Strength Parsing into Rule Parser

File: file:src/parser/mod.rs

Update parse_text_magic_file() to handle strength directives:

• Track the last parsed rule during preprocessing
• When a line starts with !:strength, parse using parse_strength_modifier()
• Attach modifier to the last parsed rule in the hierarchy
• Validate that !:strength doesn't appear before any rules
• Preserve line number information for error reporting

Create helper function attach_strength_modifier_to_last_rule():

• Traverses rule hierarchy to find the last rule at appropriate level
• Attaches strength modifier to that rule
• Returns error if no previous rule exists

4. Implement Default Strength Calculation Algorithm

File: file:src/parser/mod.rs

Create calculate_default_strength(rule: &MagicRule) -> i32 function implementing libmagic's algorithm:

Type Contribution:

• TypeKind::String → +20 points
• TypeKind::Long → +15 points
• TypeKind::Short → +10 points
• TypeKind::Byte → +5 points

Operator Contribution:

• Operator::Equal → +10 points
• Operator::NotEqual → +5 points
• Operator::BitwiseAnd / Operator::BitwiseAndMask → +3 points

Offset Contribution:

• OffsetSpec::Absolute → +10 points
• OffsetSpec::Indirect → +5 points
• OffsetSpec::Relative → +3 points
• OffsetSpec::FromEnd → +3 points

Value Length Contribution (for strings):

• Add min(value_length, 20) for string types

Start with baseline of 0 and sum all contributions. Clamp final result to valid range (0-255).

5. Implement Strength Modifier Application

File: file:src/parser/mod.rs

Create apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32:

• Match on modifier variant
• Apply operation: Add → base + n, Subtract → base - n, Multiply → base * n, Divide → base / n, Set → n
• Clamp result to valid range (0-255)
• Handle division by zero gracefully (return base unchanged)
• Validate modifier values are within 0-255 range

Add comprehensive unit tests for:

• Each operation type
• Boundary conditions (overflow, underflow)
• Clamping behavior
• Division by zero handling

6. Integrate Strength Calculation into Parser Pipeline

File: file:src/parser/mod.rs

Create calculate_rule_strengths() post-processing function:

• After build_rule_hierarchy() completes, traverse entire rule tree
• For each rule, calculate default strength using calculate_default_strength()
• If strength_modifier is present, apply using apply_strength_modifier()
• Store final value in calculated_strength field
• Recursively process child rules (each calculates independently)

Integrate into parse_text_magic_file():

• Call calculate_rule_strengths() after build_rule_hierarchy()
• Ensure all rules have strength values before returning

7. Use Strength for Rule Ordering in Evaluator

File: file:src/evaluator/mod.rs

Create sort_rules_by_strength(rules: &mut [MagicRule]):

• Sort rules by calculated_strength in descending order (higher strength first)
• Preserve original order for rules with equal strength (stable sort)
• Apply recursively to child rules before returning

Integrate into evaluate_rules():

• Sort sibling rules by strength before iterating through them
• When multiple rules match at same level, strength determines priority
• Higher strength matches are returned first
• Strength affects confidence scoring in results

8. Expose Strength in Evaluation Results

File: file:src/evaluator/mod.rs

Update MatchResult struct to include strength: i32 field.

Populate strength value during rule evaluation:

• In evaluate_single_rule(), retrieve calculated_strength from matched rule
• Pass strength to MatchResult constructor
• Ensure strength is available for all match results

File: file:src/output/json.rs and file:src/output/text.rs

Include strength in output:

• JSON output: add "strength" field alongside existing fields
• Text output: optionally display strength with -l flag compatibility

9. Add Comprehensive Testing

File: file:tests/strength_tests.rs (new file)

Create unit tests for:

• Strength modifier parsing (all operators, edge cases, whitespace)
• Default strength calculation (all type/operator/offset combinations)
• Strength modifier application (all operations, clamping, division by zero)
• Rule sorting by strength (stable sort, equal strength preservation)
• Strength calculation with child rules (independent calculation)
• Integration test with real magic files containing !:strength directives

Add integration tests:

• Parse magic files with !:strength modifiers
• Verify strength values are calculated correctly
• Compare rule ordering with and without strength
• Test compatibility with GNU file output using file -l command

Add edge case tests:

• Strength modifiers on rules with different types
• Multiple strength modifiers (last one wins)
• Out-of-range strength values (clamping behavior)
• Division by zero in strength calculation

10. Update Documentation

File: file:docs/src/magic-format.md or file:README.md

Add comprehensive section documenting !:strength directive:

• Syntax: !:strength OP VALUE where OP is one of +, -, *, /, =
• Semantics: modifies strength of preceding rule
• Examples of each operator with expected results
• Explanation of default strength calculation algorithm
• Best practices for using strength modifiers
• Impact on rule priority and matching behavior

File: src/parser/ast.rs

Add rustdoc comments to StrengthModifier enum:

• Document each variant with examples
• Explain valid value ranges (0-255)
• Describe clamping behavior

File: src/parser/mod.rs

Add rustdoc comments to strength functions:

• calculate_default_strength(): explain algorithm, type/operator/offset contributions
• apply_strength_modifier(): explain operations, clamping, division by zero handling
• calculate_rule_strengths(): explain tree traversal and recursive calculation
• Provide usage examples for each function
• Reference libmagic's apprentice_magic_strength algorithm

Acceptance Criteria Mapping

• !:strength modifiers parsed correctly → Steps 2, 3, 9
• Default strength calculated for all rule types → Steps 4, 9
• Strength modifiers applied correctly → Steps 5, 9
• Rules sorted by strength during evaluation → Steps 7, 9
• Strength affects confidence scoring → Steps 8, 9
• Unit tests for strength parsing → Step 9
• Unit tests for default strength calculation → Step 9
• Unit tests for strength modifiers → Step 9
• Integration test comparing with GNU file output → Step 9
• Compatibility test shows improved match rate → Step 9

Sequence Diagram

Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase is a pure-Rust libmagic implementation with complete parser components (`file:src/parser/grammar.rs`, `file:src/parser/ast.rs`) and evaluator infrastructure (`file:src/evaluator/mod.rs`). The `MagicRule` structure currently lacks strength-related fields. Issue #21 provides detailed specifications for implementing libmagic's `apprentice_magic_strength` algorithm with `!:strength` modifier parsing. The baseline strength starts at 0, with contributions from type specificity, operator specificity, offset type, and value length.

## Approach

Implement strength calculation in three phases: (1) extend AST structures to store strength modifiers and calculated strength values, (2) add parser support for `!:strength` directive syntax, and (3) implement the default strength calculation algorithm based on rule characteristics. The implementation follows libmagic's approach where strength affects rule priority and matching confidence, critical for achieving 95%+ GNU file compatibility.

## Implementation Steps

### 1. Extend AST Structures for Strength Support

**File: `file:src/parser/ast.rs`**

Add `StrengthModifier` enum to represent the five strength operations:
- `Add(i32)` for `!:strength +N`
- `Subtract(i32)` for `!:strength -N`
- `Multiply(i32)` for `!:strength *N`
- `Divide(i32)` for `!:strength /N`
- `Set(i32)` for `!:strength =N`

Update `MagicRule` struct to include:
- `strength_modifier: Option<StrengthModifier>` - stores parsed `!:strength` directive
- `calculated_strength: Option<i32>` - stores final computed strength value

Add `impl` blocks for `StrengthModifier` with validation methods ensuring values are within 0-255 range.

### 2. Implement Strength Modifier Parsing

**File: `file:src/parser/grammar.rs`**

Create `parse_strength_modifier()` function using nom combinators:
- Match the literal `!:strength` prefix
- Parse operator: one of `+`, `-`, `*`, `/`, `=`
- Parse numeric value (0-255 range)
- Return `StrengthModifier` enum variant
- Handle whitespace around operators

Add unit tests covering:
- All five operator types
- Edge cases (0, 255, out-of-range values)
- Whitespace variations
- Invalid syntax error cases

### 3. Integrate Strength Parsing into Rule Parser

**File: `file:src/parser/mod.rs`**

Update `parse_text_magic_file()` to handle strength directives:
- Track the last parsed rule during preprocessing
- When a line starts with `!:strength`, parse using `parse_strength_modifier()`
- Attach modifier to the last parsed rule in the hierarchy
- Validate that `!:strength` doesn't appear before any rules
- Preserve line number information for error reporting

Create helper function `attach_strength_modifier_to_last_rule()`:
- Traverses rule hierarchy to find the last rule at appropriate level
- Attaches strength modifier to that rule
- Returns error if no previous rule exists

### 4. Implement Default Strength Calculation Algorithm

**File: `file:src/parser/mod.rs`**

Create `calculate_default_strength(rule: &MagicRule) -> i32` function implementing libmagic's algorithm:

**Type Contribution:**
- `TypeKind::String` → +20 points
- `TypeKind::Long` → +15 points
- `TypeKind::Short` → +10 points
- `TypeKind::Byte` → +5 points

**Operator Contribution:**
- `Operator::Equal` → +10 points
- `Operator::NotEqual` → +5 points
- `Operator::BitwiseAnd` / `Operator::BitwiseAndMask` → +3 points

**Offset Contribution:**
- `OffsetSpec::Absolute` → +10 points
- `OffsetSpec::Indirect` → +5 points
- `OffsetSpec::Relative` → +3 points
- `OffsetSpec::FromEnd` → +3 points

**Value Length Contribution (for strings):**
- Add `min(value_length, 20)` for string types

Start with baseline of 0 and sum all contributions. Clamp final result to valid range (0-255).

### 5. Implement Strength Modifier Application

**File: `file:src/parser/mod.rs`**

Create `apply_strength_modifier(base: i32, modifier: &StrengthModifier) -> i32`:
- Match on modifier variant
- Apply operation: `Add` → `base + n`, `Subtract` → `base - n`, `Multiply` → `base * n`, `Divide` → `base / n`, `Set` → `n`
- Clamp result to valid range (0-255)
- Handle division by zero gracefully (return base unchanged)
- Validate modifier values are within 0-255 range

Add comprehensive unit tests for:
- Each operation type
- Boundary conditions (overflow, underflow)
- Clamping behavior
- Division by zero handling

### 6. Integrate Strength Calculation into Parser Pipeline

**File: `file:src/parser/mod.rs`**

Create `calculate_rule_strengths()` post-processing function:
- After `build_rule_hierarchy()` completes, traverse entire rule tree
- For each rule, calculate default strength using `calculate_default_strength()`
- If `strength_modifier` is present, apply using `apply_strength_modifier()`
- Store final value in `calculated_strength` field
- Recursively process child rules (each calculates independently)

Integrate into `parse_text_magic_file()`:
- Call `calculate_rule_strengths()` after `build_rule_hierarchy()`
- Ensure all rules have strength values before returning

### 7. Use Strength for Rule Ordering in Evaluator

**File: `file:src/evaluator/mod.rs`**

Create `sort_rules_by_strength(rules: &mut [MagicRule])`:
- Sort rules by `calculated_strength` in descending order (higher strength first)
- Preserve original order for rules with equal strength (stable sort)
- Apply recursively to child rules before returning

Integrate into `evaluate_rules()`:
- Sort sibling rules by strength before iterating through them
- When multiple rules match at same level, strength determines priority
- Higher strength matches are returned first
- Strength affects confidence scoring in results

### 8. Expose Strength in Evaluation Results

**File: `file:src/evaluator/mod.rs`**

Update `MatchResult` struct to include `strength: i32` field.

Populate strength value during rule evaluation:
- In `evaluate_single_rule()`, retrieve `calculated_strength` from matched rule
- Pass strength to `MatchResult` constructor
- Ensure strength is available for all match results

**File: `file:src/output/json.rs`** and **`file:src/output/text.rs`**

Include strength in output:
- JSON output: add `"strength"` field alongside existing fields
- Text output: optionally display strength with `-l` flag compatibility

### 9. Add Comprehensive Testing

**File: `file:tests/strength_tests.rs`** (new file)

Create unit tests for:
- Strength modifier parsing (all operators, edge cases, whitespace)
- Default strength calculation (all type/operator/offset combinations)
- Strength modifier application (all operations, clamping, division by zero)
- Rule sorting by strength (stable sort, equal strength preservation)
- Strength calculation with child rules (independent calculation)
- Integration test with real magic files containing `!:strength` directives

Add integration tests:
- Parse magic files with `!:strength` modifiers
- Verify strength values are calculated correctly
- Compare rule ordering with and without strength
- Test compatibility with GNU file output using `file -l` command

Add edge case tests:
- Strength modifiers on rules with different types
- Multiple strength modifiers (last one wins)
- Out-of-range strength values (clamping behavior)
- Division by zero in strength calculation

### 10. Update Documentation

**File: `file:docs/src/magic-format.md`** or **`file:README.md`**

Add comprehensive section documenting `!:strength` directive:
- Syntax: `!:strength OP VALUE` where OP is one of `+`, `-`, `*`, `/`, `=`
- Semantics: modifies strength of preceding rule
- Examples of each operator with expected results
- Explanation of default strength calculation algorithm
- Best practices for using strength modifiers
- Impact on rule priority and matching behavior

**File: `src/parser/ast.rs`**

Add rustdoc comments to `StrengthModifier` enum:
- Document each variant with examples
- Explain valid value ranges (0-255)
- Describe clamping behavior

**File: `src/parser/mod.rs`**

Add rustdoc comments to strength functions:
- `calculate_default_strength()`: explain algorithm, type/operator/offset contributions
- `apply_strength_modifier()`: explain operations, clamping, division by zero handling
- `calculate_rule_strengths()`: explain tree traversal and recursive calculation
- Provide usage examples for each function
- Reference libmagic's `apprentice_magic_strength` algorithm

## Acceptance Criteria Mapping

- `!:strength` modifiers parsed correctly → Steps 2, 3, 9
- Default strength calculated for all rule types → Steps 4, 9
- Strength modifiers applied correctly → Steps 5, 9
- Rules sorted by strength during evaluation → Steps 7, 9
- Strength affects confidence scoring → Steps 8, 9
- Unit tests for strength parsing → Step 9
- Unit tests for default strength calculation → Step 9
- Unit tests for strength modifiers → Step 9
- Integration test comparing with GNU file output → Step 9
- Compatibility test shows improved match rate → Step 9

## Sequence Diagram

```
Parser Module → AST Structures → Strength Calculator → Evaluator
     ↓                ↓                  ↓                  ↓
parse_magic_rule_line() → MagicRule with strength_modifier
     ↓
parse_strength_modifier() → StrengthModifier enum
     ↓
calculate_default_strength() → base_strength (0-255)
     ↓
apply_strength_modifier() → final_strength (0-255)
     ↓
sort_rules_by_strength() → ordered rules
     ↓
evaluate_rules() → MatchResult with strength field
```

---

Execution Information

Branch: main
Commit: 50b370c