Implement comprehensive text-based magic file parser

[unclesp1d3r]

Overview

Implement a complete text-based magic file parser that reads entire files and converts them into a hierarchical tree of MagicRule structures. This is a critical component for Phase 1 MVP completion, as it bridges the gap between existing parser components (offsets, types, operators, values) and the evaluator engine.

Background

The project has completed core parsing components in src/parser/grammar.rs:

• ✅ parse_number - Parses decimal, hex, and octal numbers
• ✅ parse_offset - Parses offset specifications (absolute, indirect, relative)
• ✅ parse_operator - Parses comparison operators (=, !=, <, >, &)
• ✅ parse_value - Parses values (strings, numbers, byte sequences)

The AST structures in src/parser/ast.rs are also complete with full serialization support.

What's Missing: A higher-level parser that orchestrates these components to parse complete magic files line-by-line, handling:

• File-level structure and organization
• Line continuation and comments
• Hierarchical rule nesting based on indentation
• Error reporting with line numbers
• Special directives (!:mime, !:strength, etc.)

Magic File Format Reference

Magic files follow this structure:

# Comment lines start with #
offset  type  operator  value  message

# Example: ELF file detection
0       string    \x7fELF         ELF
>4      byte      1               32-bit
>4      byte      2               64-bit
>>16    leshort   >0              executable

# Continuation lines end with backslash\
0       string    PK\003\004     ZIP archive data, \
        at least v2.0 to extract

Key Features:

• Level 0 rules: Start with offset (0, 16, 0x20)
• Child rules: Prefixed with > characters (>, >>, >>>)
• Comments: Lines starting with #
• Empty lines: Should be ignored
• Continuation: Lines ending with \ continue on next line
• Special directives: !:mime, !:strength, !:ext

See docs/src/magic-format.md for complete format specification.

Technical Requirements

Core Function Signature

/// Parse a complete text-based magic file
///
/// # Arguments
/// * `input` - String content of the magic file
///
/// # Returns
/// * `Result<Vec<MagicRule>, ParseError>` - Top-level rules with nested children
///
/// # Errors
/// Returns ParseError with line number and description for:
/// - Invalid syntax
/// - Unrecognized types or operators
/// - Malformed offset specifications
/// - Orphaned child rules (> without parent)
pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError> {
    // Implementation needed
}

Implementation Components

1. Line Processing Pipeline

   • Strip comments (preserve content before #)
   • Skip empty lines
   • Handle continuation lines (join lines ending with \)
   • Track original line numbers for error reporting

2. Rule Level Detection

   • Count leading > characters to determine hierarchy level
   • Level 0: No > prefix
   • Level 1: > prefix
   • Level 2: >> prefix, etc.

3. Rule Parsing

   • Extract offset, type, operator, value, and message from each line
   • Use existing parse_offset, parse_value, etc. from grammar.rs
   • Handle optional operator (default to Operator::Equal)
   • Parse message text (may contain escape sequences)

4. Hierarchy Building

   • Maintain a stack of parent rules at each level
   • Attach child rules to the appropriate parent based on level
   • Validate that child rules have valid parents
   • Error if level increases by more than 1

5. Special Directive Handling (optional for v1)

   • !:mime - MIME type metadata
   • !:strength - Match strength/priority
   • !:ext - File extension hints
   • Store as metadata on the last parsed rule

6. Error Handling

   • Include line number in all error messages
   • Provide descriptive error messages (e.g., "Invalid offset specification at line 42")
   • Continue parsing after non-fatal errors (optional: collect all errors)

Proposed Solution

Phase 1: Basic Line Processing

// In src/parser/mod.rs

struct LineInfo {
    content: String,
    line_number: usize,
    level: u32,
}

fn preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError> {
    // 1. Handle continuation lines
    // 2. Strip comments
    // 3. Detect hierarchy level (count >)
    // 4. Track line numbers
}

Phase 2: Rule Parsing

fn parse_magic_rule_line(line: &LineInfo) -> Result<MagicRule, ParseError> {
    // Use nom combinators with existing grammar.rs functions
    // Pattern: offset  type  [operator]  value  message
}

Phase 3: Hierarchy Construction

fn build_rule_hierarchy(lines: Vec<LineInfo>) -> Result<Vec<MagicRule>, ParseError> {
    // Stack-based approach to build parent-child relationships
    // Validate level transitions
}

Phase 4: Integration

pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError> {
    let lines = preprocess_lines(input)?;
    let rules = lines.into_iter()
        .map(|line| parse_magic_rule_line(&line))
        .collect::<Result<Vec<_>, _>>()?;
    build_rule_hierarchy(rules)
}

Testing Requirements

Unit Tests (Required)

#[cfg(test)]
mod tests {
    #[test]
    fn test_parse_simple_rule() {
        let input = "0    string    PK\\x03\\x04    ZIP archive";
        let rules = parse_text_magic_file(input).unwrap();
        assert_eq!(rules.len(), 1);
        assert_eq!(rules[0].message, "ZIP archive");
    }

    #[test]
    fn test_parse_hierarchical_rules() {
        let input = r#"
0       string    \x7fELF         ELF
>4      byte      1               32-bit
>4      byte      2               64-bit
        "#;
        let rules = parse_text_magic_file(input).unwrap();
        assert_eq!(rules.len(), 1);
        assert_eq!(rules[0].children.len(), 2);
    }

    #[test]
    fn test_parse_comments_and_empty_lines() {
        let input = r#"
# This is a comment

0       string    test    Test file
        "#;
        let rules = parse_text_magic_file(input).unwrap();
        assert_eq!(rules.len(), 1);
    }

    #[test]
    fn test_parse_continuation_lines() {
        let input = "0    string    test    Long message \\\n        continued here";
        let rules = parse_text_magic_file(input).unwrap();
        assert!(rules[0].message.contains("continued"));
    }

    #[test]
    fn test_error_orphaned_child() {
        let input = ">4    byte    1    orphaned";
        assert!(parse_text_magic_file(input).is_err());
    }

    #[test]
    fn test_error_invalid_level_jump() {
        let input = r#"
0       string    test    Parent
>>>4    byte      1       Invalid jump
        "#;
        assert!(parse_text_magic_file(input).is_err());
    }
}

Integration Tests (Recommended)

• Parse actual magic files from third_party/tests/*.magic
• Validate against known-good outputs
• Performance testing with large magic databases

Acceptance Criteria

• parse_text_magic_file function implemented in src/parser/mod.rs
• Line preprocessing handles comments, empty lines, continuation lines
• Hierarchy detection based on > prefix works correctly
• Rule parsing integrates existing grammar.rs functions
• Parent-child relationships built correctly
• Error messages include line numbers
• At least 10 unit tests covering various scenarios
• All existing tests continue to pass
• Documentation updated with examples
• Code passes cargo clippy -- -D warnings

Dependencies

• Existing parser components in src/parser/grammar.rs
• AST structures in src/parser/ast.rs
• Error types in src/error.rs

Related Work

• Phase 1 MVP completion depends on this parser
• Unblocks evaluator implementation (next major milestone)
• Enables integration testing with real magic files

References

• Magic file format specification: docs/src/magic-format.md
• Example magic files: third_party/tests/*.magic
• Original libmagic: https://www.darwinsys.com/file/
• File project test suite: https://github.com/file/file/tree/master/tests

[traycerai]

Plan

Observations

src/parser/grammar.rs implements parse_magic_rule with comprehensive tests for offsets, types, operators, values, levels (via > prefixes), whitespace, strings, hex, real-world ELF/ZIP examples, edge/invalid cases. src/parser/ast.rs defines MagicRule with level, children. Utilities like multispace0 exist. third_party/tests/*.magic provides real files. No file-level parser yet; src/error.rs supports line-numbered ParseError. docs/src/magic-format.md details format.

Approach

Orchestrate parse_text_magic_file in src/parser/mod.rs: split lines, skip empty/comments, join continuations (ending \), parse via parse_magic_rule, build hierarchy using stack (push/pop by level diffs, validate no skips >1). Defer special directives (!:mime). Prioritizes incremental build on existing parser, enables MagicDatabase loading. Tests use fixtures + real files for full coverage.

Implementation Steps

1. Define Supporting Structures

File: src/parser/mod.rs

Add:

#[derive(Debug)]
struct LineInfo {
    content: String,
    original_line: usize,
}

2. Implement Line Preprocessing

Function: preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError>

• Split input into lines (1-indexed).
• Buffer for continuations: if line.trim_end().ends_with('\'), append next (strip \n, \).
• Skip empty (trim().is_empty()) or comment (starts_with('#')).
• Strip inline comments (split at first #, keep before).
• Track original_line as start of continuation block.
• Return non-skipped LineInfos.

3. Parse Individual Lines

Function: parse_rule_line(info: &LineInfo) -> Result<MagicRule, ParseError>

• Call parse_magic_rule(info.content.as_str()).
• Map nom::Err to ParseError::invalid_syntax(info.original_line, msg).

4. Build Hierarchy

Function: build_hierarchy(rules: Vec<MagicRule>) -> Result<Vec<MagicRule>, ParseError>

Stack-based:

• VecDeque<MagicRule> stack.
• For each rule:
  • While stack top level >= current level, pop.
  • If level > stack.top().level + 1: error "level jump".
  • If stack not empty, add to stack.top().children.
  • Push current.
• Return stack roots (level 0).

5. Core Parser Function

Public: pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError>

let lines = preprocess_lines(input)?;
let flat_rules: Vec<_> = lines.iter()
    .map(|li| parse_rule_line(li))
    .collect::<Result<_,_>>()?;
build_hierarchy(flat_rules)

Add rustdoc with examples from ticket.

6. Error Enhancements

Extend ParseError usage:

• invalid_syntax(line, msg) for parse fails.
• invalid_hierarchy(line, msg) for level issues.
• Include snippet: content[0..min(50, len)].

flowchart TD
    A[input] --> B[preprocess_lines]
    B --> C[Vec<LineInfo>]
    C --> D[for each: parse_rule_line]
    D --> E[Vec<MagicRule> flat]
    E --> F[build_hierarchy stack]
    F --> G[Vec<MagicRule> tree]
    B -.-> H[continuation<br/>comment/empty skip]
    D -.-> I[parse_magic_rule]
    F -.-> J[level validate<br/>push/pop]

Loading

7. Unit Tests in src/parser/mod.rs

Match ticket:

• test_parse_simple_rule
• test_parse_hierarchical_rules
• test_parse_comments_and_empty_lines
• test_parse_continuation_lines
• test_error_orphaned_child
• test_error_invalid_level_jump
• Add: multi-continuation, inline comments, level 3+, empty file.

Use #[test] with unwrap() assertions.

8. Integration Tests

New file: tests/parser_integration.rs

• Load third_party/tests/multiple-*.magic, assert rule counts.
• Fixtures: create tests/fixtures/ with simple.magic, hierarchical.magic, invalid.magic.
• Snapshot with insta::assert_snapshot!.
• Test real ELF/ZIP excerpts.

Test Type	Count	Coverage
Unit	10+	Line proc, parse, hierarchy
Integration	5+	Real files, fixtures
Error	5+	Syntax, levels, orphans

9. Module Exports & Docs

src/parser/mod.rs:

• pub use grammar::parse_magic_rule;
• Export new functions.
• Update lib.rs docs.

10. Clippy & Validation

• Run cargo clippy.
• Ensure no new warnings.
• Performance: handle 10k+ line files.

Dependencies

graph LR
    parse_text_magic_file --> preprocess_lines
    preprocess_lines --> grammar.rs[is_comment? empty? cont?]
    parse_rule_line --> grammar::parse_magic_rule
    build_hierarchy --> ast::MagicRule{level children}
    parse_text_magic_file --> error::ParseError{line}
    MagicDatabase -.-> parse_text_magic_file

Loading

Next: Integrate into MagicDatabase::from_file(path).

Import In IDE

🤖 Prompt for AI Agents

## Observations

`src/parser/grammar.rs` implements `parse_magic_rule` with comprehensive tests for offsets, types, operators, values, levels (via `>` prefixes), whitespace, strings, hex, real-world ELF/ZIP examples, edge/invalid cases. `src/parser/ast.rs` defines `MagicRule` with `level`, `children`. Utilities like `multispace0` exist. `third_party/tests/*.magic` provides real files. No file-level parser yet; `src/error.rs` supports line-numbered `ParseError`. `docs/src/magic-format.md` details format.

## Approach

Orchestrate `parse_text_magic_file` in `src/parser/mod.rs`: split lines, skip empty/comments, join continuations (ending `\`), parse via `parse_magic_rule`, build hierarchy using stack (push/pop by level diffs, validate no skips >1). Defer special directives (`!:mime`). Prioritizes incremental build on existing parser, enables `MagicDatabase` loading. Tests use fixtures + real files for full coverage.

## Implementation Steps

### 1. Define Supporting Structures

**File:** `src/parser/mod.rs`

Add:
```rust
#[derive(Debug)]
struct LineInfo {
    content: String,
    original_line: usize,
}
```

### 2. Implement Line Preprocessing

**Function:** `preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError>`

- Split `input` into lines (1-indexed).
- Buffer for continuations: if line.trim_end().ends_with('\\'), append next (strip `\n`, `\`).
- Skip empty (`trim().is_empty()`) or comment (`starts_with('#')`).
- Strip inline comments (split at first `#`, keep before).
- Track `original_line` as start of continuation block.
- Return non-skipped `LineInfo`s.

### 3. Parse Individual Lines

**Function:** `parse_rule_line(info: &LineInfo) -> Result<MagicRule, ParseError>`

- Call `parse_magic_rule(info.content.as_str())`.
- Map `nom::Err` to `ParseError::invalid_syntax(info.original_line, msg)`.

### 4. Build Hierarchy

**Function:** `build_hierarchy(rules: Vec<MagicRule>) -> Result<Vec<MagicRule>, ParseError>`

Stack-based:
- `VecDeque<MagicRule>` stack.
- For each rule:
  - While stack top level >= current level, pop.
  - If level > stack.top().level + 1: error "level jump".
  - If stack not empty, add to stack.top().children.
  - Push current.
- Return stack roots (level 0).

### 5. Core Parser Function

**Public:** `pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError>`

```rust
let lines = preprocess_lines(input)?;
let flat_rules: Vec<_> = lines.iter()
    .map(|li| parse_rule_line(li))
    .collect::<Result<_,_>>()?;
build_hierarchy(flat_rules)
```

Add rustdoc with examples from ticket.

### 6. Error Enhancements

Extend `ParseError` usage:
- `invalid_syntax(line, msg)` for parse fails.
- `invalid_hierarchy(line, msg)` for level issues.
- Include snippet: `content[0..min(50, len)]`.

```mermaid
flowchart TD
    A[input] --> B[preprocess_lines]
    B --> C[Vec<LineInfo>]
    C --> D[for each: parse_rule_line]
    D --> E[Vec<MagicRule> flat]
    E --> F[build_hierarchy stack]
    F --> G[Vec<MagicRule> tree]
    B -.-> H[continuation<br/>comment/empty skip]
    D -.-> I[parse_magic_rule]
    F -.-> J[level validate<br/>push/pop]
```

### 7. Unit Tests in `src/parser/mod.rs`

Match ticket:
- `test_parse_simple_rule`
- `test_parse_hierarchical_rules`
- `test_parse_comments_and_empty_lines`
- `test_parse_continuation_lines`
- `test_error_orphaned_child`
- `test_error_invalid_level_jump`
- Add: multi-continuation, inline comments, level 3+, empty file.

Use `#[test]` with `unwrap()` assertions.

### 8. Integration Tests

**New file:** `tests/parser_integration.rs`

- Load `third_party/tests/multiple-*.magic`, assert rule counts.
- Fixtures: create `tests/fixtures/` with simple.magic, hierarchical.magic, invalid.magic.
- Snapshot with `insta::assert_snapshot!`.
- Test real ELF/ZIP excerpts.

| Test Type | Count | Coverage |
|-----------|-------|----------|
| Unit | 10+ | Line proc, parse, hierarchy |
| Integration | 5+ | Real files, fixtures |
| Error | 5+ | Syntax, levels, orphans |

### 9. Module Exports & Docs

**`src/parser/mod.rs`:**
- `pub use grammar::parse_magic_rule;`
- Export new functions.
- Update `lib.rs` docs.

### 10. Clippy & Validation

- Run `cargo clippy`.
- Ensure no new warnings.
- Performance: handle 10k+ line files.

## Dependencies

```mermaid
graph LR
    parse_text_magic_file --> preprocess_lines
    preprocess_lines --> grammar.rs[is_comment? empty? cont?]
    parse_rule_line --> grammar::parse_magic_rule
    build_hierarchy --> ast::MagicRule{level children}
    parse_text_magic_file --> error::ParseError{line}
    MagicDatabase -.-> parse_text_magic_file
```

**Next:** Integrate into `MagicDatabase::from_file(path)`.

---

Execution Information

Branch: main
Commit: 4410a5c

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Get details of issue #11
gh issue view 11 --json title,body,labels,assignees,milestone,state,author,createdAt,updatedAt,url

Length of output: 1208

---

🏁 Script executed:

#!/bin/bash
# Read tasks.md to understand the full context of Task 14
cat tasks.md

echo "---SEPARATOR---"

# Get list of available labels
gh label list --json name,description,color

echo "---SEPARATOR---"

# Get list of milestones
gh milestone list --json title,description,state

echo "---SEPARATOR---"

# Check the parser/mod.rs file to understand current state
cat parser/mod.rs 2>/dev/null || echo "parser/mod.rs not found"

Length of output: 2227

---

🏁 Script executed:

#!/bin/bash
# Get project structure
echo "=== Repository structure ==="
fd -t f -d 3 | head -30

echo ""
echo "=== README content ==="
cat README.md 2>/dev/null || echo "README.md not found"

echo ""
echo "=== Check for parser directory ==="
ls -la parser/ 2>/dev/null || echo "parser directory not found"
ls -la src/parser/ 2>/dev/null || echo "src/parser directory not found"

echo ""
echo "=== Search for magic-related files ==="
fd -e magic -e rs | grep -i magic | head -20

echo ""
echo "=== Search for existing parser or magic references ==="
rg -l "magic file|parse.*magic|MagicRule" --type rust | head -10

Length of output: 15161

---

🏁 Script executed:

#!/bin/bash
# Read current parser/mod.rs
echo "=== Current src/parser/mod.rs ==="
cat src/parser/mod.rs

echo ""
echo "=== Magic format documentation ==="
cat docs/src/magic-format.md 2>/dev/null || echo "magic-format.md not found"

echo ""
echo "=== Example magic file ==="
cat third_party/tests/regex-eol.magic 2>/dev/null | head -30

echo ""
echo "=== Another example ==="
cat third_party/tests/multiple-A.magic 2>/dev/null | head -30

echo ""
echo "=== Check for standard magic files ==="
fd magic$ | head -10

echo ""
echo "=== Look at MagicRule structure in ast.rs ==="
rg -A 15 "pub struct MagicRule" src/parser/ast.rs

Length of output: 4346

---

🏁 Script executed:

#!/bin/bash
# Update the issue with enhanced title, body, and metadata

gh issue edit 11 \
  --title "Implement comprehensive text-based magic file parser" \
  --body "## Overview

Implement a complete text-based magic file parser that reads entire `.magic` files and converts them into a hierarchical tree of \`MagicRule\` structures. This is a critical component for Phase 1 MVP completion, as it bridges the gap between existing parser components (offsets, types, operators, values) and the evaluator engine.

## Background

The project has completed core parsing components in \`src/parser/grammar.rs\`:
- ✅ \`parse_number\` - Parses decimal, hex, and octal numbers
- ✅ \`parse_offset\` - Parses offset specifications (absolute, indirect, relative)
- ✅ \`parse_operator\` - Parses comparison operators (=, !=, <, >, &)
- ✅ \`parse_value\` - Parses values (strings, numbers, byte sequences)

The AST structures in \`src/parser/ast.rs\` are also complete with full serialization support.

**What's Missing**: A higher-level parser that orchestrates these components to parse complete magic files line-by-line, handling:
- File-level structure and organization
- Line continuation and comments
- Hierarchical rule nesting based on indentation
- Error reporting with line numbers
- Special directives (\`!:mime\`, \`!:strength\`, etc.)

## Magic File Format Reference

Magic files follow this structure:

\`\`\`text
# Comment lines start with #
offset  type  operator  value  message

# Example: ELF file detection
0       string    \\x7fELF         ELF
>4      byte      1               32-bit
>4      byte      2               64-bit
>>16    leshort   >0              executable

# Continuation lines end with backslash\\
0       string    PK\\003\\004     ZIP archive data, \\
        at least v2.0 to extract
\`\`\`

**Key Features**:
- **Level 0 rules**: Start with offset (0, 16, 0x20)
- **Child rules**: Prefixed with \`>\` characters (>, >>, >>>)
- **Comments**: Lines starting with \`#\`
- **Empty lines**: Should be ignored
- **Continuation**: Lines ending with \`\\\` continue on next line
- **Special directives**: \`!:mime\`, \`!:strength\`, \`!:ext\`

See \`docs/src/magic-format.md\` for complete format specification.

## Technical Requirements

### Core Function Signature

\`\`\`rust
/// Parse a complete text-based magic file
///
/// # Arguments
/// * \`input\` - String content of the magic file
///
/// # Returns
/// * \`Result<Vec<MagicRule>, ParseError>\` - Top-level rules with nested children
///
/// # Errors
/// Returns ParseError with line number and description for:
/// - Invalid syntax
/// - Unrecognized types or operators
/// - Malformed offset specifications
/// - Orphaned child rules (> without parent)
pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError> {
    // Implementation needed
}
\`\`\`

### Implementation Components

1. **Line Processing Pipeline**
   - Strip comments (preserve content before \`#\`)
   - Skip empty lines
   - Handle continuation lines (join lines ending with \`\\\`)
   - Track original line numbers for error reporting

2. **Rule Level Detection**
   - Count leading \`>\` characters to determine hierarchy level
   - Level 0: No \`>\` prefix
   - Level 1: \`>\` prefix
   - Level 2: \`>>\` prefix, etc.

3. **Rule Parsing**
   - Extract offset, type, operator, value, and message from each line
   - Use existing \`parse_offset\`, \`parse_value\`, etc. from \`grammar.rs\`
   - Handle optional operator (default to \`Operator::Equal\`)
   - Parse message text (may contain escape sequences)

4. **Hierarchy Building**
   - Maintain a stack of parent rules at each level
   - Attach child rules to the appropriate parent based on level
   - Validate that child rules have valid parents
   - Error if level increases by more than 1

5. **Special Directive Handling** (optional for v1)
   - \`!:mime\` - MIME type metadata
   - \`!:strength\` - Match strength/priority
   - \`!:ext\` - File extension hints
   - Store as metadata on the last parsed rule

6. **Error Handling**
   - Include line number in all error messages
   - Provide descriptive error messages (e.g., \"Invalid offset specification at line 42\")
   - Continue parsing after non-fatal errors (optional: collect all errors)

## Proposed Solution

### Phase 1: Basic Line Processing
\`\`\`rust
// In src/parser/mod.rs

struct LineInfo {
    content: String,
    line_number: usize,
    level: u32,
}

fn preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError> {
    // 1. Handle continuation lines
    // 2. Strip comments
    // 3. Detect hierarchy level (count >)
    // 4. Track line numbers
}
\`\`\`

### Phase 2: Rule Parsing
\`\`\`rust
fn parse_magic_rule_line(line: &LineInfo) -> Result<MagicRule, ParseError> {
    // Use nom combinators with existing grammar.rs functions
    // Pattern: offset  type  [operator]  value  message
}
\`\`\`

### Phase 3: Hierarchy Construction
\`\`\`rust
fn build_rule_hierarchy(lines: Vec<LineInfo>) -> Result<Vec<MagicRule>, ParseError> {
    // Stack-based approach to build parent-child relationships
    // Validate level transitions
}
\`\`\`

### Phase 4: Integration
\`\`\`rust
pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError> {
    let lines = preprocess_lines(input)?;
    let rules = lines.into_iter()
        .map(|line| parse_magic_rule_line(&line))
        .collect::<Result<Vec<_>, _>>()?;
    build_rule_hierarchy(rules)
}
\`\`\`

## Testing Requirements

### Unit Tests (Required)

\`\`\`rust
#[cfg(test)]
mod tests {
    #[test]
    fn test_parse_simple_rule() {
        let input = \"0    string    PK\\\\x03\\\\x04    ZIP archive\";
        let rules = parse_text_magic_file(input).unwrap();
        assert_eq!(rules.len(), 1);
        assert_eq!(rules[0].message, \"ZIP archive\");
    }

    #[test]
    fn test_parse_hierarchical_rules() {
        let input = r#\"
0       string    \\x7fELF         ELF
>4      byte      1               32-bit
>4      byte      2               64-bit
        \"#;
        let rules = parse_text_magic_file(input).unwrap();
        assert_eq!(rules.len(), 1);
        assert_eq!(rules[0].children.len(), 2);
    }

    #[test]
    fn test_parse_comments_and_empty_lines() {
        let input = r#\"
# This is a comment

0       string    test    Test file
        \"#;
        let rules = parse_text_magic_file(input).unwrap();
        assert_eq!(rules.len(), 1);
    }

    #[test]
    fn test_parse_continuation_lines() {
        let input = \"0    string    test    Long message \\\\\\n        continued here\";
        let rules = parse_text_magic_file(input).unwrap();
        assert!(rules[0].message.contains(\"continued\"));
    }

    #[test]
    fn test_error_orphaned_child() {
        let input = \">4    byte    1    orphaned\";
        assert!(parse_text_magic_file(input).is_err());
    }

    #[test]
    fn test_error_invalid_level_jump() {
        let input = r#\"
0       string    test    Parent
>>>4    byte      1       Invalid jump
        \"#;
        assert!(parse_text_magic_file(input).is_err());
    }
}
\`\`\`

### Integration Tests (Recommended)

- Parse actual magic files from \`third_party/tests/*.magic\`
- Validate against known-good outputs
- Performance testing with large magic databases

## Acceptance Criteria

- [ ] \`parse_text_magic_file\` function implemented in \`src/parser/mod.rs\`
- [ ] Line preprocessing handles comments, empty lines, continuation lines
- [ ] Hierarchy detection based on \`>\` prefix works correctly
- [ ] Rule parsing integrates existing grammar.rs functions
- [ ] Parent-child relationships built correctly
- [ ] Error messages include line numbers
- [ ] At least 10 unit tests covering various scenarios
- [ ] All existing tests continue to pass
- [ ] Documentation updated with examples
- [ ] Code passes \`cargo clippy -- -D warnings\`

## Dependencies

- Existing parser components in \`src/parser/grammar.rs\`
- AST structures in \`src/parser/ast.rs\`
- Error types in \`src/error.rs\`

## Related Work

- Phase 1 MVP completion depends on this parser
- Unblocks evaluator implementation (next major milestone)
- Enables integration testing with real magic files

## References

- Magic file format specification: \`docs/src/magic-format.md\`
- Example magic files: \`third_party/tests/*.magic\`
- Original libmagic: https://www.darwinsys.com/file/
- File project test suite: https://github.com/file/file/tree/master/tests" \
  --add-label "good first issue" \
  --add-label "help wanted" \
  --add-label "testing"

# Also add assignee
gh issue edit 11 --add-assignee unclesp1d3r

echo ""
echo "Issue updated successfully!"

Length of output: 181

---

@unclesp1d3r I've successfully enhanced issue #11 with comprehensive details! Here's what was updated:

🎯 Enhanced Title

Before: "Implement text magic file parsing (subtask 14.2)"
After: "Implement comprehensive text-based magic file parser"

📝 Enhanced Body

The issue now includes:

Overview & Background

• Clear explanation of what needs to be built and why it's critical for Phase 1 MVP
• Context about existing completed components (parse_number, parse_offset, parse_value, etc.)
• Specific gap identification (file-level orchestration missing)

Magic File Format Reference

• Concrete examples of magic file syntax
• Key features explanation (hierarchy, comments, continuation lines)
• Reference to full documentation at docs/src/magic-format.md

Technical Requirements

• Complete function signature for parse_text_magic_file
• 6 implementation components broken down:
  1. Line Processing Pipeline
  2. Rule Level Detection
  3. Rule Parsing
  4. Hierarchy Building
  5. Special Directive Handling
  6. Error Handling

Proposed Solution

• 4-phase implementation approach with code examples:
  • Phase 1: Basic Line Processing
  • Phase 2: Rule Parsing
  • Phase 3: Hierarchy Construction
  • Phase 4: Integration

Testing Requirements

• 6 specific unit tests with code examples
• Integration test recommendations
• Coverage expectations

Acceptance Criteria

• 10 clear checkboxes for completion validation
• Includes quality requirements (clippy, existing tests pass)

References & Dependencies

• Links to relevant documentation and example files
• Clear dependency mapping to existing code

🏷️ Labels Added

• ✅ good first issue - Well-defined scope with clear guidance
• ✅ help wanted - Open for community contribution
• ✅ testing - Emphasizes test coverage importance
• (Kept existing: enhancement, parser)

👤 Assignment

• ✅ Assigned to you (@unclesp1d3r)

The issue is now ready for implementation with clear expectations, concrete examples, and a structured approach that leverages all the existing parser components you've already built! 🚀

View the enhanced issue: #11

(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

Previous Plan

Observations

The codebase has a well-structured parser module with completed Task 14.1 (parse_magic_rule in file:src/parser/grammar.rs). The AST types are comprehensive (file:src/parser/ast.rs), and utility functions for line classification (is_empty_line, is_comment_line, has_continuation) already exist. The error handling system (file:src/error.rs) provides ParseError with line number tracking. The MagicDatabase in file:src/lib.rs currently returns empty rules and needs integration with the text parser.

Approach

Implement parse_text_magic_file function in file:src/parser/mod.rs that orchestrates line-by-line parsing using existing utilities from file:src/parser/grammar.rs. The function will handle continuation lines by concatenating them, skip comments and empty lines, parse each rule line using parse_magic_rule, and build hierarchical rule structures by tracking indentation levels. Unit tests will use existing sample files from file:third_party/tests/ and create new test fixtures to validate all parsing scenarios including error cases.

Implementation Steps

1. Add parse_text_magic_file Function to Parser Module

File: file:src/parser/mod.rs

Add a public function parse_text_magic_file(content: &str) -> Result<Vec<MagicRule>, ParseError> that:

• Takes the entire magic file content as a string
• Returns a flat list of MagicRule structs with proper level and children populated
• Uses Result type with ParseError for error handling

2. Implement Line-by-Line Processing Logic

Within parse_text_magic_file function:

Create a line processing loop that:

• Iterates through lines with enumerate() to track line numbers (1-indexed)
• Uses is_empty_line() from file:src/parser/grammar.rs to skip empty lines
• Uses is_comment_line() from file:src/parser/grammar.rs to skip comment lines
• Maintains current line number for error reporting

3. Handle Continuation Lines

Within the line processing loop:

Implement continuation line handling:

• Use has_continuation() from file:src/parser/grammar.rs to detect lines ending with \
• Accumulate continued lines into a buffer, removing the trailing \ character
• Track the starting line number of the continued sequence for error reporting
• Process the complete concatenated line once continuation ends

4. Parse Individual Rule Lines

For each complete line (after continuation handling):

Parse the rule using existing infrastructure:

• Call parse_magic_rule() from file:src/parser/grammar.rs to parse the line
• Wrap nom parsing errors in ParseError::invalid_syntax() with the current line number
• Extract the MagicRule with its level field (determined by > prefix count)
• Collect parsed rules in a flat vector

5. Build Hierarchical Rule Structure

After collecting all flat rules:

Implement hierarchy construction algorithm:

• Use a stack-based approach to track parent rules at each level
• Iterate through flat rules in order
• For each rule, compare its level with the previous rule's level
• If level increases by 1, add as child to the previous rule
• If level stays same or decreases, pop stack to find correct parent
• Validate that level increases are always by 1 (no skipping levels)
• Return only top-level rules (level 0) with children properly nested

flowchart TD
    A[Start: parse_text_magic_file] --> B[Split content into lines]
    B --> C[Initialize line iterator with line numbers]
    C --> D{More lines?}
    D -->|No| M[Build hierarchy from flat rules]
    D -->|Yes| E[Get next line]
    E --> F{Empty or comment?}
    F -->|Yes| D
    F -->|No| G{Has continuation?}
    G -->|Yes| H[Accumulate line, remove backslash]
    H --> D
    G -->|No| I[Process complete line]
    I --> J[Call parse_magic_rule]
    J --> K{Parse success?}
    K -->|No| L[Return ParseError with line number]
    K -->|Yes| N[Add rule to flat list]
    N --> D
    M --> O[Stack-based hierarchy builder]
    O --> P[Return top-level rules with children]
    L --> Q[End: Error]
    P --> R[End: Success]

Loading

6. Add Comprehensive Error Handling

Throughout the implementation:

Ensure proper error reporting:

• Wrap all nom parsing errors with ParseError::invalid_syntax(line_number, message)
• Validate hierarchy level transitions (no level skips)
• Use ParseError::invalid_offset(), ParseError::invalid_type(), etc. for specific errors
• Provide clear error messages that include the problematic line content
• Handle edge cases like files with only comments or empty files

7. Export the Function

In file:src/parser/mod.rs:

Add to the public API:

• Export parse_text_magic_file function
• Update module documentation to describe the text file parsing capability
• Ensure the function is accessible from file:src/lib.rs for MagicDatabase integration

8. Create Unit Tests for Basic Parsing

In file:src/parser/mod.rs (test module):

Add tests for basic functionality:

• Test parsing a simple single-rule file
• Test parsing multiple top-level rules
• Test parsing rules with hierarchical children (using > prefix)
• Test parsing deeply nested rules (level 2, 3, etc.)
• Test empty file handling (should return empty vector)
• Test file with only comments (should return empty vector)

9. Create Unit Tests for Line Handling

In test module:

Add tests for special line cases:

• Test continuation lines (lines ending with \)
• Test multi-line continuations (3+ lines)
• Test mixed content (rules, comments, empty lines, continuations)
• Test comments with various formats (#, # comment, # indented)
• Test whitespace handling (leading/trailing spaces, tabs)

10. Create Unit Tests for Error Cases

In test module:

Add tests for error handling:

• Test invalid syntax with proper line number reporting
• Test invalid offset specifications
• Test invalid type specifications
• Test invalid operator specifications
• Test invalid value specifications
• Test hierarchy level skips (e.g., level 0 followed by level 2)
• Test malformed continuation lines

11. Create Sample Magic File Test Fixtures

Create test files in file:tests/fixtures/ directory:

Create sample .magic files:

• simple.magic - Basic rules without hierarchy
• hierarchical.magic - Rules with parent-child relationships
• complex.magic - Mix of all features (comments, continuations, hierarchy)
• elf.magic - Real-world ELF file detection rules
• zip.magic - Real-world ZIP archive detection rules
• invalid_syntax.magic - File with intentional syntax errors
• invalid_hierarchy.magic - File with level skips

12. Create Integration Tests with Sample Files

In file:tests/parser_integration_tests.rs (new file):

Add integration tests:

• Test parsing each sample magic file from fixtures
• Verify correct number of rules parsed
• Verify hierarchy structure is correct
• Verify rule properties (offset, type, operator, value, message)
• Test parsing existing files from file:third_party/tests/
• Use snapshot testing with insta crate for regression prevention

13. Test with Real Magic Files

In integration tests:

Test with actual magic files:

• Parse file:third_party/tests/multiple-A.magic
• Parse file:third_party/tests/multiple-B.magic
• Parse file:third_party/tests/regex-eol.magic
• Parse file:third_party/tests/searchbug.magic
• Verify parsing succeeds and produces expected rule count
• Handle unsupported features gracefully (e.g., search, regex, use, name)

14. Add Documentation and Examples

In file:src/parser/mod.rs:

Add comprehensive documentation:

• Document the parse_text_magic_file function with rustdoc comments
• Include usage examples showing how to parse a magic file
• Document the expected format and supported features
• Document error handling and line number reporting
• Add examples of valid magic file syntax
• Reference the magic file format documentation in file:docs/src/magic-format.md

Testing Strategy

Test Category	Test Files	Coverage
Unit Tests	file:src/parser/mod.rs	Line processing, continuation, hierarchy building
Integration Tests	file:tests/parser_integration_tests.rs	End-to-end parsing with sample files
Fixture Files	file:tests/fixtures/*.magic	Real-world magic file formats
Error Cases	Test module in file:src/parser/mod.rs	Invalid syntax, hierarchy errors
Existing Files	file:third_party/tests/*.magic	Compatibility with canonical test suite

Dependencies and Integration Points

graph LR
    A[parse_text_magic_file] --> B[parse_magic_rule]
    A --> C[is_empty_line]
    A --> D[is_comment_line]
    A --> E[has_continuation]
    B --> F[parse_rule_offset]
    B --> G[parse_type_and_operator]
    B --> H[parse_value]
    B --> I[parse_message]
    A --> J[ParseError]
    A --> K[MagicRule AST]
    L[MagicDatabase] -.Future.-> A

Loading

Key Integration Points:

• parse_magic_rule from file:src/parser/grammar.rs - Core rule parsing
• Utility functions from file:src/parser/grammar.rs - Line classification
• ParseError from file:src/error.rs - Error reporting with line numbers
• MagicRule from file:src/parser/ast.rs - AST representation
• Future integration with MagicDatabase::load_from_file in file:src/lib.rs

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured parser module with completed Task 14.1 (`parse_magic_rule` in `file:src/parser/grammar.rs`). The AST types are comprehensive (`file:src/parser/ast.rs`), and utility functions for line classification (`is_empty_line`, `is_comment_line`, `has_continuation`) already exist. The error handling system (`file:src/error.rs`) provides `ParseError` with line number tracking. The `MagicDatabase` in `file:src/lib.rs` currently returns empty rules and needs integration with the text parser.

## Approach

Implement `parse_text_magic_file` function in `file:src/parser/mod.rs` that orchestrates line-by-line parsing using existing utilities from `file:src/parser/grammar.rs`. The function will handle continuation lines by concatenating them, skip comments and empty lines, parse each rule line using `parse_magic_rule`, and build hierarchical rule structures by tracking indentation levels. Unit tests will use existing sample files from `file:third_party/tests/` and create new test fixtures to validate all parsing scenarios including error cases.

## Implementation Steps

### 1. Add `parse_text_magic_file` Function to Parser Module

**File:** `file:src/parser/mod.rs`

Add a public function `parse_text_magic_file(content: &str) -> Result<Vec<MagicRule>, ParseError>` that:
- Takes the entire magic file content as a string
- Returns a flat list of `MagicRule` structs with proper `level` and `children` populated
- Uses `Result` type with `ParseError` for error handling

### 2. Implement Line-by-Line Processing Logic

**Within `parse_text_magic_file` function:**

Create a line processing loop that:
- Iterates through lines with `enumerate()` to track line numbers (1-indexed)
- Uses `is_empty_line()` from `file:src/parser/grammar.rs` to skip empty lines
- Uses `is_comment_line()` from `file:src/parser/grammar.rs` to skip comment lines
- Maintains current line number for error reporting

### 3. Handle Continuation Lines

**Within the line processing loop:**

Implement continuation line handling:
- Use `has_continuation()` from `file:src/parser/grammar.rs` to detect lines ending with `\`
- Accumulate continued lines into a buffer, removing the trailing `\` character
- Track the starting line number of the continued sequence for error reporting
- Process the complete concatenated line once continuation ends

### 4. Parse Individual Rule Lines

**For each complete line (after continuation handling):**

Parse the rule using existing infrastructure:
- Call `parse_magic_rule()` from `file:src/parser/grammar.rs` to parse the line
- Wrap nom parsing errors in `ParseError::invalid_syntax()` with the current line number
- Extract the `MagicRule` with its `level` field (determined by `>` prefix count)
- Collect parsed rules in a flat vector

### 5. Build Hierarchical Rule Structure

**After collecting all flat rules:**

Implement hierarchy construction algorithm:
- Use a stack-based approach to track parent rules at each level
- Iterate through flat rules in order
- For each rule, compare its `level` with the previous rule's level
- If level increases by 1, add as child to the previous rule
- If level stays same or decreases, pop stack to find correct parent
- Validate that level increases are always by 1 (no skipping levels)
- Return only top-level rules (level 0) with children properly nested

```mermaid
flowchart TD
    A[Start: parse_text_magic_file] --> B[Split content into lines]
    B --> C[Initialize line iterator with line numbers]
    C --> D{More lines?}
    D -->|No| M[Build hierarchy from flat rules]
    D -->|Yes| E[Get next line]
    E --> F{Empty or comment?}
    F -->|Yes| D
    F -->|No| G{Has continuation?}
    G -->|Yes| H[Accumulate line, remove backslash]
    H --> D
    G -->|No| I[Process complete line]
    I --> J[Call parse_magic_rule]
    J --> K{Parse success?}
    K -->|No| L[Return ParseError with line number]
    K -->|Yes| N[Add rule to flat list]
    N --> D
    M --> O[Stack-based hierarchy builder]
    O --> P[Return top-level rules with children]
    L --> Q[End: Error]
    P --> R[End: Success]
```

### 6. Add Comprehensive Error Handling

**Throughout the implementation:**

Ensure proper error reporting:
- Wrap all nom parsing errors with `ParseError::invalid_syntax(line_number, message)`
- Validate hierarchy level transitions (no level skips)
- Use `ParseError::invalid_offset()`, `ParseError::invalid_type()`, etc. for specific errors
- Provide clear error messages that include the problematic line content
- Handle edge cases like files with only comments or empty files

### 7. Export the Function

**In `file:src/parser/mod.rs`:**

Add to the public API:
- Export `parse_text_magic_file` function
- Update module documentation to describe the text file parsing capability
- Ensure the function is accessible from `file:src/lib.rs` for `MagicDatabase` integration

### 8. Create Unit Tests for Basic Parsing

**In `file:src/parser/mod.rs` (test module):**

Add tests for basic functionality:
- Test parsing a simple single-rule file
- Test parsing multiple top-level rules
- Test parsing rules with hierarchical children (using `>` prefix)
- Test parsing deeply nested rules (level 2, 3, etc.)
- Test empty file handling (should return empty vector)
- Test file with only comments (should return empty vector)

### 9. Create Unit Tests for Line Handling

**In test module:**

Add tests for special line cases:
- Test continuation lines (lines ending with `\`)
- Test multi-line continuations (3+ lines)
- Test mixed content (rules, comments, empty lines, continuations)
- Test comments with various formats (`#`, `# comment`, `  # indented`)
- Test whitespace handling (leading/trailing spaces, tabs)

### 10. Create Unit Tests for Error Cases

**In test module:**

Add tests for error handling:
- Test invalid syntax with proper line number reporting
- Test invalid offset specifications
- Test invalid type specifications
- Test invalid operator specifications
- Test invalid value specifications
- Test hierarchy level skips (e.g., level 0 followed by level 2)
- Test malformed continuation lines

### 11. Create Sample Magic File Test Fixtures

**Create test files in `file:tests/fixtures/` directory:**

Create sample `.magic` files:
- `simple.magic` - Basic rules without hierarchy
- `hierarchical.magic` - Rules with parent-child relationships
- `complex.magic` - Mix of all features (comments, continuations, hierarchy)
- `elf.magic` - Real-world ELF file detection rules
- `zip.magic` - Real-world ZIP archive detection rules
- `invalid_syntax.magic` - File with intentional syntax errors
- `invalid_hierarchy.magic` - File with level skips

### 12. Create Integration Tests with Sample Files

**In `file:tests/parser_integration_tests.rs` (new file):**

Add integration tests:
- Test parsing each sample magic file from fixtures
- Verify correct number of rules parsed
- Verify hierarchy structure is correct
- Verify rule properties (offset, type, operator, value, message)
- Test parsing existing files from `file:third_party/tests/`
- Use snapshot testing with `insta` crate for regression prevention

### 13. Test with Real Magic Files

**In integration tests:**

Test with actual magic files:
- Parse `file:third_party/tests/multiple-A.magic`
- Parse `file:third_party/tests/multiple-B.magic`
- Parse `file:third_party/tests/regex-eol.magic`
- Parse `file:third_party/tests/searchbug.magic`
- Verify parsing succeeds and produces expected rule count
- Handle unsupported features gracefully (e.g., `search`, `regex`, `use`, `name`)

### 14. Add Documentation and Examples

**In `file:src/parser/mod.rs`:**

Add comprehensive documentation:
- Document the `parse_text_magic_file` function with rustdoc comments
- Include usage examples showing how to parse a magic file
- Document the expected format and supported features
- Document error handling and line number reporting
- Add examples of valid magic file syntax
- Reference the magic file format documentation in `file:docs/src/magic-format.md`

## Testing Strategy

| Test Category | Test Files | Coverage |
|--------------|------------|----------|
| Unit Tests | `file:src/parser/mod.rs` | Line processing, continuation, hierarchy building |
| Integration Tests | `file:tests/parser_integration_tests.rs` | End-to-end parsing with sample files |
| Fixture Files | `file:tests/fixtures/*.magic` | Real-world magic file formats |
| Error Cases | Test module in `file:src/parser/mod.rs` | Invalid syntax, hierarchy errors |
| Existing Files | `file:third_party/tests/*.magic` | Compatibility with canonical test suite |

## Dependencies and Integration Points

```mermaid
graph LR
    A[parse_text_magic_file] --> B[parse_magic_rule]
    A --> C[is_empty_line]
    A --> D[is_comment_line]
    A --> E[has_continuation]
    B --> F[parse_rule_offset]
    B --> G[parse_type_and_operator]
    B --> H[parse_value]
    B --> I[parse_message]
    A --> J[ParseError]
    A --> K[MagicRule AST]
    L[MagicDatabase] -.Future.-> A
```

**Key Integration Points:**
- `parse_magic_rule` from `file:src/parser/grammar.rs` - Core rule parsing
- Utility functions from `file:src/parser/grammar.rs` - Line classification
- `ParseError` from `file:src/error.rs` - Error reporting with line numbers
- `MagicRule` from `file:src/parser/ast.rs` - AST representation
- Future integration with `MagicDatabase::load_from_file` in `file:src/lib.rs`

---

Execution Information

Branch: main
Commit: f466e36

New Plan

Observations

src/parser/grammar.rs implements parse_magic_rule with comprehensive tests for offsets, types, operators, values, levels (via > prefixes), whitespace, strings, hex, real-world ELF/ZIP examples, edge/invalid cases. src/parser/ast.rs defines MagicRule with level, children. Utilities like multispace0 exist. third_party/tests/*.magic provides real files. No file-level parser yet; src/error.rs supports line-numbered ParseError. docs/src/magic-format.md details format.

Approach

Orchestrate parse_text_magic_file in src/parser/mod.rs: split lines, skip empty/comments, join continuations (ending \), parse via parse_magic_rule, build hierarchy using stack (push/pop by level diffs, validate no skips >1). Defer special directives (!:mime). Prioritizes incremental build on existing parser, enables MagicDatabase loading. Tests use fixtures + real files for full coverage.

Implementation Steps

1. Define Supporting Structures

File: src/parser/mod.rs

Add:

#[derive(Debug)]
struct LineInfo {
    content: String,
    original_line: usize,
}

2. Implement Line Preprocessing

Function: preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError>

• Split input into lines (1-indexed).
• Buffer for continuations: if line.trim_end().ends_with('\'), append next (strip \n, \).
• Skip empty (trim().is_empty()) or comment (starts_with('#')).
• Strip inline comments (split at first #, keep before).
• Track original_line as start of continuation block.
• Return non-skipped LineInfos.

3. Parse Individual Lines

Function: parse_rule_line(info: &LineInfo) -> Result<MagicRule, ParseError>

• Call parse_magic_rule(info.content.as_str()).
• Map nom::Err to ParseError::invalid_syntax(info.original_line, msg).

4. Build Hierarchy

Function: build_hierarchy(rules: Vec<MagicRule>) -> Result<Vec<MagicRule>, ParseError>

Stack-based:

• VecDeque<MagicRule> stack.
• For each rule:
  • While stack top level >= current level, pop.
  • If level > stack.top().level + 1: error "level jump".
  • If stack not empty, add to stack.top().children.
  • Push current.
• Return stack roots (level 0).

5. Core Parser Function

Public: pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError>

let lines = preprocess_lines(input)?;
let flat_rules: Vec<_> = lines.iter()
    .map(|li| parse_rule_line(li))
    .collect::<Result<_,_>>()?;
build_hierarchy(flat_rules)

Add rustdoc with examples from ticket.

6. Error Enhancements

Extend ParseError usage:

• invalid_syntax(line, msg) for parse fails.
• invalid_hierarchy(line, msg) for level issues.
• Include snippet: content[0..min(50, len)].

flowchart TD
    A[input] --> B[preprocess_lines]
    B --> C[Vec<LineInfo>]
    C --> D[for each: parse_rule_line]
    D --> E[Vec<MagicRule> flat]
    E --> F[build_hierarchy stack]
    F --> G[Vec<MagicRule> tree]
    B -.-> H[continuation<br/>comment/empty skip]
    D -.-> I[parse_magic_rule]
    F -.-> J[level validate<br/>push/pop]

Loading

7. Unit Tests in src/parser/mod.rs

Match ticket:

• test_parse_simple_rule
• test_parse_hierarchical_rules
• test_parse_comments_and_empty_lines
• test_parse_continuation_lines
• test_error_orphaned_child
• test_error_invalid_level_jump
• Add: multi-continuation, inline comments, level 3+, empty file.

Use #[test] with unwrap() assertions.

8. Integration Tests

New file: tests/parser_integration.rs

• Load third_party/tests/multiple-*.magic, assert rule counts.
• Fixtures: create tests/fixtures/ with simple.magic, hierarchical.magic, invalid.magic.
• Snapshot with insta::assert_snapshot!.
• Test real ELF/ZIP excerpts.

Test Type	Count	Coverage
Unit	10+	Line proc, parse, hierarchy
Integration	5+	Real files, fixtures
Error	5+	Syntax, levels, orphans

9. Module Exports & Docs

src/parser/mod.rs:

• pub use grammar::parse_magic_rule;
• Export new functions.
• Update lib.rs docs.

10. Clippy & Validation

• Run cargo clippy.
• Ensure no new warnings.
• Performance: handle 10k+ line files.

Dependencies

graph LR
    parse_text_magic_file --> preprocess_lines
    preprocess_lines --> grammar.rs[is_comment? empty? cont?]
    parse_rule_line --> grammar::parse_magic_rule
    build_hierarchy --> ast::MagicRule{level children}
    parse_text_magic_file --> error::ParseError{line}
    MagicDatabase -.-> parse_text_magic_file

Loading

Next: Integrate into MagicDatabase::from_file(path).

Import In IDE

🤖 Prompt for AI Agents

## Observations

`src/parser/grammar.rs` implements `parse_magic_rule` with comprehensive tests for offsets, types, operators, values, levels (via `>` prefixes), whitespace, strings, hex, real-world ELF/ZIP examples, edge/invalid cases. `src/parser/ast.rs` defines `MagicRule` with `level`, `children`. Utilities like `multispace0` exist. `third_party/tests/*.magic` provides real files. No file-level parser yet; `src/error.rs` supports line-numbered `ParseError`. `docs/src/magic-format.md` details format.

## Approach

Orchestrate `parse_text_magic_file` in `src/parser/mod.rs`: split lines, skip empty/comments, join continuations (ending `\`), parse via `parse_magic_rule`, build hierarchy using stack (push/pop by level diffs, validate no skips >1). Defer special directives (`!:mime`). Prioritizes incremental build on existing parser, enables `MagicDatabase` loading. Tests use fixtures + real files for full coverage.

## Implementation Steps

### 1. Define Supporting Structures

**File:** `src/parser/mod.rs`

Add:
```rust
#[derive(Debug)]
struct LineInfo {
    content: String,
    original_line: usize,
}
```

### 2. Implement Line Preprocessing

**Function:** `preprocess_lines(input: &str) -> Result<Vec<LineInfo>, ParseError>`

- Split `input` into lines (1-indexed).
- Buffer for continuations: if line.trim_end().ends_with('\\'), append next (strip `\n`, `\`).
- Skip empty (`trim().is_empty()`) or comment (`starts_with('#')`).
- Strip inline comments (split at first `#`, keep before).
- Track `original_line` as start of continuation block.
- Return non-skipped `LineInfo`s.

### 3. Parse Individual Lines

**Function:** `parse_rule_line(info: &LineInfo) -> Result<MagicRule, ParseError>`

- Call `parse_magic_rule(info.content.as_str())`.
- Map `nom::Err` to `ParseError::invalid_syntax(info.original_line, msg)`.

### 4. Build Hierarchy

**Function:** `build_hierarchy(rules: Vec<MagicRule>) -> Result<Vec<MagicRule>, ParseError>`

Stack-based:
- `VecDeque<MagicRule>` stack.
- For each rule:
  - While stack top level >= current level, pop.
  - If level > stack.top().level + 1: error "level jump".
  - If stack not empty, add to stack.top().children.
  - Push current.
- Return stack roots (level 0).

### 5. Core Parser Function

**Public:** `pub fn parse_text_magic_file(input: &str) -> Result<Vec<MagicRule>, ParseError>`

```rust
let lines = preprocess_lines(input)?;
let flat_rules: Vec<_> = lines.iter()
    .map(|li| parse_rule_line(li))
    .collect::<Result<_,_>>()?;
build_hierarchy(flat_rules)
```

Add rustdoc with examples from ticket.

### 6. Error Enhancements

Extend `ParseError` usage:
- `invalid_syntax(line, msg)` for parse fails.
- `invalid_hierarchy(line, msg)` for level issues.
- Include snippet: `content[0..min(50, len)]`.

```mermaid
flowchart TD
    A[input] --> B[preprocess_lines]
    B --> C[Vec<LineInfo>]
    C --> D[for each: parse_rule_line]
    D --> E[Vec<MagicRule> flat]
    E --> F[build_hierarchy stack]
    F --> G[Vec<MagicRule> tree]
    B -.-> H[continuation<br/>comment/empty skip]
    D -.-> I[parse_magic_rule]
    F -.-> J[level validate<br/>push/pop]
```

### 7. Unit Tests in `src/parser/mod.rs`

Match ticket:
- `test_parse_simple_rule`
- `test_parse_hierarchical_rules`
- `test_parse_comments_and_empty_lines`
- `test_parse_continuation_lines`
- `test_error_orphaned_child`
- `test_error_invalid_level_jump`
- Add: multi-continuation, inline comments, level 3+, empty file.

Use `#[test]` with `unwrap()` assertions.

### 8. Integration Tests

**New file:** `tests/parser_integration.rs`

- Load `third_party/tests/multiple-*.magic`, assert rule counts.
- Fixtures: create `tests/fixtures/` with simple.magic, hierarchical.magic, invalid.magic.
- Snapshot with `insta::assert_snapshot!`.
- Test real ELF/ZIP excerpts.

| Test Type | Count | Coverage |
|-----------|-------|----------|
| Unit | 10+ | Line proc, parse, hierarchy |
| Integration | 5+ | Real files, fixtures |
| Error | 5+ | Syntax, levels, orphans |

### 9. Module Exports & Docs

**`src/parser/mod.rs`:**
- `pub use grammar::parse_magic_rule;`
- Export new functions.
- Update `lib.rs` docs.

### 10. Clippy & Validation

- Run `cargo clippy`.
- Ensure no new warnings.
- Performance: handle 10k+ line files.

## Dependencies

```mermaid
graph LR
    parse_text_magic_file --> preprocess_lines
    preprocess_lines --> grammar.rs[is_comment? empty? cont?]
    parse_rule_line --> grammar::parse_magic_rule
    build_hierarchy --> ast::MagicRule{level children}
    parse_text_magic_file --> error::ParseError{line}
    MagicDatabase -.-> parse_text_magic_file
```

**Next:** Integrate into `MagicDatabase::from_file(path)`.

---

Execution Information

Branch: main
Commit: 4410a5c