Parser: implement regex and search types

[unclesp1d3r]

Summary

Implement regex and search type specifiers for pattern-based matching in magic rules.

Context

The regex type matches binary-safe regular expressions against file content. The search type searches for a byte sequence within a range. Both are essential for text file detection (JSON, XML, scripts, etc.) and many binary formats.

Current state: Neither type exists in the TypeKind enum or parser.

Acceptance Criteria

• TypeKind enum extended with Regex and Search variants
• Parser recognizes regex and search type specifiers including flags (e.g., regex/c for case-insensitive)
• Evaluator uses regex::bytes::Regex for binary-safe regex matching
• Search type supports range-limited byte sequence searching
• Common regex flags supported: case-insensitive, line-start/end
• Unit tests for regex and search matching
• Integration tests with text file detection rules

Impact

HIGH - Required for text file detection (JSON, XML, HTML, scripts, config files). The test corpus includes JSON tests, shell script tests, and regex-eol.magic that all require this.

Test Corpus Files Blocked

• json1-json8, jsonlines1 (JSON detection)
• cmd1-cmd4 (shell script detection)
• regex-eol (custom magic using regex)
• gedcom (text format detection)

Files to Modify

• src/parser/ast.rs - Extend TypeKind enum
• src/parser/grammar.rs - Parse regex/search type specifiers
• src/evaluator/types.rs - Implement regex/search evaluation
• Cargo.toml - Ensure regex crate bytes feature is enabled

References

• GNU file magic(5) man page
• Epic Epic: libmagic-rs Phase 1 MVP #24

[traycerai]

Plan

Observations

The codebase currently supports basic types (Byte, Short, Long, String) with a clean separation between AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and type evaluation (file:src/evaluator/types.rs). The regex crate is already available in dev-dependencies but needs to be promoted to main dependencies with the bytes feature for binary-safe matching. Test corpus files like regex-eol.magic, searchbug.magic, and JSON/shell script tests are present and waiting for regex/search support.

Approach

Extend the type system with Regex and Search variants following the existing pattern: add enum variants to TypeKind, implement parser combinators for regex/[flags] and search/number syntax, create evaluation functions using regex::bytes::Regex for binary-safe matching, and add comprehensive unit tests. The implementation prioritizes compatibility with GNU file's magic format, supporting common regex flags (case-insensitive, line-based limits) and range-limited byte sequence searching.

Implementation Steps

1. Update Dependencies

File: file:Cargo.toml

• Move regex from [dev-dependencies] to [dependencies] section
• Enable the bytes feature: regex = { version = "1.12.3", features = ["bytes"] }
• This enables binary-safe regex matching required for file content analysis

2. Extend TypeKind Enum

File: file:src/parser/ast.rs

Add two new variants to the TypeKind enum (around line 78):

/// Regular expression pattern matching (binary-safe)
Regex {
    /// Regex flags: 'c' (case-insensitive), 's' (start offset), 'l' (line-based limit)
    flags: RegexFlags,
    /// Search range in bytes (or lines if 'l' flag set)
    range: Option<usize>,
},
/// Byte sequence search within a range
Search {
    /// Number of positions to attempt match from start offset
    range: usize,
},

Define a new struct for regex flags:

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct RegexFlags {
    /// Case-insensitive matching
    pub case_insensitive: bool,
    /// Update offset to start of match (instead of end)
    pub start_offset: bool,
    /// Limit is in lines instead of bytes
    pub line_based: bool,
}

Add unit tests for the new variants following the existing test pattern (lines 515-587).

3. Implement Parser for Regex Type

File: file:src/parser/grammar.rs

Add a parser function for regex flags (insert around line 1400):

/// Parse regex flags: /[c][s][l][number]
fn parse_regex_flags(input: &str) -> IResult<&str, (RegexFlags, Option<usize>)>

This parser should:

• Match the / character followed by optional flag characters
• Parse c, s, l flags in any order
• Parse optional numeric range after flags (e.g., /1l means 1 line)
• Return RegexFlags struct and optional range value

Update parse_type_and_operator function (line 1429) to include:

• Add tag("regex") to the alt combinator list
• After matching "regex", call parse_regex_flags to extract flags
• Construct TypeKind::Regex { flags, range } variant

Add unit tests for regex type parsing covering:

• regex (no flags)
• regex/c (case-insensitive)
• regex/1l (1 line limit)
• regex/cs (multiple flags)
• Invalid flag combinations

4. Implement Parser for Search Type

File: file:src/parser/grammar.rs

Update parse_type_and_operator function (line 1429):

• Add tag("search") to the alt combinator list
• After matching "search", expect / followed by a number using parse_number
• Construct TypeKind::Search { range } variant
• The range is mandatory for search type (per GNU file specification)

Add unit tests for search type parsing:

• search/12 (search within 12 bytes)
• search/256 (larger range)
• Error cases: search without range, search/ without number

5. Implement Regex Evaluation

File: file:src/evaluator/types.rs

Add a new function read_regex (insert around line 300):

/// Reads and matches a regex pattern against buffer content
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The regex pattern (from Value::String or Value::Bytes)
/// * `flags` - Regex flags for matching behavior
/// * `range` - Optional search range limit
///
/// # Returns
/// Returns Value::Bytes containing the matched content, or error if no match
pub fn read_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Value, TypeReadError>

Implementation details:

• Use regex::bytes::RegexBuilder to construct regex with flags
• Set case-insensitive flag if flags.case_insensitive is true
• Determine search window: if flags.line_based, count lines; otherwise use byte range
• Default range to 8192 bytes if not specified (per GNU file behavior)
• Use regex.find() to locate match in the search window
• Return Value::Bytes with matched content
• Handle regex compilation errors gracefully

Add comprehensive unit tests:

• Basic regex matching: [0-9]+, [a-zA-Z]+
• Case-insensitive matching with /c flag
• Line-based limits with /l flag
• Binary-safe matching with non-printable characters
• No match scenarios
• Invalid regex patterns

6. Implement Search Evaluation

File: file:src/evaluator/types.rs

Add a new function read_search (insert around line 350):

/// Searches for a byte sequence within a specified range
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The byte sequence to find (from Value::Bytes or Value::String)
/// * `range` - Number of positions to attempt match
///
/// # Returns
/// Returns Value::Bytes containing the found sequence, or error if not found
pub fn read_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Value, TypeReadError>

Implementation details:

• Extract pattern bytes from Value::Bytes or Value::String
• Use memchr crate (already a dependency) for efficient byte searching
• Search within buffer[offset..offset+range] window
• Use sliding window approach to find exact byte sequence match
• Return Value::Bytes with the matched sequence
• Return error if pattern not found within range

Add comprehensive unit tests:

• Simple byte sequence search: ABC, \x7fELF
• Search at various offsets
• Pattern found at start/middle/end of range
• Pattern not found within range
• Edge cases: empty pattern, range too small, offset beyond buffer
• Boundary conditions: pattern at exact range end

7. Update Type Dispatcher

File: src/evaluator/types.rs

Update read_typed_value function (line 348) to handle new types:

match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::Regex { flags, range } => {
        // For regex, the pattern comes from the rule's value, not the buffer
        // This requires special handling in the evaluator module
        Err(TypeReadError::UnsupportedType("Regex matching requires pattern from rule value".to_string()))
    },
    TypeKind::Search { range } => {
        // Similar to regex, search needs the pattern from the rule
        Err(TypeReadError::UnsupportedType("Search requires pattern from rule value".to_string()))
    },
}

Note: Regex and search types differ from other types because they need both the buffer AND the expected pattern from the rule. This requires special handling in the evaluator module's evaluate_single_rule function.

8. Refactor Evaluator for Pattern-Based Types

File: src/evaluator/mod.rs

Update evaluate_single_rule function (around line 246) to handle regex/search specially:

• Before calling read_typed_value, check if rule.typ is Regex or Search
• For these types, call specialized matching functions that take both buffer and pattern:
  • match_regex(buffer, offset, &rule.value, flags, range)
  • match_search(buffer, offset, &rule.value, range)
• These functions should return Result<Option<(usize, Value)>, LibmagicError> where the usize is the match offset
• For regex with start_offset flag, return start of match; otherwise return end
• For search, return the offset where pattern was found
• Update the offset tracking to use the returned match offset for child rule evaluation

Add helper functions in src/evaluator/mod.rs:

fn match_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Option<(usize, Value)>, LibmagicError>

fn match_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Option<(usize, Value)>, LibmagicError>

These functions should:

• Call the corresponding read_regex or read_search from the types module
• Handle the returned match offset for subsequent child rule evaluation
• Return None if no match is found (allowing rule evaluation to continue)
• Return Some((match_offset, matched_value)) on successful match

9. Integration Tests

File: tests/evaluator_tests.rs or create new tests/regex_search_tests.rs

Add integration tests using the test corpus files:

• Test regex-eol.magic with regex-eol.testfile (Ansible Vault detection with version/encryption)
• Test searchbug.magic with searchbug.testfile (search type with byte sequence detection)
• Test JSON detection patterns (if available in test corpus)
• Test shell script detection (if available in test corpus)

Each test should:

• Load the magic file
• Parse the rules
• Evaluate against the test file
• Verify that regex/search rules match correctly
• Compare output with expected .result file if available

10. Parser Integration Tests

File: tests/parser_integration_tests.rs

Add tests for parsing magic files with regex/search types:

• Parse regex-eol.magic and verify AST structure (check for Regex variants with /1l flags)
• Parse searchbug.magic and verify AST structure (check for Search variants with ranges)
• Verify regex flags are correctly parsed (c, s, l combinations)
• Verify search ranges are correctly parsed
• Test error handling for malformed regex/search specifications

11. Documentation

Update module documentation:

File: src/parser/ast.rs

• Add doc comments for Regex and Search variants with examples
• Document RegexFlags struct with usage examples
• Include examples showing typical regex/search patterns from test corpus

File: src/evaluator/types.rs

• Add doc comments for read_regex and read_search functions
• Include examples showing typical usage patterns
• Document the binary-safe nature of regex matching

File: src/evaluator/mod.rs

• Add doc comments for match_regex and match_search helper functions
• Document how pattern-based types are handled differently from value-based types

12. Error Handling

Ensure proper error handling throughout:

• Regex compilation errors should be caught and converted to TypeReadError::UnsupportedType
• Invalid regex patterns should not panic
• Search range overflow should be handled gracefully with bounds checking
• Buffer boundary checks for both regex and search operations
• Proper error propagation in the evaluator module

Implementation Sequence

The implementation follows this sequence:

1. Dependencies - Move regex to main dependencies with bytes feature
2. AST - Add RegexFlags struct and Regex/Search variants to TypeKind
3. Parser - Implement parse_regex_flags and update parse_type_and_operator to handle both types
4. Type Evaluation - Implement read_regex and read_search functions in types module
5. Evaluator Integration - Update evaluate_single_rule to handle pattern-based types specially
6. Tests - Add unit tests for parsing and evaluation, plus integration tests with test corpus
7. Documentation - Update doc comments and examples

Key Design Decisions

• Pattern-based types are handled specially in the evaluator because they need both the buffer and the rule's expected value, unlike value-based types
• Default range for regex is 8192 bytes (per GNU file behavior) if not specified
• Search range is mandatory (per GNU file specification)
• Regex flags support c (case-insensitive), s (start-offset), l (line-based) in any order
• Binary-safe matching uses regex::bytes::Regex for both regex and search operations
• Error handling converts regex compilation errors to TypeReadError::UnsupportedType for consistency

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase currently supports basic types (`Byte`, `Short`, `Long`, `String`) with a clean separation between AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and type evaluation (file:src/evaluator/types.rs). The `regex` crate is already available in dev-dependencies but needs to be promoted to main dependencies with the `bytes` feature for binary-safe matching. Test corpus files like `regex-eol.magic`, `searchbug.magic`, and JSON/shell script tests are present and waiting for regex/search support.

## Approach

Extend the type system with `Regex` and `Search` variants following the existing pattern: add enum variants to `TypeKind`, implement parser combinators for `regex/[flags]` and `search/number` syntax, create evaluation functions using `regex::bytes::Regex` for binary-safe matching, and add comprehensive unit tests. The implementation prioritizes compatibility with GNU file's magic format, supporting common regex flags (case-insensitive, line-based limits) and range-limited byte sequence searching.

## Implementation Steps

### 1. Update Dependencies

**File: file:Cargo.toml**

- Move `regex` from `[dev-dependencies]` to `[dependencies]` section
- Enable the `bytes` feature: `regex = { version = "1.12.3", features = ["bytes"] }`
- This enables binary-safe regex matching required for file content analysis

### 2. Extend TypeKind Enum

**File: file:src/parser/ast.rs**

Add two new variants to the `TypeKind` enum (around line 78):

```rust
/// Regular expression pattern matching (binary-safe)
Regex {
    /// Regex flags: 'c' (case-insensitive), 's' (start offset), 'l' (line-based limit)
    flags: RegexFlags,
    /// Search range in bytes (or lines if 'l' flag set)
    range: Option<usize>,
},
/// Byte sequence search within a range
Search {
    /// Number of positions to attempt match from start offset
    range: usize,
},
```

Define a new struct for regex flags:

```rust
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct RegexFlags {
    /// Case-insensitive matching
    pub case_insensitive: bool,
    /// Update offset to start of match (instead of end)
    pub start_offset: bool,
    /// Limit is in lines instead of bytes
    pub line_based: bool,
}
```

Add unit tests for the new variants following the existing test pattern (lines 515-587).

### 3. Implement Parser for Regex Type

**File: file:src/parser/grammar.rs**

Add a parser function for regex flags (insert around line 1400):

```rust
/// Parse regex flags: /[c][s][l][number]
fn parse_regex_flags(input: &str) -> IResult<&str, (RegexFlags, Option<usize>)>
```

This parser should:
- Match the `/` character followed by optional flag characters
- Parse `c`, `s`, `l` flags in any order
- Parse optional numeric range after flags (e.g., `/1l` means 1 line)
- Return `RegexFlags` struct and optional range value

Update `parse_type_and_operator` function (line 1429) to include:
- Add `tag("regex")` to the `alt` combinator list
- After matching "regex", call `parse_regex_flags` to extract flags
- Construct `TypeKind::Regex { flags, range }` variant

Add unit tests for regex type parsing covering:
- `regex` (no flags)
- `regex/c` (case-insensitive)
- `regex/1l` (1 line limit)
- `regex/cs` (multiple flags)
- Invalid flag combinations

### 4. Implement Parser for Search Type

**File: file:src/parser/grammar.rs**

Update `parse_type_and_operator` function (line 1429):
- Add `tag("search")` to the `alt` combinator list
- After matching "search", expect `/` followed by a number using `parse_number`
- Construct `TypeKind::Search { range }` variant
- The range is mandatory for search type (per GNU file specification)

Add unit tests for search type parsing:
- `search/12` (search within 12 bytes)
- `search/256` (larger range)
- Error cases: `search` without range, `search/` without number

### 5. Implement Regex Evaluation

**File: file:src/evaluator/types.rs**

Add a new function `read_regex` (insert around line 300):

```rust
/// Reads and matches a regex pattern against buffer content
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The regex pattern (from Value::String or Value::Bytes)
/// * `flags` - Regex flags for matching behavior
/// * `range` - Optional search range limit
///
/// # Returns
/// Returns Value::Bytes containing the matched content, or error if no match
pub fn read_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Value, TypeReadError>
```

Implementation details:
- Use `regex::bytes::RegexBuilder` to construct regex with flags
- Set case-insensitive flag if `flags.case_insensitive` is true
- Determine search window: if `flags.line_based`, count lines; otherwise use byte range
- Default range to 8192 bytes if not specified (per GNU file behavior)
- Use `regex.find()` to locate match in the search window
- Return `Value::Bytes` with matched content
- Handle regex compilation errors gracefully

Add comprehensive unit tests:
- Basic regex matching: `[0-9]+`, `[a-zA-Z]+`
- Case-insensitive matching with `/c` flag
- Line-based limits with `/l` flag
- Binary-safe matching with non-printable characters
- No match scenarios
- Invalid regex patterns

### 6. Implement Search Evaluation

**File: file:src/evaluator/types.rs**

Add a new function `read_search` (insert around line 350):

```rust
/// Searches for a byte sequence within a specified range
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The byte sequence to find (from Value::Bytes or Value::String)
/// * `range` - Number of positions to attempt match
///
/// # Returns
/// Returns Value::Bytes containing the found sequence, or error if not found
pub fn read_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Value, TypeReadError>
```

Implementation details:
- Extract pattern bytes from `Value::Bytes` or `Value::String`
- Use `memchr` crate (already a dependency) for efficient byte searching
- Search within `buffer[offset..offset+range]` window
- Use sliding window approach to find exact byte sequence match
- Return `Value::Bytes` with the matched sequence
- Return error if pattern not found within range

Add comprehensive unit tests:
- Simple byte sequence search: `ABC`, `\x7fELF`
- Search at various offsets
- Pattern found at start/middle/end of range
- Pattern not found within range
- Edge cases: empty pattern, range too small, offset beyond buffer
- Boundary conditions: pattern at exact range end

### 7. Update Type Dispatcher

**File: `src/evaluator/types.rs`**

Update `read_typed_value` function (line 348) to handle new types:

```rust
match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::Regex { flags, range } => {
        // For regex, the pattern comes from the rule's value, not the buffer
        // This requires special handling in the evaluator module
        Err(TypeReadError::UnsupportedType("Regex matching requires pattern from rule value".to_string()))
    },
    TypeKind::Search { range } => {
        // Similar to regex, search needs the pattern from the rule
        Err(TypeReadError::UnsupportedType("Search requires pattern from rule value".to_string()))
    },
}
```

**Note:** Regex and search types differ from other types because they need both the buffer AND the expected pattern from the rule. This requires special handling in the evaluator module's `evaluate_single_rule` function.

### 8. Refactor Evaluator for Pattern-Based Types

**File: `src/evaluator/mod.rs`**

Update `evaluate_single_rule` function (around line 246) to handle regex/search specially:

- Before calling `read_typed_value`, check if `rule.typ` is `Regex` or `Search`
- For these types, call specialized matching functions that take both buffer and pattern:
  - `match_regex(buffer, offset, &rule.value, flags, range)`
  - `match_search(buffer, offset, &rule.value, range)`
- These functions should return `Result<Option<(usize, Value)>, LibmagicError>` where the usize is the match offset
- For regex with `start_offset` flag, return start of match; otherwise return end
- For search, return the offset where pattern was found
- Update the offset tracking to use the returned match offset for child rule evaluation

Add helper functions in `src/evaluator/mod.rs`:

```rust
fn match_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Option<(usize, Value)>, LibmagicError>

fn match_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Option<(usize, Value)>, LibmagicError>
```

These functions should:
- Call the corresponding `read_regex` or `read_search` from the types module
- Handle the returned match offset for subsequent child rule evaluation
- Return `None` if no match is found (allowing rule evaluation to continue)
- Return `Some((match_offset, matched_value))` on successful match

### 9. Integration Tests

**File: `tests/evaluator_tests.rs` or create new `tests/regex_search_tests.rs`**

Add integration tests using the test corpus files:

- Test `regex-eol.magic` with `regex-eol.testfile` (Ansible Vault detection with version/encryption)
- Test `searchbug.magic` with `searchbug.testfile` (search type with byte sequence detection)
- Test JSON detection patterns (if available in test corpus)
- Test shell script detection (if available in test corpus)

Each test should:
- Load the magic file
- Parse the rules
- Evaluate against the test file
- Verify that regex/search rules match correctly
- Compare output with expected `.result` file if available

### 10. Parser Integration Tests

**File: `tests/parser_integration_tests.rs`**

Add tests for parsing magic files with regex/search types:

- Parse `regex-eol.magic` and verify AST structure (check for `Regex` variants with `/1l` flags)
- Parse `searchbug.magic` and verify AST structure (check for `Search` variants with ranges)
- Verify regex flags are correctly parsed (`c`, `s`, `l` combinations)
- Verify search ranges are correctly parsed
- Test error handling for malformed regex/search specifications

### 11. Documentation

Update module documentation:

**File: `src/parser/ast.rs`**
- Add doc comments for `Regex` and `Search` variants with examples
- Document `RegexFlags` struct with usage examples
- Include examples showing typical regex/search patterns from test corpus

**File: `src/evaluator/types.rs`**
- Add doc comments for `read_regex` and `read_search` functions
- Include examples showing typical usage patterns
- Document the binary-safe nature of regex matching

**File: `src/evaluator/mod.rs`**
- Add doc comments for `match_regex` and `match_search` helper functions
- Document how pattern-based types are handled differently from value-based types

### 12. Error Handling

Ensure proper error handling throughout:

- Regex compilation errors should be caught and converted to `TypeReadError::UnsupportedType`
- Invalid regex patterns should not panic
- Search range overflow should be handled gracefully with bounds checking
- Buffer boundary checks for both regex and search operations
- Proper error propagation in the evaluator module

### Implementation Sequence

The implementation follows this sequence:

1. **Dependencies** - Move regex to main dependencies with bytes feature
2. **AST** - Add `RegexFlags` struct and `Regex`/`Search` variants to `TypeKind`
3. **Parser** - Implement `parse_regex_flags` and update `parse_type_and_operator` to handle both types
4. **Type Evaluation** - Implement `read_regex` and `read_search` functions in types module
5. **Evaluator Integration** - Update `evaluate_single_rule` to handle pattern-based types specially
6. **Tests** - Add unit tests for parsing and evaluation, plus integration tests with test corpus
7. **Documentation** - Update doc comments and examples

### Key Design Decisions

- **Pattern-based types** are handled specially in the evaluator because they need both the buffer and the rule's expected value, unlike value-based types
- **Default range** for regex is 8192 bytes (per GNU file behavior) if not specified
- **Search range** is mandatory (per GNU file specification)
- **Regex flags** support `c` (case-insensitive), `s` (start-offset), `l` (line-based) in any order
- **Binary-safe matching** uses `regex::bytes::Regex` for both regex and search operations
- **Error handling** converts regex compilation errors to `TypeReadError::UnsupportedType` for consistency

---

Execution Information

Branch: main
Commit: 36adac5

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

Previous Plan

Observations

The codebase currently supports basic types (Byte, Short, Long, String) with a clean separation between AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and type evaluation (file:src/evaluator/types.rs). The regex crate is already available in dev-dependencies but needs to be promoted to main dependencies with the bytes feature for binary-safe matching. Test corpus files like regex-eol.magic, searchbug.magic, and JSON/shell script tests are present and waiting for regex/search support.

Approach

Extend the type system with Regex and Search variants following the existing pattern: add enum variants to TypeKind, implement parser combinators for regex/[flags] and search/number syntax, create evaluation functions using regex::bytes::Regex for binary-safe matching, and add comprehensive unit tests. The implementation prioritizes compatibility with GNU file's magic format, supporting common regex flags (case-insensitive, line-based limits) and range-limited byte sequence searching.

Implementation Steps

1. Update Dependencies

File: file:Cargo.toml

• Move regex from [dev-dependencies] to [dependencies] section
• Enable the bytes feature: regex = { version = "1.12.3", features = ["bytes"] }
• This enables binary-safe regex matching required for file content analysis

2. Extend TypeKind Enum

File: file:src/parser/ast.rs

Add two new variants to the TypeKind enum (around line 78):

/// Regular expression pattern matching (binary-safe)
Regex {
    /// Regex flags: 'c' (case-insensitive), 's' (start offset), 'l' (line-based limit)
    flags: RegexFlags,
    /// Search range in bytes (or lines if 'l' flag set)
    range: Option<usize>,
},
/// Byte sequence search within a range
Search {
    /// Number of positions to attempt match from start offset
    range: usize,
},

Define a new struct for regex flags:

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct RegexFlags {
    /// Case-insensitive matching
    pub case_insensitive: bool,
    /// Update offset to start of match (instead of end)
    pub start_offset: bool,
    /// Limit is in lines instead of bytes
    pub line_based: bool,
}

Add unit tests for the new variants following the existing test pattern (lines 515-587).

3. Implement Parser for Regex Type

File: file:src/parser/grammar.rs

Add a parser function for regex flags (insert around line 1400):

/// Parse regex flags: /[c][s][l][number]
fn parse_regex_flags(input: &str) -> IResult<&str, (RegexFlags, Option<usize>)>

This parser should:

• Match the / character followed by optional flag characters
• Parse c, s, l flags in any order
• Parse optional numeric range after flags (e.g., /1l means 1 line)
• Return RegexFlags struct and optional range value

Update parse_type_and_operator function (line 1429) to include:

• Add tag("regex") to the alt combinator list
• After matching "regex", call parse_regex_flags to extract flags
• Construct TypeKind::Regex { flags, range } variant

Add unit tests for regex type parsing covering:

• regex (no flags)
• regex/c (case-insensitive)
• regex/1l (1 line limit)
• regex/cs (multiple flags)
• Invalid flag combinations

4. Implement Parser for Search Type

File: file:src/parser/grammar.rs

Update parse_type_and_operator function (line 1429):

• Add tag("search") to the alt combinator list
• After matching "search", expect / followed by a number using parse_number
• Construct TypeKind::Search { range } variant
• The range is mandatory for search type (per GNU file specification)

Add unit tests for search type parsing:

• search/12 (search within 12 bytes)
• search/256 (larger range)
• Error cases: search without range, search/ without number

5. Implement Regex Evaluation

File: file:src/evaluator/types.rs

Add a new function read_regex (insert around line 300):

/// Reads and matches a regex pattern against buffer content
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The regex pattern (from Value::String or Value::Bytes)
/// * `flags` - Regex flags for matching behavior
/// * `range` - Optional search range limit
///
/// # Returns
/// Returns Value::Bytes containing the matched content, or error if no match
pub fn read_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Value, TypeReadError>

Implementation details:

• Use regex::bytes::RegexBuilder to construct regex with flags
• Set case-insensitive flag if flags.case_insensitive is true
• Determine search window: if flags.line_based, count lines; otherwise use byte range
• Default range to 8192 bytes if not specified (per GNU file behavior)
• Use regex.find() to locate match in the search window
• Return Value::Bytes with matched content
• Handle regex compilation errors gracefully

Add comprehensive unit tests:

• Basic regex matching: [0-9]+, [a-zA-Z]+
• Case-insensitive matching with /c flag
• Line-based limits with /l flag
• Binary-safe matching with non-printable characters
• No match scenarios
• Invalid regex patterns

6. Implement Search Evaluation

File: file:src/evaluator/types.rs

Add a new function read_search (insert around line 350):

/// Searches for a byte sequence within a specified range
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The byte sequence to find (from Value::Bytes or Value::String)
/// * `range` - Number of positions to attempt match
///
/// # Returns
/// Returns Value::Bytes containing the found sequence, or error if not found
pub fn read_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Value, TypeReadError>

Implementation details:

• Extract pattern bytes from Value::Bytes or Value::String
• Use memchr crate (already a dependency) for efficient byte searching
• Search within buffer[offset..offset+range] window
• Use sliding window approach to find exact byte sequence match
• Return Value::Bytes with the matched sequence
• Return error if pattern not found within range

Add comprehensive unit tests:

• Simple byte sequence search: ABC, \x7fELF
• Search at various offsets
• Pattern found at start/middle/end of range
• Pattern not found within range
• Edge cases: empty pattern, range too small, offset beyond buffer

7. Update Type Dispatcher

File: file:src/evaluator/types.rs

Update read_typed_value function (line 348) to handle new types:

match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::Regex { flags, range } => {
        // For regex, the pattern comes from the rule's value, not the buffer
        // This requires refactoring to pass the expected value to read_typed_value
        // or handling regex matching in the evaluator module instead
        Err(TypeReadError::UnsupportedType("Regex matching requires pattern from rule value".to_string()))
    },
    TypeKind::Search { range } => {
        // Similar to regex, search needs the pattern from the rule
        Err(TypeReadError::UnsupportedType("Search requires pattern from rule value".to_string()))
    },
}

Note: Regex and search types differ from other types because they need both the buffer AND the expected pattern from the rule. This requires refactoring the evaluation flow.

8. Refactor Evaluator for Pattern-Based Types

File: file:src/evaluator/mod.rs

Update evaluate_single_rule function (around line 299) to handle regex/search specially:

• Before calling read_typed_value, check if rule.typ is Regex or Search
• For these types, call specialized matching functions that take both buffer and pattern:
  • match_regex(buffer, offset, &rule.value, flags, range)
  • match_search(buffer, offset, &rule.value, range)
• These functions should return Option<(usize, Value)> where the usize is the match offset
• For regex with start_offset flag, return start of match; otherwise return end
• For search, return the offset where pattern was found

Add helper functions in file:src/evaluator/mod.rs:

fn match_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Option<(usize, Value)>, LibmagicError>

fn match_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Option<(usize, Value)>, LibmagicError>

9. Integration Tests

File: file:tests/evaluator_tests.rs (or create new file:tests/regex_search_tests.rs)

Add integration tests using the test corpus files:

• Test regex-eol.magic with regex-eol.testfile (Ansible Vault detection)
• Test JSON detection patterns (json1-json8 test files)
• Test shell script detection (cmd1-cmd4 test files)
• Test searchbug.magic with searchbug.testfile
• Test GEDCOM text format detection

Each test should:

• Load the magic file
• Parse the rules
• Evaluate against the test file
• Compare output with expected .result file

10. Parser Integration Tests

File: file:tests/parser_integration_tests.rs

Add tests for parsing magic files with regex/search types:

• Parse regex-eol.magic and verify AST structure
• Parse searchbug.magic and verify AST structure
• Verify regex flags are correctly parsed
• Verify search ranges are correctly parsed
• Test error handling for malformed regex/search specifications

11. Documentation

Update module documentation:

File: file:src/parser/ast.rs

• Add doc comments for Regex and Search variants with examples
• Document RegexFlags struct with usage examples

File: file:src/evaluator/types.rs

• Add doc comments for read_regex and read_search functions
• Include examples showing typical usage patterns

File: file:README.md (if applicable)

• Update feature list to mention regex and search type support
• Add examples of magic rules using regex/search

12. Error Handling

Ensure proper error handling throughout:

• Regex compilation errors should be caught and converted to TypeReadError
• Invalid regex patterns should not panic
• Search range overflow should be handled gracefully
• Buffer boundary checks for both regex and search operations

Implementation Sequence Diagram

sequenceDiagram
    participant P as Parser
    participant A as AST
    participant E as Evaluator
    participant R as Regex Engine
    
    P->>P: parse_type_and_operator("regex/c")
    P->>P: parse_regex_flags("/c")
    P->>A: Create TypeKind::Regex{flags, range}
    
    E->>E: evaluate_single_rule(rule, buffer)
    E->>E: Check if Regex/Search type
    E->>E: match_regex(buffer, offset, pattern, flags)
    E->>R: RegexBuilder::new(pattern).case_insensitive(true)
    R->>R: Compile regex
    R->>R: find(buffer[offset..offset+range])
    R-->>E: Match result
    E-->>E: Return (match_offset, matched_bytes)

Loading

Type Hierarchy

classDiagram
    class TypeKind {
        <<enumeration>>
        Byte
        Short
        Long
        String
        Regex
        Search
    }
    
    class RegexFlags {
        +bool case_insensitive
        +bool start_offset
        +bool line_based
    }
    
    class Regex {
        +RegexFlags flags
        +Option~usize~ range
    }
    
    class Search {
        +usize range
    }
    
    TypeKind --> Regex
    TypeKind --> Search
    Regex --> RegexFlags

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase currently supports basic types (`Byte`, `Short`, `Long`, `String`) with a clean separation between AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and type evaluation (file:src/evaluator/types.rs). The `regex` crate is already available in dev-dependencies but needs to be promoted to main dependencies with the `bytes` feature for binary-safe matching. Test corpus files like `regex-eol.magic`, `searchbug.magic`, and JSON/shell script tests are present and waiting for regex/search support.

## Approach

Extend the type system with `Regex` and `Search` variants following the existing pattern: add enum variants to `TypeKind`, implement parser combinators for `regex/[flags]` and `search/number` syntax, create evaluation functions using `regex::bytes::Regex` for binary-safe matching, and add comprehensive unit tests. The implementation prioritizes compatibility with GNU file's magic format, supporting common regex flags (case-insensitive, line-based limits) and range-limited byte sequence searching.

## Implementation Steps

### 1. Update Dependencies

**File: file:Cargo.toml**

- Move `regex` from `[dev-dependencies]` to `[dependencies]` section
- Enable the `bytes` feature: `regex = { version = "1.12.3", features = ["bytes"] }`
- This enables binary-safe regex matching required for file content analysis

### 2. Extend TypeKind Enum

**File: file:src/parser/ast.rs**

Add two new variants to the `TypeKind` enum (around line 78):

```rust
/// Regular expression pattern matching (binary-safe)
Regex {
    /// Regex flags: 'c' (case-insensitive), 's' (start offset), 'l' (line-based limit)
    flags: RegexFlags,
    /// Search range in bytes (or lines if 'l' flag set)
    range: Option<usize>,
},
/// Byte sequence search within a range
Search {
    /// Number of positions to attempt match from start offset
    range: usize,
},
```

Define a new struct for regex flags:

```rust
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct RegexFlags {
    /// Case-insensitive matching
    pub case_insensitive: bool,
    /// Update offset to start of match (instead of end)
    pub start_offset: bool,
    /// Limit is in lines instead of bytes
    pub line_based: bool,
}
```

Add unit tests for the new variants following the existing test pattern (lines 515-587).

### 3. Implement Parser for Regex Type

**File: file:src/parser/grammar.rs**

Add a parser function for regex flags (insert around line 1400):

```rust
/// Parse regex flags: /[c][s][l][number]
fn parse_regex_flags(input: &str) -> IResult<&str, (RegexFlags, Option<usize>)>
```

This parser should:
- Match the `/` character followed by optional flag characters
- Parse `c`, `s`, `l` flags in any order
- Parse optional numeric range after flags (e.g., `/1l` means 1 line)
- Return `RegexFlags` struct and optional range value

Update `parse_type_and_operator` function (line 1429) to include:
- Add `tag("regex")` to the `alt` combinator list
- After matching "regex", call `parse_regex_flags` to extract flags
- Construct `TypeKind::Regex { flags, range }` variant

Add unit tests for regex type parsing covering:
- `regex` (no flags)
- `regex/c` (case-insensitive)
- `regex/1l` (1 line limit)
- `regex/cs` (multiple flags)
- Invalid flag combinations

### 4. Implement Parser for Search Type

**File: file:src/parser/grammar.rs**

Update `parse_type_and_operator` function (line 1429):
- Add `tag("search")` to the `alt` combinator list
- After matching "search", expect `/` followed by a number using `parse_number`
- Construct `TypeKind::Search { range }` variant
- The range is mandatory for search type (per GNU file specification)

Add unit tests for search type parsing:
- `search/12` (search within 12 bytes)
- `search/256` (larger range)
- Error cases: `search` without range, `search/` without number

### 5. Implement Regex Evaluation

**File: file:src/evaluator/types.rs**

Add a new function `read_regex` (insert around line 300):

```rust
/// Reads and matches a regex pattern against buffer content
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The regex pattern (from Value::String or Value::Bytes)
/// * `flags` - Regex flags for matching behavior
/// * `range` - Optional search range limit
///
/// # Returns
/// Returns Value::Bytes containing the matched content, or error if no match
pub fn read_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Value, TypeReadError>
```

Implementation details:
- Use `regex::bytes::RegexBuilder` to construct regex with flags
- Set case-insensitive flag if `flags.case_insensitive` is true
- Determine search window: if `flags.line_based`, count lines; otherwise use byte range
- Default range to 8192 bytes if not specified (per GNU file behavior)
- Use `regex.find()` to locate match in the search window
- Return `Value::Bytes` with matched content
- Handle regex compilation errors gracefully

Add comprehensive unit tests:
- Basic regex matching: `[0-9]+`, `[a-zA-Z]+`
- Case-insensitive matching with `/c` flag
- Line-based limits with `/l` flag
- Binary-safe matching with non-printable characters
- No match scenarios
- Invalid regex patterns

### 6. Implement Search Evaluation

**File: file:src/evaluator/types.rs**

Add a new function `read_search` (insert around line 350):

```rust
/// Searches for a byte sequence within a specified range
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The byte sequence to find (from Value::Bytes or Value::String)
/// * `range` - Number of positions to attempt match
///
/// # Returns
/// Returns Value::Bytes containing the found sequence, or error if not found
pub fn read_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Value, TypeReadError>
```

Implementation details:
- Extract pattern bytes from `Value::Bytes` or `Value::String`
- Use `memchr` crate (already a dependency) for efficient byte searching
- Search within `buffer[offset..offset+range]` window
- Use sliding window approach to find exact byte sequence match
- Return `Value::Bytes` with the matched sequence
- Return error if pattern not found within range

Add comprehensive unit tests:
- Simple byte sequence search: `ABC`, `\x7fELF`
- Search at various offsets
- Pattern found at start/middle/end of range
- Pattern not found within range
- Edge cases: empty pattern, range too small, offset beyond buffer

### 7. Update Type Dispatcher

**File: file:src/evaluator/types.rs**

Update `read_typed_value` function (line 348) to handle new types:

```rust
match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::Regex { flags, range } => {
        // For regex, the pattern comes from the rule's value, not the buffer
        // This requires refactoring to pass the expected value to read_typed_value
        // or handling regex matching in the evaluator module instead
        Err(TypeReadError::UnsupportedType("Regex matching requires pattern from rule value".to_string()))
    },
    TypeKind::Search { range } => {
        // Similar to regex, search needs the pattern from the rule
        Err(TypeReadError::UnsupportedType("Search requires pattern from rule value".to_string()))
    },
}
```

**Note:** Regex and search types differ from other types because they need both the buffer AND the expected pattern from the rule. This requires refactoring the evaluation flow.

### 8. Refactor Evaluator for Pattern-Based Types

**File: file:src/evaluator/mod.rs**

Update `evaluate_single_rule` function (around line 299) to handle regex/search specially:

- Before calling `read_typed_value`, check if `rule.typ` is `Regex` or `Search`
- For these types, call specialized matching functions that take both buffer and pattern:
  - `match_regex(buffer, offset, &rule.value, flags, range)`
  - `match_search(buffer, offset, &rule.value, range)`
- These functions should return `Option<(usize, Value)>` where the usize is the match offset
- For regex with `start_offset` flag, return start of match; otherwise return end
- For search, return the offset where pattern was found

Add helper functions in file:src/evaluator/mod.rs:

```rust
fn match_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Option<(usize, Value)>, LibmagicError>

fn match_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Option<(usize, Value)>, LibmagicError>
```

### 9. Integration Tests

**File: file:tests/evaluator_tests.rs** (or create new file:tests/regex_search_tests.rs)

Add integration tests using the test corpus files:

- Test `regex-eol.magic` with `regex-eol.testfile` (Ansible Vault detection)
- Test JSON detection patterns (json1-json8 test files)
- Test shell script detection (cmd1-cmd4 test files)
- Test `searchbug.magic` with `searchbug.testfile`
- Test GEDCOM text format detection

Each test should:
- Load the magic file
- Parse the rules
- Evaluate against the test file
- Compare output with expected `.result` file

### 10. Parser Integration Tests

**File: file:tests/parser_integration_tests.rs**

Add tests for parsing magic files with regex/search types:

- Parse `regex-eol.magic` and verify AST structure
- Parse `searchbug.magic` and verify AST structure
- Verify regex flags are correctly parsed
- Verify search ranges are correctly parsed
- Test error handling for malformed regex/search specifications

### 11. Documentation

Update module documentation:

**File: file:src/parser/ast.rs**
- Add doc comments for `Regex` and `Search` variants with examples
- Document `RegexFlags` struct with usage examples

**File: file:src/evaluator/types.rs**
- Add doc comments for `read_regex` and `read_search` functions
- Include examples showing typical usage patterns

**File: file:README.md** (if applicable)
- Update feature list to mention regex and search type support
- Add examples of magic rules using regex/search

### 12. Error Handling

Ensure proper error handling throughout:

- Regex compilation errors should be caught and converted to `TypeReadError`
- Invalid regex patterns should not panic
- Search range overflow should be handled gracefully
- Buffer boundary checks for both regex and search operations

### Implementation Sequence Diagram

```mermaid
sequenceDiagram
    participant P as Parser
    participant A as AST
    participant E as Evaluator
    participant R as Regex Engine
    
    P->>P: parse_type_and_operator("regex/c")
    P->>P: parse_regex_flags("/c")
    P->>A: Create TypeKind::Regex{flags, range}
    
    E->>E: evaluate_single_rule(rule, buffer)
    E->>E: Check if Regex/Search type
    E->>E: match_regex(buffer, offset, pattern, flags)
    E->>R: RegexBuilder::new(pattern).case_insensitive(true)
    R->>R: Compile regex
    R->>R: find(buffer[offset..offset+range])
    R-->>E: Match result
    E-->>E: Return (match_offset, matched_bytes)
```

### Type Hierarchy

```mermaid
classDiagram
    class TypeKind {
        <<enumeration>>
        Byte
        Short
        Long
        String
        Regex
        Search
    }
    
    class RegexFlags {
        +bool case_insensitive
        +bool start_offset
        +bool line_based
    }
    
    class Regex {
        +RegexFlags flags
        +Option~usize~ range
    }
    
    class Search {
        +usize range
    }
    
    TypeKind --> Regex
    TypeKind --> Search
    Regex --> RegexFlags
```

---

Execution Information

Branch: main
Commit: 36adac5

New Plan

Observations

The codebase currently supports basic types (Byte, Short, Long, String) with a clean separation between AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and type evaluation (file:src/evaluator/types.rs). The regex crate is already available in dev-dependencies but needs to be promoted to main dependencies with the bytes feature for binary-safe matching. Test corpus files like regex-eol.magic, searchbug.magic, and JSON/shell script tests are present and waiting for regex/search support.

Approach

Extend the type system with Regex and Search variants following the existing pattern: add enum variants to TypeKind, implement parser combinators for regex/[flags] and search/number syntax, create evaluation functions using regex::bytes::Regex for binary-safe matching, and add comprehensive unit tests. The implementation prioritizes compatibility with GNU file's magic format, supporting common regex flags (case-insensitive, line-based limits) and range-limited byte sequence searching.

Implementation Steps

1. Update Dependencies

File: file:Cargo.toml

• Move regex from [dev-dependencies] to [dependencies] section
• Enable the bytes feature: regex = { version = "1.12.3", features = ["bytes"] }
• This enables binary-safe regex matching required for file content analysis

2. Extend TypeKind Enum

File: file:src/parser/ast.rs

Add two new variants to the TypeKind enum (around line 78):

/// Regular expression pattern matching (binary-safe)
Regex {
    /// Regex flags: 'c' (case-insensitive), 's' (start offset), 'l' (line-based limit)
    flags: RegexFlags,
    /// Search range in bytes (or lines if 'l' flag set)
    range: Option<usize>,
},
/// Byte sequence search within a range
Search {
    /// Number of positions to attempt match from start offset
    range: usize,
},

Define a new struct for regex flags:

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct RegexFlags {
    /// Case-insensitive matching
    pub case_insensitive: bool,
    /// Update offset to start of match (instead of end)
    pub start_offset: bool,
    /// Limit is in lines instead of bytes
    pub line_based: bool,
}

Add unit tests for the new variants following the existing test pattern (lines 515-587).

3. Implement Parser for Regex Type

File: file:src/parser/grammar.rs

Add a parser function for regex flags (insert around line 1400):

/// Parse regex flags: /[c][s][l][number]
fn parse_regex_flags(input: &str) -> IResult<&str, (RegexFlags, Option<usize>)>

This parser should:

• Match the / character followed by optional flag characters
• Parse c, s, l flags in any order
• Parse optional numeric range after flags (e.g., /1l means 1 line)
• Return RegexFlags struct and optional range value

Update parse_type_and_operator function (line 1429) to include:

• Add tag("regex") to the alt combinator list
• After matching "regex", call parse_regex_flags to extract flags
• Construct TypeKind::Regex { flags, range } variant

Add unit tests for regex type parsing covering:

• regex (no flags)
• regex/c (case-insensitive)
• regex/1l (1 line limit)
• regex/cs (multiple flags)
• Invalid flag combinations

4. Implement Parser for Search Type

File: file:src/parser/grammar.rs

Update parse_type_and_operator function (line 1429):

• Add tag("search") to the alt combinator list
• After matching "search", expect / followed by a number using parse_number
• Construct TypeKind::Search { range } variant
• The range is mandatory for search type (per GNU file specification)

Add unit tests for search type parsing:

• search/12 (search within 12 bytes)
• search/256 (larger range)
• Error cases: search without range, search/ without number

5. Implement Regex Evaluation

File: file:src/evaluator/types.rs

Add a new function read_regex (insert around line 300):

/// Reads and matches a regex pattern against buffer content
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The regex pattern (from Value::String or Value::Bytes)
/// * `flags` - Regex flags for matching behavior
/// * `range` - Optional search range limit
///
/// # Returns
/// Returns Value::Bytes containing the matched content, or error if no match
pub fn read_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Value, TypeReadError>

Implementation details:

• Use regex::bytes::RegexBuilder to construct regex with flags
• Set case-insensitive flag if flags.case_insensitive is true
• Determine search window: if flags.line_based, count lines; otherwise use byte range
• Default range to 8192 bytes if not specified (per GNU file behavior)
• Use regex.find() to locate match in the search window
• Return Value::Bytes with matched content
• Handle regex compilation errors gracefully

Add comprehensive unit tests:

• Basic regex matching: [0-9]+, [a-zA-Z]+
• Case-insensitive matching with /c flag
• Line-based limits with /l flag
• Binary-safe matching with non-printable characters
• No match scenarios
• Invalid regex patterns

6. Implement Search Evaluation

File: file:src/evaluator/types.rs

Add a new function read_search (insert around line 350):

/// Searches for a byte sequence within a specified range
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The byte sequence to find (from Value::Bytes or Value::String)
/// * `range` - Number of positions to attempt match
///
/// # Returns
/// Returns Value::Bytes containing the found sequence, or error if not found
pub fn read_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Value, TypeReadError>

Implementation details:

• Extract pattern bytes from Value::Bytes or Value::String
• Use memchr crate (already a dependency) for efficient byte searching
• Search within buffer[offset..offset+range] window
• Use sliding window approach to find exact byte sequence match
• Return Value::Bytes with the matched sequence
• Return error if pattern not found within range

Add comprehensive unit tests:

• Simple byte sequence search: ABC, \x7fELF
• Search at various offsets
• Pattern found at start/middle/end of range
• Pattern not found within range
• Edge cases: empty pattern, range too small, offset beyond buffer
• Boundary conditions: pattern at exact range end

7. Update Type Dispatcher

File: src/evaluator/types.rs

Update read_typed_value function (line 348) to handle new types:

match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::Regex { flags, range } => {
        // For regex, the pattern comes from the rule's value, not the buffer
        // This requires special handling in the evaluator module
        Err(TypeReadError::UnsupportedType("Regex matching requires pattern from rule value".to_string()))
    },
    TypeKind::Search { range } => {
        // Similar to regex, search needs the pattern from the rule
        Err(TypeReadError::UnsupportedType("Search requires pattern from rule value".to_string()))
    },
}

Note: Regex and search types differ from other types because they need both the buffer AND the expected pattern from the rule. This requires special handling in the evaluator module's evaluate_single_rule function.

8. Refactor Evaluator for Pattern-Based Types

File: src/evaluator/mod.rs

Update evaluate_single_rule function (around line 246) to handle regex/search specially:

• Before calling read_typed_value, check if rule.typ is Regex or Search
• For these types, call specialized matching functions that take both buffer and pattern:
  • match_regex(buffer, offset, &rule.value, flags, range)
  • match_search(buffer, offset, &rule.value, range)
• These functions should return Result<Option<(usize, Value)>, LibmagicError> where the usize is the match offset
• For regex with start_offset flag, return start of match; otherwise return end
• For search, return the offset where pattern was found
• Update the offset tracking to use the returned match offset for child rule evaluation

Add helper functions in src/evaluator/mod.rs:

fn match_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Option<(usize, Value)>, LibmagicError>

fn match_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Option<(usize, Value)>, LibmagicError>

These functions should:

• Call the corresponding read_regex or read_search from the types module
• Handle the returned match offset for subsequent child rule evaluation
• Return None if no match is found (allowing rule evaluation to continue)
• Return Some((match_offset, matched_value)) on successful match

9. Integration Tests

File: tests/evaluator_tests.rs or create new tests/regex_search_tests.rs

Add integration tests using the test corpus files:

• Test regex-eol.magic with regex-eol.testfile (Ansible Vault detection with version/encryption)
• Test searchbug.magic with searchbug.testfile (search type with byte sequence detection)
• Test JSON detection patterns (if available in test corpus)
• Test shell script detection (if available in test corpus)

Each test should:

• Load the magic file
• Parse the rules
• Evaluate against the test file
• Verify that regex/search rules match correctly
• Compare output with expected .result file if available

10. Parser Integration Tests

File: tests/parser_integration_tests.rs

Add tests for parsing magic files with regex/search types:

• Parse regex-eol.magic and verify AST structure (check for Regex variants with /1l flags)
• Parse searchbug.magic and verify AST structure (check for Search variants with ranges)
• Verify regex flags are correctly parsed (c, s, l combinations)
• Verify search ranges are correctly parsed
• Test error handling for malformed regex/search specifications

11. Documentation

Update module documentation:

File: src/parser/ast.rs

• Add doc comments for Regex and Search variants with examples
• Document RegexFlags struct with usage examples
• Include examples showing typical regex/search patterns from test corpus

File: src/evaluator/types.rs

• Add doc comments for read_regex and read_search functions
• Include examples showing typical usage patterns
• Document the binary-safe nature of regex matching

File: src/evaluator/mod.rs

• Add doc comments for match_regex and match_search helper functions
• Document how pattern-based types are handled differently from value-based types

12. Error Handling

Ensure proper error handling throughout:

• Regex compilation errors should be caught and converted to TypeReadError::UnsupportedType
• Invalid regex patterns should not panic
• Search range overflow should be handled gracefully with bounds checking
• Buffer boundary checks for both regex and search operations
• Proper error propagation in the evaluator module

Implementation Sequence

The implementation follows this sequence:

1. Dependencies - Move regex to main dependencies with bytes feature
2. AST - Add RegexFlags struct and Regex/Search variants to TypeKind
3. Parser - Implement parse_regex_flags and update parse_type_and_operator to handle both types
4. Type Evaluation - Implement read_regex and read_search functions in types module
5. Evaluator Integration - Update evaluate_single_rule to handle pattern-based types specially
6. Tests - Add unit tests for parsing and evaluation, plus integration tests with test corpus
7. Documentation - Update doc comments and examples

Key Design Decisions

• Pattern-based types are handled specially in the evaluator because they need both the buffer and the rule's expected value, unlike value-based types
• Default range for regex is 8192 bytes (per GNU file behavior) if not specified
• Search range is mandatory (per GNU file specification)
• Regex flags support c (case-insensitive), s (start-offset), l (line-based) in any order
• Binary-safe matching uses regex::bytes::Regex for both regex and search operations
• Error handling converts regex compilation errors to TypeReadError::UnsupportedType for consistency

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase currently supports basic types (`Byte`, `Short`, `Long`, `String`) with a clean separation between AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and type evaluation (file:src/evaluator/types.rs). The `regex` crate is already available in dev-dependencies but needs to be promoted to main dependencies with the `bytes` feature for binary-safe matching. Test corpus files like `regex-eol.magic`, `searchbug.magic`, and JSON/shell script tests are present and waiting for regex/search support.

## Approach

Extend the type system with `Regex` and `Search` variants following the existing pattern: add enum variants to `TypeKind`, implement parser combinators for `regex/[flags]` and `search/number` syntax, create evaluation functions using `regex::bytes::Regex` for binary-safe matching, and add comprehensive unit tests. The implementation prioritizes compatibility with GNU file's magic format, supporting common regex flags (case-insensitive, line-based limits) and range-limited byte sequence searching.

## Implementation Steps

### 1. Update Dependencies

**File: file:Cargo.toml**

- Move `regex` from `[dev-dependencies]` to `[dependencies]` section
- Enable the `bytes` feature: `regex = { version = "1.12.3", features = ["bytes"] }`
- This enables binary-safe regex matching required for file content analysis

### 2. Extend TypeKind Enum

**File: file:src/parser/ast.rs**

Add two new variants to the `TypeKind` enum (around line 78):

```rust
/// Regular expression pattern matching (binary-safe)
Regex {
    /// Regex flags: 'c' (case-insensitive), 's' (start offset), 'l' (line-based limit)
    flags: RegexFlags,
    /// Search range in bytes (or lines if 'l' flag set)
    range: Option<usize>,
},
/// Byte sequence search within a range
Search {
    /// Number of positions to attempt match from start offset
    range: usize,
},
```

Define a new struct for regex flags:

```rust
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct RegexFlags {
    /// Case-insensitive matching
    pub case_insensitive: bool,
    /// Update offset to start of match (instead of end)
    pub start_offset: bool,
    /// Limit is in lines instead of bytes
    pub line_based: bool,
}
```

Add unit tests for the new variants following the existing test pattern (lines 515-587).

### 3. Implement Parser for Regex Type

**File: file:src/parser/grammar.rs**

Add a parser function for regex flags (insert around line 1400):

```rust
/// Parse regex flags: /[c][s][l][number]
fn parse_regex_flags(input: &str) -> IResult<&str, (RegexFlags, Option<usize>)>
```

This parser should:
- Match the `/` character followed by optional flag characters
- Parse `c`, `s`, `l` flags in any order
- Parse optional numeric range after flags (e.g., `/1l` means 1 line)
- Return `RegexFlags` struct and optional range value

Update `parse_type_and_operator` function (line 1429) to include:
- Add `tag("regex")` to the `alt` combinator list
- After matching "regex", call `parse_regex_flags` to extract flags
- Construct `TypeKind::Regex { flags, range }` variant

Add unit tests for regex type parsing covering:
- `regex` (no flags)
- `regex/c` (case-insensitive)
- `regex/1l` (1 line limit)
- `regex/cs` (multiple flags)
- Invalid flag combinations

### 4. Implement Parser for Search Type

**File: file:src/parser/grammar.rs**

Update `parse_type_and_operator` function (line 1429):
- Add `tag("search")` to the `alt` combinator list
- After matching "search", expect `/` followed by a number using `parse_number`
- Construct `TypeKind::Search { range }` variant
- The range is mandatory for search type (per GNU file specification)

Add unit tests for search type parsing:
- `search/12` (search within 12 bytes)
- `search/256` (larger range)
- Error cases: `search` without range, `search/` without number

### 5. Implement Regex Evaluation

**File: file:src/evaluator/types.rs**

Add a new function `read_regex` (insert around line 300):

```rust
/// Reads and matches a regex pattern against buffer content
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The regex pattern (from Value::String or Value::Bytes)
/// * `flags` - Regex flags for matching behavior
/// * `range` - Optional search range limit
///
/// # Returns
/// Returns Value::Bytes containing the matched content, or error if no match
pub fn read_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Value, TypeReadError>
```

Implementation details:
- Use `regex::bytes::RegexBuilder` to construct regex with flags
- Set case-insensitive flag if `flags.case_insensitive` is true
- Determine search window: if `flags.line_based`, count lines; otherwise use byte range
- Default range to 8192 bytes if not specified (per GNU file behavior)
- Use `regex.find()` to locate match in the search window
- Return `Value::Bytes` with matched content
- Handle regex compilation errors gracefully

Add comprehensive unit tests:
- Basic regex matching: `[0-9]+`, `[a-zA-Z]+`
- Case-insensitive matching with `/c` flag
- Line-based limits with `/l` flag
- Binary-safe matching with non-printable characters
- No match scenarios
- Invalid regex patterns

### 6. Implement Search Evaluation

**File: file:src/evaluator/types.rs**

Add a new function `read_search` (insert around line 350):

```rust
/// Searches for a byte sequence within a specified range
///
/// # Arguments
/// * `buffer` - The byte buffer to search
/// * `offset` - Starting offset for search
/// * `pattern` - The byte sequence to find (from Value::Bytes or Value::String)
/// * `range` - Number of positions to attempt match
///
/// # Returns
/// Returns Value::Bytes containing the found sequence, or error if not found
pub fn read_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Value, TypeReadError>
```

Implementation details:
- Extract pattern bytes from `Value::Bytes` or `Value::String`
- Use `memchr` crate (already a dependency) for efficient byte searching
- Search within `buffer[offset..offset+range]` window
- Use sliding window approach to find exact byte sequence match
- Return `Value::Bytes` with the matched sequence
- Return error if pattern not found within range

Add comprehensive unit tests:
- Simple byte sequence search: `ABC`, `\x7fELF`
- Search at various offsets
- Pattern found at start/middle/end of range
- Pattern not found within range
- Edge cases: empty pattern, range too small, offset beyond buffer
- Boundary conditions: pattern at exact range end

### 7. Update Type Dispatcher

**File: `src/evaluator/types.rs`**

Update `read_typed_value` function (line 348) to handle new types:

```rust
match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::Regex { flags, range } => {
        // For regex, the pattern comes from the rule's value, not the buffer
        // This requires special handling in the evaluator module
        Err(TypeReadError::UnsupportedType("Regex matching requires pattern from rule value".to_string()))
    },
    TypeKind::Search { range } => {
        // Similar to regex, search needs the pattern from the rule
        Err(TypeReadError::UnsupportedType("Search requires pattern from rule value".to_string()))
    },
}
```

**Note:** Regex and search types differ from other types because they need both the buffer AND the expected pattern from the rule. This requires special handling in the evaluator module's `evaluate_single_rule` function.

### 8. Refactor Evaluator for Pattern-Based Types

**File: `src/evaluator/mod.rs`**

Update `evaluate_single_rule` function (around line 246) to handle regex/search specially:

- Before calling `read_typed_value`, check if `rule.typ` is `Regex` or `Search`
- For these types, call specialized matching functions that take both buffer and pattern:
  - `match_regex(buffer, offset, &rule.value, flags, range)`
  - `match_search(buffer, offset, &rule.value, range)`
- These functions should return `Result<Option<(usize, Value)>, LibmagicError>` where the usize is the match offset
- For regex with `start_offset` flag, return start of match; otherwise return end
- For search, return the offset where pattern was found
- Update the offset tracking to use the returned match offset for child rule evaluation

Add helper functions in `src/evaluator/mod.rs`:

```rust
fn match_regex(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    flags: RegexFlags,
    range: Option<usize>,
) -> Result<Option<(usize, Value)>, LibmagicError>

fn match_search(
    buffer: &[u8],
    offset: usize,
    pattern: &Value,
    range: usize,
) -> Result<Option<(usize, Value)>, LibmagicError>
```

These functions should:
- Call the corresponding `read_regex` or `read_search` from the types module
- Handle the returned match offset for subsequent child rule evaluation
- Return `None` if no match is found (allowing rule evaluation to continue)
- Return `Some((match_offset, matched_value))` on successful match

### 9. Integration Tests

**File: `tests/evaluator_tests.rs` or create new `tests/regex_search_tests.rs`**

Add integration tests using the test corpus files:

- Test `regex-eol.magic` with `regex-eol.testfile` (Ansible Vault detection with version/encryption)
- Test `searchbug.magic` with `searchbug.testfile` (search type with byte sequence detection)
- Test JSON detection patterns (if available in test corpus)
- Test shell script detection (if available in test corpus)

Each test should:
- Load the magic file
- Parse the rules
- Evaluate against the test file
- Verify that regex/search rules match correctly
- Compare output with expected `.result` file if available

### 10. Parser Integration Tests

**File: `tests/parser_integration_tests.rs`**

Add tests for parsing magic files with regex/search types:

- Parse `regex-eol.magic` and verify AST structure (check for `Regex` variants with `/1l` flags)
- Parse `searchbug.magic` and verify AST structure (check for `Search` variants with ranges)
- Verify regex flags are correctly parsed (`c`, `s`, `l` combinations)
- Verify search ranges are correctly parsed
- Test error handling for malformed regex/search specifications

### 11. Documentation

Update module documentation:

**File: `src/parser/ast.rs`**
- Add doc comments for `Regex` and `Search` variants with examples
- Document `RegexFlags` struct with usage examples
- Include examples showing typical regex/search patterns from test corpus

**File: `src/evaluator/types.rs`**
- Add doc comments for `read_regex` and `read_search` functions
- Include examples showing typical usage patterns
- Document the binary-safe nature of regex matching

**File: `src/evaluator/mod.rs`**
- Add doc comments for `match_regex` and `match_search` helper functions
- Document how pattern-based types are handled differently from value-based types

### 12. Error Handling

Ensure proper error handling throughout:

- Regex compilation errors should be caught and converted to `TypeReadError::UnsupportedType`
- Invalid regex patterns should not panic
- Search range overflow should be handled gracefully with bounds checking
- Buffer boundary checks for both regex and search operations
- Proper error propagation in the evaluator module

### Implementation Sequence

The implementation follows this sequence:

1. **Dependencies** - Move regex to main dependencies with bytes feature
2. **AST** - Add `RegexFlags` struct and `Regex`/`Search` variants to `TypeKind`
3. **Parser** - Implement `parse_regex_flags` and update `parse_type_and_operator` to handle both types
4. **Type Evaluation** - Implement `read_regex` and `read_search` functions in types module
5. **Evaluator Integration** - Update `evaluate_single_rule` to handle pattern-based types specially
6. **Tests** - Add unit tests for parsing and evaluation, plus integration tests with test corpus
7. **Documentation** - Update doc comments and examples

### Key Design Decisions

- **Pattern-based types** are handled specially in the evaluator because they need both the buffer and the rule's expected value, unlike value-based types
- **Default range** for regex is 8192 bytes (per GNU file behavior) if not specified
- **Search range** is mandatory (per GNU file specification)
- **Regex flags** support `c` (case-insensitive), `s` (start-offset), `l` (line-based) in any order
- **Binary-safe matching** uses `regex::bytes::Regex` for both regex and search operations
- **Error handling** converts regex compilation errors to `TypeReadError::UnsupportedType` for consistency

---

Execution Information

Branch: main
Commit: 36adac5