Evaluator: implement relative offset resolution

[unclesp1d3r]

Summary

Implement evaluation of relative offsets. The AST representation (OffsetSpec::Relative) already exists, but the evaluator returns a "not yet implemented" error at src/evaluator/offset.rs:167.

Context

Relative offsets use the syntax &+N or &-N to reference positions relative to the end of the previous match. This enables sequential field parsing where each field follows the previous one without hardcoding absolute positions.

Current state: AST variant defined, evaluation stubbed with TODO.

Acceptance Criteria

• resolve_offset() handles OffsetSpec::Relative using the previous match's end position
• Evaluation context tracks the current match position for relative offset resolution
• Positive and negative relative offsets work correctly
• Bounds checking prevents out-of-buffer access
• Unit tests for relative offset resolution
• Integration tests with nested rules using relative offsets

Impact

MEDIUM-HIGH - Used in nested rule structures where child rules reference positions relative to parent matches.

Files to Modify

• src/evaluator/offset.rs - Implement relative resolution (line ~167)
• src/evaluator/mod.rs - Track match position in evaluation context

References

• GNU file magic(5) man page
• Epic Epic: libmagic-rs Phase 1 MVP #24

[traycerai]

Plan

Observations

The codebase has a well-structured evaluator module with clear separation of concerns. The OffsetSpec::Relative variant is already defined in the AST (file:src/parser/ast.rs line 63), but the evaluation logic returns a "not yet implemented" error at file:src/evaluator/offset.rs line 166-172. The EvaluationContext struct exists with a current_offset field, but it doesn't track the end position of matches needed for relative offset resolution. The evaluation flow goes through evaluate_rules → evaluate_single_rule → resolve_offset, where context is available in the outer function but not passed down to the offset resolution layer.

Approach

The implementation will extend the existing evaluation infrastructure to support relative offsets by tracking match end positions in the EvaluationContext and using this information during offset resolution. The approach maintains backward compatibility by making context optional in the offset resolution function, allowing gradual migration. The solution follows the existing patterns for offset resolution (similar to how absolute and from-end offsets are handled) and integrates seamlessly with the hierarchical rule evaluation system. Relative offsets will be calculated from the end of the previous match at the same or higher level in the rule hierarchy, enabling sequential field parsing as described in the GNU magic(5) specification.

Implementation Steps

1. Extend EvaluationContext to Track Match End Position

File: file:src/evaluator/mod.rs

Add a new field to the EvaluationContext struct to track the end position of the last successful match:

• Add last_match_end: Option<usize> field to the struct (around line 40-45)
• Initialize it to None in the new() constructor (line 66)
• Add getter method last_match_end(&self) -> Option<usize>
• Add setter method set_last_match_end(&mut self, end_pos: usize)
• Update the reset() method to reset last_match_end to None (line 195)

2. Create Helper Function to Calculate Value Size

File: file:src/evaluator/types.rs

Add a public function to calculate the size of a read value based on TypeKind and the actual Value:

• Create pub fn calculate_value_size(type_kind: &TypeKind, value: &Value) -> usize
• For TypeKind::Byte: return 1
• For TypeKind::Short: return 2
• For TypeKind::Long: return 4
• For TypeKind::String: extract string length from Value::String and return it (the actual bytes read, not including null terminator)
• Add unit tests for this function covering all type variants

3. Implement Relative Offset Resolution

File: file:src/evaluator/offset.rs

Modify the resolve_offset function to accept an optional context parameter and implement relative offset logic:

• Change signature from pub fn resolve_offset(spec: &OffsetSpec, buffer: &[u8]) to pub fn resolve_offset(spec: &OffsetSpec, buffer: &[u8], context: Option<&EvaluationContext>)
• In the OffsetSpec::Relative(offset) match arm (line 166-172):
  • Extract last_match_end from context, return error if context is None or last_match_end is None
  • Calculate absolute position: base_position = last_match_end
  • Apply the relative offset: absolute_offset = base_position + offset (handle both positive and negative offsets)
  • Perform bounds checking using existing resolve_absolute_offset logic or similar
  • Return the resolved absolute offset as usize
• Handle arithmetic overflow cases (when adding offset to base_position)
• Ensure proper error messages for missing context or invalid relative offsets

4. Update evaluate_single_rule Signature and Logic

File: file:src/evaluator/mod.rs

Modify evaluate_single_rule to accept context and update match end position:

• Change signature from pub fn evaluate_single_rule(rule: &MagicRule, buffer: &[u8]) to pub fn evaluate_single_rule(rule: &MagicRule, buffer: &[u8], context: Option<&mut EvaluationContext>) (line 296)
• Pass context to offset::resolve_offset call (line 301)
• After successful match (when returning Some((absolute_offset, read_value))):
  • Calculate value size using the helper function from step 2
  • Calculate end position: end_pos = absolute_offset + value_size
  • If context is Some, update last_match_end with the calculated end position
• Update function documentation to reflect the new parameter

5. Update All Callers of evaluate_single_rule

File: file:src/evaluator/mod.rs

Update the evaluate_rules function to pass context to evaluate_single_rule:

• Change the call at line 409 from evaluate_single_rule(rule, buffer) to evaluate_single_rule(rule, buffer, Some(context))
• The context is already mutable in evaluate_rules, so this should work seamlessly

File: file:src/evaluator/mod.rs (tests section)

Update all test calls to evaluate_single_rule:

• For tests that don't need context, pass None
• For tests that will test relative offsets, create a context and pass Some(&mut context)

6. Add Unit Tests for Relative Offset Resolution

File: file:src/evaluator/offset.rs

Add comprehensive unit tests in the existing test module:

• test_resolve_offset_relative_positive: Test &+N with positive offset
• test_resolve_offset_relative_negative: Test &-N with negative offset
• test_resolve_offset_relative_zero: Test &+0 (same as end of previous match)
• test_resolve_offset_relative_no_context: Verify error when context is None
• test_resolve_offset_relative_no_previous_match: Verify error when last_match_end is None
• test_resolve_offset_relative_out_of_bounds: Test bounds checking for relative offsets
• test_resolve_offset_relative_arithmetic_overflow: Test overflow handling

7. Add Integration Tests for Nested Rules with Relative Offsets

File: file:tests/evaluator_tests.rs

Add integration tests demonstrating relative offsets in hierarchical rule evaluation:

• test_relative_offset_sequential_fields: Create a parent rule and child rules using relative offsets to parse sequential fields (e.g., header followed by length field followed by data)
• test_relative_offset_nested_hierarchy: Test relative offsets across multiple nesting levels
• test_relative_offset_with_different_types: Test relative offsets with different type sizes (byte, short, long, string)
• Create test data buffers that match the expected structure

8. Update Documentation

File: file:src/evaluator/offset.rs

Update module and function documentation:

• Update the module-level documentation to mention relative offset support
• Update resolve_offset function documentation to describe the context parameter and relative offset behavior
• Add examples showing relative offset usage in the documentation

File: file:src/evaluator/mod.rs

Update EvaluationContext documentation:

• Document the new last_match_end field and its purpose
• Add examples showing how relative offsets work with the context

Sequence Diagram

sequenceDiagram
    participant Caller
    participant evaluate_rules
    participant evaluate_single_rule
    participant resolve_offset
    participant EvaluationContext

    Caller->>evaluate_rules: rules, buffer, context
    evaluate_rules->>evaluate_single_rule: rule, buffer, Some(context)
    evaluate_single_rule->>resolve_offset: OffsetSpec::Relative(N), buffer, Some(context)
    resolve_offset->>EvaluationContext: get last_match_end()
    EvaluationContext-->>resolve_offset: Some(end_position)
    resolve_offset->>resolve_offset: calculate absolute_offset = end_position + N
    resolve_offset->>resolve_offset: bounds check
    resolve_offset-->>evaluate_single_rule: Ok(absolute_offset)
    evaluate_single_rule->>evaluate_single_rule: read value at offset
    evaluate_single_rule->>evaluate_single_rule: calculate end_pos = offset + value_size
    evaluate_single_rule->>EvaluationContext: set_last_match_end(end_pos)
    evaluate_single_rule-->>evaluate_rules: Ok(Some((offset, value)))
    evaluate_rules-->>Caller: Ok(matches)

Loading

Key Design Decisions

Decision	Rationale
Make context optional in resolve_offset	Maintains backward compatibility and allows gradual migration of existing code
Track last_match_end instead of last_match_start	Relative offsets are defined as positions relative to the END of the previous match per GNU magic(5) spec
Calculate value size from TypeKind and Value	Necessary because different types have different sizes, and strings have variable length
Update context after successful match only	Failed matches should not affect the relative offset base position
Use existing bounds checking infrastructure	Leverages proven code for safety and consistency

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured evaluator module with clear separation of concerns. The `OffsetSpec::Relative` variant is already defined in the AST (`file:src/parser/ast.rs` line 63), but the evaluation logic returns a "not yet implemented" error at `file:src/evaluator/offset.rs` line 166-172. The `EvaluationContext` struct exists with a `current_offset` field, but it doesn't track the end position of matches needed for relative offset resolution. The evaluation flow goes through `evaluate_rules` → `evaluate_single_rule` → `resolve_offset`, where context is available in the outer function but not passed down to the offset resolution layer.

## Approach

The implementation will extend the existing evaluation infrastructure to support relative offsets by tracking match end positions in the `EvaluationContext` and using this information during offset resolution. The approach maintains backward compatibility by making context optional in the offset resolution function, allowing gradual migration. The solution follows the existing patterns for offset resolution (similar to how absolute and from-end offsets are handled) and integrates seamlessly with the hierarchical rule evaluation system. Relative offsets will be calculated from the end of the previous match at the same or higher level in the rule hierarchy, enabling sequential field parsing as described in the GNU magic(5) specification.

## Implementation Steps

### 1. Extend EvaluationContext to Track Match End Position

**File:** `file:src/evaluator/mod.rs`

Add a new field to the `EvaluationContext` struct to track the end position of the last successful match:

- Add `last_match_end: Option<usize>` field to the struct (around line 40-45)
- Initialize it to `None` in the `new()` constructor (line 66)
- Add getter method `last_match_end(&self) -> Option<usize>`
- Add setter method `set_last_match_end(&mut self, end_pos: usize)`
- Update the `reset()` method to reset `last_match_end` to `None` (line 195)

### 2. Create Helper Function to Calculate Value Size

**File:** `file:src/evaluator/types.rs`

Add a public function to calculate the size of a read value based on `TypeKind` and the actual `Value`:

- Create `pub fn calculate_value_size(type_kind: &TypeKind, value: &Value) -> usize`
- For `TypeKind::Byte`: return 1
- For `TypeKind::Short`: return 2
- For `TypeKind::Long`: return 4
- For `TypeKind::String`: extract string length from `Value::String` and return it (the actual bytes read, not including null terminator)
- Add unit tests for this function covering all type variants

### 3. Implement Relative Offset Resolution

**File:** `file:src/evaluator/offset.rs`

Modify the `resolve_offset` function to accept an optional context parameter and implement relative offset logic:

- Change signature from `pub fn resolve_offset(spec: &OffsetSpec, buffer: &[u8])` to `pub fn resolve_offset(spec: &OffsetSpec, buffer: &[u8], context: Option<&EvaluationContext>)`
- In the `OffsetSpec::Relative(offset)` match arm (line 166-172):
  - Extract `last_match_end` from context, return error if context is `None` or `last_match_end` is `None`
  - Calculate absolute position: `base_position = last_match_end`
  - Apply the relative offset: `absolute_offset = base_position + offset` (handle both positive and negative offsets)
  - Perform bounds checking using existing `resolve_absolute_offset` logic or similar
  - Return the resolved absolute offset as `usize`
- Handle arithmetic overflow cases (when adding offset to base_position)
- Ensure proper error messages for missing context or invalid relative offsets

### 4. Update evaluate_single_rule Signature and Logic

**File:** `file:src/evaluator/mod.rs`

Modify `evaluate_single_rule` to accept context and update match end position:

- Change signature from `pub fn evaluate_single_rule(rule: &MagicRule, buffer: &[u8])` to `pub fn evaluate_single_rule(rule: &MagicRule, buffer: &[u8], context: Option<&mut EvaluationContext>)` (line 296)
- Pass context to `offset::resolve_offset` call (line 301)
- After successful match (when returning `Some((absolute_offset, read_value))`):
  - Calculate value size using the helper function from step 2
  - Calculate end position: `end_pos = absolute_offset + value_size`
  - If context is `Some`, update `last_match_end` with the calculated end position
- Update function documentation to reflect the new parameter

### 5. Update All Callers of evaluate_single_rule

**File:** `file:src/evaluator/mod.rs`

Update the `evaluate_rules` function to pass context to `evaluate_single_rule`:

- Change the call at line 409 from `evaluate_single_rule(rule, buffer)` to `evaluate_single_rule(rule, buffer, Some(context))`
- The context is already mutable in `evaluate_rules`, so this should work seamlessly

**File:** `file:src/evaluator/mod.rs` (tests section)

Update all test calls to `evaluate_single_rule`:

- For tests that don't need context, pass `None`
- For tests that will test relative offsets, create a context and pass `Some(&mut context)`

### 6. Add Unit Tests for Relative Offset Resolution

**File:** `file:src/evaluator/offset.rs`

Add comprehensive unit tests in the existing test module:

- `test_resolve_offset_relative_positive`: Test `&+N` with positive offset
- `test_resolve_offset_relative_negative`: Test `&-N` with negative offset
- `test_resolve_offset_relative_zero`: Test `&+0` (same as end of previous match)
- `test_resolve_offset_relative_no_context`: Verify error when context is `None`
- `test_resolve_offset_relative_no_previous_match`: Verify error when `last_match_end` is `None`
- `test_resolve_offset_relative_out_of_bounds`: Test bounds checking for relative offsets
- `test_resolve_offset_relative_arithmetic_overflow`: Test overflow handling

### 7. Add Integration Tests for Nested Rules with Relative Offsets

**File:** `file:tests/evaluator_tests.rs`

Add integration tests demonstrating relative offsets in hierarchical rule evaluation:

- `test_relative_offset_sequential_fields`: Create a parent rule and child rules using relative offsets to parse sequential fields (e.g., header followed by length field followed by data)
- `test_relative_offset_nested_hierarchy`: Test relative offsets across multiple nesting levels
- `test_relative_offset_with_different_types`: Test relative offsets with different type sizes (byte, short, long, string)
- Create test data buffers that match the expected structure

### 8. Update Documentation

**File:** `file:src/evaluator/offset.rs`

Update module and function documentation:

- Update the module-level documentation to mention relative offset support
- Update `resolve_offset` function documentation to describe the context parameter and relative offset behavior
- Add examples showing relative offset usage in the documentation

**File:** `file:src/evaluator/mod.rs`

Update `EvaluationContext` documentation:

- Document the new `last_match_end` field and its purpose
- Add examples showing how relative offsets work with the context

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Caller
    participant evaluate_rules
    participant evaluate_single_rule
    participant resolve_offset
    participant EvaluationContext

    Caller->>evaluate_rules: rules, buffer, context
    evaluate_rules->>evaluate_single_rule: rule, buffer, Some(context)
    evaluate_single_rule->>resolve_offset: OffsetSpec::Relative(N), buffer, Some(context)
    resolve_offset->>EvaluationContext: get last_match_end()
    EvaluationContext-->>resolve_offset: Some(end_position)
    resolve_offset->>resolve_offset: calculate absolute_offset = end_position + N
    resolve_offset->>resolve_offset: bounds check
    resolve_offset-->>evaluate_single_rule: Ok(absolute_offset)
    evaluate_single_rule->>evaluate_single_rule: read value at offset
    evaluate_single_rule->>evaluate_single_rule: calculate end_pos = offset + value_size
    evaluate_single_rule->>EvaluationContext: set_last_match_end(end_pos)
    evaluate_single_rule-->>evaluate_rules: Ok(Some((offset, value)))
    evaluate_rules-->>Caller: Ok(matches)
```

## Key Design Decisions

| Decision | Rationale |
|----------|-----------|
| Make context optional in `resolve_offset` | Maintains backward compatibility and allows gradual migration of existing code |
| Track `last_match_end` instead of `last_match_start` | Relative offsets are defined as positions relative to the END of the previous match per GNU magic(5) spec |
| Calculate value size from `TypeKind` and `Value` | Necessary because different types have different sizes, and strings have variable length |
| Update context after successful match only | Failed matches should not affect the relative offset base position |
| Use existing bounds checking infrastructure | Leverages proven code for safety and consistency |

---

Execution Information

Branch: main
Commit: 36adac5

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.