Parser: implement float and double types with endian variants

[unclesp1d3r]

Summary

Implement floating-point type specifiers: float, double, befloat, bedouble, lefloat, ledouble.

Context

Some binary formats store floating-point values (scientific data, image metadata, audio parameters). While less common than integer types, float/double support is needed for full magic file compatibility.

Current state: No floating-point types exist in TypeKind or the parser.

Acceptance Criteria

• TypeKind extended with Float and Double variants
• Parser recognizes float, double, befloat, bedouble, lefloat, ledouble
• Evaluator reads 4/8 bytes and interprets as IEEE 754 float/double
• Endianness handling for all variants
• Comparison operators work with floating-point values (with appropriate epsilon for equality)
• Unit tests for each type and endian variant

Impact

MEDIUM - Less commonly used than integer types but needed for full compatibility.

Files to Modify

• src/parser/ast.rs - Extend TypeKind enum
• src/parser/grammar.rs - Parse float/double specifiers
• src/evaluator/types.rs - Implement read_float() and read_double()

References

• GNU file magic(5) man page
• Epic Epic: libmagic-rs Phase 1 MVP #24

[traycerai]

Plan

Observations

The codebase follows a clean three-layer architecture: AST definitions (file:src/parser/ast.rs), parser grammar (file:src/parser/grammar.rs), and evaluator (file:src/evaluator/types.rs). Integer types (Byte, Short, Long) already exist with endianness support using the byteorder crate. The TypeKind enum uses struct variants with endian and signed fields for multi-byte types. The Value enum currently supports Uint, Int, Bytes, and String. The parser uses nom combinators with an alt() pattern for type matching, and the evaluator dispatches via pattern matching on TypeKind.

Approach

Implement floating-point types following the established pattern for integer types. Add Float and Double variants to both TypeKind and Value enums. Extend the parser to recognize six new type specifiers (float, double, befloat, bedouble, lefloat, ledouble) using the existing endianness prefix convention. Implement IEEE 754 reading functions using byteorder's read_f32() and read_f64() methods. Handle floating-point comparisons with epsilon-based equality to account for precision limitations. Create comprehensive unit tests mirroring the existing test patterns for integer types.

Implementation Steps

1. Extend AST Type Definitions

File: file:src/parser/ast.rs

Add two new variants to the TypeKind enum (around line 79):

• Add Float { endian: Endianness } variant after the Long variant
• Add Double { endian: Endianness } variant after the Float variant
• Follow the same pattern as Short and Long, but without the signed field since IEEE 754 floats have sign bits built-in

Add two new variants to the Value enum (around line 117):

• Add Float(f32) variant after the Int variant
• Add Double(f64) variant after the Float variant
• These will store the actual floating-point values read from buffers

Update the existing unit tests in the same file:

• Add test cases for TypeKind::Float and TypeKind::Double with all three endianness variants (Little, Big, Native)
• Add test cases for Value::Float and Value::Double including edge cases (zero, negative, infinity, NaN)
• Add serialization tests for the new variants

2. Extend Parser Grammar

File: file:src/parser/grammar.rs

Update the parse_type_and_operator function (around line 1429):

• Add six new type specifiers to the alt() combinator in this order: "ledouble", "bedouble", "double", "lefloat", "befloat", "float"
• Order matters: longer prefixed variants must come before shorter ones to avoid partial matches
• Add these before the existing "lelong" entry to maintain consistent ordering

Update the type mapping logic (around line 1458):

• Add case for "float" → TypeKind::Float { endian: Endianness::Native }
• Add case for "lefloat" → TypeKind::Float { endian: Endianness::Little }
• Add case for "befloat" → TypeKind::Float { endian: Endianness::Big }
• Add case for "double" → TypeKind::Double { endian: Endianness::Native }
• Add case for "ledouble" → TypeKind::Double { endian: Endianness::Little }
• Add case for "bedouble" → TypeKind::Double { endian: Endianness::Big }

Update the documentation comment (around line 1402):

• Add the six new type specifiers to the list of supported formats
• Include examples showing usage of float and double types

Add comprehensive unit tests (in the tests section around line 1806):

• Test parsing each of the six type specifiers
• Test that parsed types have correct endianness
• Test that the existing test expecting "float" to fail (line 1917) is updated to expect success
• Test type specifiers with attached operators (e.g., "float&0xFF")
• Test complete magic rules using float/double types

3. Implement Evaluator Type Reading

File: file:src/evaluator/types.rs

Add a new read_float function (after read_long, around line 207):

• Signature: pub fn read_float(buffer: &[u8], offset: usize, endian: Endianness) -> Result<Value, TypeReadError>
• Check buffer bounds: require at least 4 bytes from offset
• Use byteorder::ByteOrder::read_f32() for Little/Big endian
• Use f32::from_ne_bytes() for Native endian
• Return Value::Float(value) on success
• Return TypeReadError::BufferOverrun if insufficient bytes
• Include comprehensive documentation with examples

Add a new read_double function (after read_float):

• Signature: pub fn read_double(buffer: &[u8], offset: usize, endian: Endianness) -> Result<Value, TypeReadError>
• Check buffer bounds: require at least 8 bytes from offset
• Use byteorder::ByteOrder::read_f64() for Little/Big endian
• Use f64::from_ne_bytes() for Native endian
• Return Value::Double(value) on success
• Return TypeReadError::BufferOverrun if insufficient bytes
• Include comprehensive documentation with examples

Update the read_typed_value function (around line 354):

• Add match arm for TypeKind::Float { endian } → call read_float(buffer, offset, *endian)
• Add match arm for TypeKind::Double { endian } → call read_double(buffer, offset, *endian)

Add comprehensive unit tests (in the tests section around line 360):

• Test read_float with all three endianness variants
• Test read_double with all three endianness variants
• Test reading positive, negative, zero, infinity, and NaN values
• Test buffer overrun scenarios (offset too large, insufficient bytes)
• Test reading from different buffer positions
• Test edge cases: smallest/largest representable values, denormalized numbers
• Test read_typed_value integration with Float and Double types

4. Extend Operator Comparison Logic

File: file:src/evaluator/operators.rs

Update the apply_equal function (around line 9):

• Add match arms for (Value::Float(a), Value::Float(b)) → use epsilon comparison
• Add match arms for (Value::Double(a), Value::Double(b)) → use epsilon comparison
• For epsilon comparison, use: (a - b).abs() < f32::EPSILON or (a - b).abs() < f64::EPSILON
• Add cross-type comparisons: (Value::Float(a), Value::Double(b)) and vice versa
• Add integer-to-float coercion: (Value::Uint(a), Value::Float(b)), (Value::Int(a), Value::Float(b)), etc.
• Handle special cases: NaN never equals anything (including itself), infinities equal only matching infinities

Update the apply_not_equal function (around line 68):

• Leverage the updated apply_equal function by negating its result
• No additional changes needed if apply_equal handles all float cases

Update the apply_bitwise_and function (around line 109):

• Add explicit handling to return false for float/double types
• Bitwise operations are not meaningful for IEEE 754 floating-point values
• Document this limitation in the function's documentation

Add comprehensive unit tests (in the tests section around line 238):

• Test equality comparison between identical float values
• Test equality comparison with values within epsilon
• Test inequality for values outside epsilon
• Test NaN comparison behavior (NaN != NaN)
• Test infinity comparisons (+inf, -inf)
• Test cross-type comparisons (float vs double, float vs int)
• Test that bitwise AND returns false for float/double types
• Test edge cases: zero vs negative zero, denormalized numbers

5. Add Integration Tests

File: file:tests/parser_integration_tests.rs

Add test cases for parsing magic files with float/double rules:

• Create test magic file with float type: "0 float 3.14159 Pi constant"
• Create test magic file with double type: "0 double 2.718281828 Euler's number"
• Create test magic file with endian variants: "0 lefloat 1.0 Little-endian float"
• Test that rules parse correctly and maintain proper structure

File: file:tests/evaluator_tests.rs

Add test cases for evaluating buffers with float/double values:

• Create test buffer with IEEE 754 float bytes
• Create test buffer with IEEE 754 double bytes
• Test detection with different endianness
• Verify confidence scores are calculated correctly

6. Update Documentation

File: file:src/parser/ast.rs

• Add documentation examples for TypeKind::Float and TypeKind::Double
• Document the endianness field and its impact on byte interpretation
• Add examples showing how to create these variants

File: file:src/evaluator/types.rs

• Document IEEE 754 compliance in module-level documentation
• Add examples showing float/double reading with different endianness
• Document precision limitations and epsilon-based comparison rationale

File: file:src/evaluator/operators.rs

• Document floating-point comparison behavior
• Explain epsilon-based equality and why it's necessary
• Document NaN and infinity handling
• Note that bitwise operations are not supported for float/double types

Testing Strategy

Create unit tests following the existing patterns:

1. AST Tests - Verify enum variants serialize/deserialize correctly
2. Parser Tests - Verify all six type specifiers parse correctly with proper endianness
3. Evaluator Tests - Verify float/double reading with various byte patterns and endianness
4. Operator Tests - Verify epsilon-based comparison, NaN handling, infinity handling
5. Integration Tests - Verify end-to-end parsing and evaluation of magic files with float/double rules

Test coverage should include:

• All three endianness variants (Little, Big, Native)
• Positive and negative values
• Zero and negative zero
• Infinity and negative infinity
• NaN (Not a Number)
• Denormalized numbers
• Smallest and largest representable values
• Buffer overrun scenarios
• Cross-type comparisons

Dependencies

No new dependencies required. The existing byteorder crate (version 1.5.0) already provides the necessary read_f32() and read_f64() methods for IEEE 754 floating-point reading with endianness support.

Architecture Diagram

sequenceDiagram
    participant Parser as Parser<br/>(grammar.rs)
    participant AST as AST<br/>(ast.rs)
    participant Evaluator as Evaluator<br/>(types.rs)
    participant Operators as Operators<br/>(operators.rs)
    
    Parser->>Parser: parse_type_and_operator()
    Note over Parser: Recognizes float/double<br/>with endian prefixes
    Parser->>AST: Create TypeKind::Float/Double
    Note over AST: Stores endianness variant
    
    Evaluator->>Evaluator: read_typed_value()
    Evaluator->>Evaluator: read_float() or read_double()
    Note over Evaluator: Uses byteorder crate<br/>for IEEE 754 reading
    Evaluator->>AST: Create Value::Float/Double
    
    Operators->>Operators: apply_equal()
    Note over Operators: Epsilon-based comparison<br/>for floating-point values
    Operators-->>Evaluator: Comparison result

Loading

Type Hierarchy

Type Specifier	TypeKind Variant	Bytes	Endianness	Value Variant
float	Float { endian: Native }	4	Native	Float(f32)
lefloat	Float { endian: Little }	4	Little	Float(f32)
befloat	Float { endian: Big }	4	Big	Float(f32)
double	Double { endian: Native }	8	Native	Double(f64)
ledouble	Double { endian: Little }	8	Little	Double(f64)
bedouble	Double { endian: Big }	8	Big	Double(f64)

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase follows a clean three-layer architecture: AST definitions (`file:src/parser/ast.rs`), parser grammar (`file:src/parser/grammar.rs`), and evaluator (`file:src/evaluator/types.rs`). Integer types (Byte, Short, Long) already exist with endianness support using the `byteorder` crate. The `TypeKind` enum uses struct variants with `endian` and `signed` fields for multi-byte types. The `Value` enum currently supports Uint, Int, Bytes, and String. The parser uses `nom` combinators with an `alt()` pattern for type matching, and the evaluator dispatches via pattern matching on `TypeKind`.

## Approach

Implement floating-point types following the established pattern for integer types. Add `Float` and `Double` variants to both `TypeKind` and `Value` enums. Extend the parser to recognize six new type specifiers (`float`, `double`, `befloat`, `bedouble`, `lefloat`, `ledouble`) using the existing endianness prefix convention. Implement IEEE 754 reading functions using `byteorder`'s `read_f32()` and `read_f64()` methods. Handle floating-point comparisons with epsilon-based equality to account for precision limitations. Create comprehensive unit tests mirroring the existing test patterns for integer types.

## Implementation Steps

### 1. Extend AST Type Definitions

**File:** `file:src/parser/ast.rs`

Add two new variants to the `TypeKind` enum (around line 79):

- Add `Float { endian: Endianness }` variant after the `Long` variant
- Add `Double { endian: Endianness }` variant after the `Float` variant
- Follow the same pattern as `Short` and `Long`, but without the `signed` field since IEEE 754 floats have sign bits built-in

Add two new variants to the `Value` enum (around line 117):

- Add `Float(f32)` variant after the `Int` variant
- Add `Double(f64)` variant after the `Float` variant
- These will store the actual floating-point values read from buffers

Update the existing unit tests in the same file:

- Add test cases for `TypeKind::Float` and `TypeKind::Double` with all three endianness variants (Little, Big, Native)
- Add test cases for `Value::Float` and `Value::Double` including edge cases (zero, negative, infinity, NaN)
- Add serialization tests for the new variants

### 2. Extend Parser Grammar

**File:** `file:src/parser/grammar.rs`

Update the `parse_type_and_operator` function (around line 1429):

- Add six new type specifiers to the `alt()` combinator in this order: `"ledouble"`, `"bedouble"`, `"double"`, `"lefloat"`, `"befloat"`, `"float"`
- Order matters: longer prefixed variants must come before shorter ones to avoid partial matches
- Add these before the existing `"lelong"` entry to maintain consistent ordering

Update the type mapping logic (around line 1458):

- Add case for `"float"` → `TypeKind::Float { endian: Endianness::Native }`
- Add case for `"lefloat"` → `TypeKind::Float { endian: Endianness::Little }`
- Add case for `"befloat"` → `TypeKind::Float { endian: Endianness::Big }`
- Add case for `"double"` → `TypeKind::Double { endian: Endianness::Native }`
- Add case for `"ledouble"` → `TypeKind::Double { endian: Endianness::Little }`
- Add case for `"bedouble"` → `TypeKind::Double { endian: Endianness::Big }`

Update the documentation comment (around line 1402):

- Add the six new type specifiers to the list of supported formats
- Include examples showing usage of float and double types

Add comprehensive unit tests (in the tests section around line 1806):

- Test parsing each of the six type specifiers
- Test that parsed types have correct endianness
- Test that the existing test expecting `"float"` to fail (line 1917) is updated to expect success
- Test type specifiers with attached operators (e.g., `"float&0xFF"`)
- Test complete magic rules using float/double types

### 3. Implement Evaluator Type Reading

**File:** `file:src/evaluator/types.rs`

Add a new `read_float` function (after `read_long`, around line 207):

- Signature: `pub fn read_float(buffer: &[u8], offset: usize, endian: Endianness) -> Result<Value, TypeReadError>`
- Check buffer bounds: require at least 4 bytes from offset
- Use `byteorder::ByteOrder::read_f32()` for Little/Big endian
- Use `f32::from_ne_bytes()` for Native endian
- Return `Value::Float(value)` on success
- Return `TypeReadError::BufferOverrun` if insufficient bytes
- Include comprehensive documentation with examples

Add a new `read_double` function (after `read_float`):

- Signature: `pub fn read_double(buffer: &[u8], offset: usize, endian: Endianness) -> Result<Value, TypeReadError>`
- Check buffer bounds: require at least 8 bytes from offset
- Use `byteorder::ByteOrder::read_f64()` for Little/Big endian
- Use `f64::from_ne_bytes()` for Native endian
- Return `Value::Double(value)` on success
- Return `TypeReadError::BufferOverrun` if insufficient bytes
- Include comprehensive documentation with examples

Update the `read_typed_value` function (around line 354):

- Add match arm for `TypeKind::Float { endian }` → call `read_float(buffer, offset, *endian)`
- Add match arm for `TypeKind::Double { endian }` → call `read_double(buffer, offset, *endian)`

Add comprehensive unit tests (in the tests section around line 360):

- Test `read_float` with all three endianness variants
- Test `read_double` with all three endianness variants
- Test reading positive, negative, zero, infinity, and NaN values
- Test buffer overrun scenarios (offset too large, insufficient bytes)
- Test reading from different buffer positions
- Test edge cases: smallest/largest representable values, denormalized numbers
- Test `read_typed_value` integration with Float and Double types

### 4. Extend Operator Comparison Logic

**File:** `file:src/evaluator/operators.rs`

Update the `apply_equal` function (around line 9):

- Add match arms for `(Value::Float(a), Value::Float(b))` → use epsilon comparison
- Add match arms for `(Value::Double(a), Value::Double(b))` → use epsilon comparison
- For epsilon comparison, use: `(a - b).abs() < f32::EPSILON` or `(a - b).abs() < f64::EPSILON`
- Add cross-type comparisons: `(Value::Float(a), Value::Double(b))` and vice versa
- Add integer-to-float coercion: `(Value::Uint(a), Value::Float(b))`, `(Value::Int(a), Value::Float(b))`, etc.
- Handle special cases: NaN never equals anything (including itself), infinities equal only matching infinities

Update the `apply_not_equal` function (around line 68):

- Leverage the updated `apply_equal` function by negating its result
- No additional changes needed if `apply_equal` handles all float cases

Update the `apply_bitwise_and` function (around line 109):

- Add explicit handling to return `false` for float/double types
- Bitwise operations are not meaningful for IEEE 754 floating-point values
- Document this limitation in the function's documentation

Add comprehensive unit tests (in the tests section around line 238):

- Test equality comparison between identical float values
- Test equality comparison with values within epsilon
- Test inequality for values outside epsilon
- Test NaN comparison behavior (NaN != NaN)
- Test infinity comparisons (+inf, -inf)
- Test cross-type comparisons (float vs double, float vs int)
- Test that bitwise AND returns false for float/double types
- Test edge cases: zero vs negative zero, denormalized numbers

### 5. Add Integration Tests

**File:** `file:tests/parser_integration_tests.rs`

Add test cases for parsing magic files with float/double rules:

- Create test magic file with float type: `"0 float 3.14159 Pi constant"`
- Create test magic file with double type: `"0 double 2.718281828 Euler's number"`
- Create test magic file with endian variants: `"0 lefloat 1.0 Little-endian float"`
- Test that rules parse correctly and maintain proper structure

**File:** `file:tests/evaluator_tests.rs`

Add test cases for evaluating buffers with float/double values:

- Create test buffer with IEEE 754 float bytes
- Create test buffer with IEEE 754 double bytes
- Test detection with different endianness
- Verify confidence scores are calculated correctly

### 6. Update Documentation

**File:** `file:src/parser/ast.rs`

- Add documentation examples for `TypeKind::Float` and `TypeKind::Double`
- Document the endianness field and its impact on byte interpretation
- Add examples showing how to create these variants

**File:** `file:src/evaluator/types.rs`

- Document IEEE 754 compliance in module-level documentation
- Add examples showing float/double reading with different endianness
- Document precision limitations and epsilon-based comparison rationale

**File:** `file:src/evaluator/operators.rs`

- Document floating-point comparison behavior
- Explain epsilon-based equality and why it's necessary
- Document NaN and infinity handling
- Note that bitwise operations are not supported for float/double types

## Testing Strategy

Create unit tests following the existing patterns:

1. **AST Tests** - Verify enum variants serialize/deserialize correctly
2. **Parser Tests** - Verify all six type specifiers parse correctly with proper endianness
3. **Evaluator Tests** - Verify float/double reading with various byte patterns and endianness
4. **Operator Tests** - Verify epsilon-based comparison, NaN handling, infinity handling
5. **Integration Tests** - Verify end-to-end parsing and evaluation of magic files with float/double rules

Test coverage should include:
- All three endianness variants (Little, Big, Native)
- Positive and negative values
- Zero and negative zero
- Infinity and negative infinity
- NaN (Not a Number)
- Denormalized numbers
- Smallest and largest representable values
- Buffer overrun scenarios
- Cross-type comparisons

## Dependencies

No new dependencies required. The existing `byteorder` crate (version 1.5.0) already provides the necessary `read_f32()` and `read_f64()` methods for IEEE 754 floating-point reading with endianness support.

## Architecture Diagram

```mermaid
sequenceDiagram
    participant Parser as Parser<br/>(grammar.rs)
    participant AST as AST<br/>(ast.rs)
    participant Evaluator as Evaluator<br/>(types.rs)
    participant Operators as Operators<br/>(operators.rs)
    
    Parser->>Parser: parse_type_and_operator()
    Note over Parser: Recognizes float/double<br/>with endian prefixes
    Parser->>AST: Create TypeKind::Float/Double
    Note over AST: Stores endianness variant
    
    Evaluator->>Evaluator: read_typed_value()
    Evaluator->>Evaluator: read_float() or read_double()
    Note over Evaluator: Uses byteorder crate<br/>for IEEE 754 reading
    Evaluator->>AST: Create Value::Float/Double
    
    Operators->>Operators: apply_equal()
    Note over Operators: Epsilon-based comparison<br/>for floating-point values
    Operators-->>Evaluator: Comparison result
```

## Type Hierarchy

| Type Specifier | TypeKind Variant | Bytes | Endianness | Value Variant |
|----------------|------------------|-------|------------|---------------|
| `float` | `Float { endian: Native }` | 4 | Native | `Float(f32)` |
| `lefloat` | `Float { endian: Little }` | 4 | Little | `Float(f32)` |
| `befloat` | `Float { endian: Big }` | 4 | Big | `Float(f32)` |
| `double` | `Double { endian: Native }` | 8 | Native | `Double(f64)` |
| `ledouble` | `Double { endian: Little }` | 8 | Little | `Double(f64)` |
| `bedouble` | `Double { endian: Big }` | 8 | Big | `Double(f64)` |

---

Execution Information

Branch: main
Commit: 36adac5

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.