Parser: implement pstring (Pascal string) type

[unclesp1d3r]

Summary

Implement the pstring type for Pascal-style length-prefixed strings.

Context

The pstring type reads a length byte followed by that many bytes of string data. Some binary formats use this encoding for embedded strings.

Current state: Not implemented.

Acceptance Criteria

• TypeKind extended with PString variant
• Parser recognizes pstring type specifier
• Evaluator reads length prefix byte, then reads that many bytes as string
• String comparison operators work with pstring values
• Bounds checking for length prefix exceeding buffer size
• Unit tests

Impact

LOW - Less commonly used than other string types, but needed for full format compatibility.

Files to Modify

• src/parser/ast.rs - Extend TypeKind enum
• src/parser/grammar.rs - Parse pstring specifier
• src/evaluator/types.rs - Implement read_pstring()

References

• GNU file magic(5) man page
• Epic Epic: libmagic-rs Phase 1 MVP #24

[traycerai]

Plan

Observations

The codebase implements a magic file parser and evaluator for file type detection. The type system is defined in file:src/parser/ast.rs with a TypeKind enum, parsed in file:src/parser/grammar.rs using nom combinators, and evaluated in file:src/evaluator/types.rs. Currently, string types include only null-terminated strings. The pstring (Pascal string) type uses a length-prefix byte (0-255) followed by that many bytes of string data, without null termination. String comparison operators in file:src/evaluator/operators.rs already handle Value::String, so pstring values will work automatically once implemented.

Approach

The implementation follows the existing pattern for type variants: extend the AST enum, add parser recognition, implement the evaluator function, and add comprehensive tests. The pstring type will be added as a new TypeKind::PString variant with an optional max_length field (similar to String). The parser will recognize "pstring" as a type specifier. The evaluator will implement read_pstring() that reads one length byte, validates bounds, then reads that many bytes as a UTF-8 string. This approach maintains consistency with existing string handling while adding Pascal string support.

Implementation Steps

1. Extend TypeKind Enum in AST

File: file:src/parser/ast.rs

Add a new variant to the TypeKind enum (around line 96, after the String variant):

/// Pascal string (length-prefixed)
PString {
    /// Maximum length to read (limits the length byte value)
    max_length: Option<usize>,
},

Update the TypeKind tests section (around line 514) to include tests for the new PString variant:

• Test PString with max_length: None
• Test PString with max_length: Some(value)
• Test serialization/deserialization of PString variant
• Verify PString is distinct from other TypeKind variants

2. Add Parser Support for pstring Type

File: file:src/parser/grammar.rs

In the parse_type_and_operator function (around line 1432), add "pstring" to the alt combinator list:

let (input, type_name) = alt((
    tag("lelong"),
    tag("belong"),
    tag("leshort"),
    tag("beshort"),
    tag("long"),
    tag("short"),
    tag("byte"),
    tag("pstring"),  // Add this line
    tag("string"),
))
.parse(input)?;

In the type_kind match statement (around line 1484), add the pstring case:

"pstring" => TypeKind::PString { max_length: None },

Add parser tests in the test section (around line 1806):

• Test parsing "pstring" returns correct TypeKind::PString
• Test "pstring" with attached operators (&, &mask)
• Test "pstring" in complete magic rules
• Test whitespace handling around "pstring"

Update the function documentation (around line 1412) to include pstring in the supported types list.

3. Implement read_pstring Function in Evaluator

File: file:src/evaluator/types.rs

Add a new function after read_string (around line 307):

/// Reads a Pascal-style length-prefixed string from the buffer
///
/// Pascal strings store the length as the first byte (0-255), followed by
/// that many bytes of string data. Unlike C strings, they are not null-terminated.
///
/// # Arguments
///
/// * `buffer` - The byte buffer to read from
/// * `offset` - The offset position to start reading from
/// * `max_length` - Optional maximum length limit (caps the length byte value)
///
/// # Returns
///
/// Returns `Ok(Value::String(string))` if successful, or an error if:
/// - The offset is beyond buffer bounds
/// - The length byte + string data would exceed buffer bounds
/// - The length exceeds max_length (if specified)
///
/// # Examples
///
/// ```
/// // Buffer: [5, 'H', 'e', 'l', 'l', 'o']
/// let buffer = b"\x05Hello";
/// let result = read_pstring(buffer, 0, None).unwrap();
/// assert_eq!(result, Value::String("Hello".to_string()));
/// ```
pub fn read_pstring(
    buffer: &[u8],
    offset: usize,
    max_length: Option<usize>,
) -> Result<Value, TypeReadError> {
    // Check if we can read the length byte
    if offset >= buffer.len() {
        return Err(TypeReadError::BufferOverrun {
            offset,
            buffer_len: buffer.len(),
        });
    }

    // Read the length prefix byte
    let length_byte = buffer[offset];
    let string_length = usize::from(length_byte);

    // Apply max_length limit if specified
    let actual_length = if let Some(max_len) = max_length {
        std::cmp::min(string_length, max_len)
    } else {
        string_length
    };

    // Check if we have enough bytes for the string data
    let string_start = offset + 1;
    let string_end = string_start + actual_length;
    
    if string_end > buffer.len() {
        return Err(TypeReadError::BufferOverrun {
            offset: string_end,
            buffer_len: buffer.len(),
        });
    }

    // Extract string bytes
    let string_bytes = &buffer[string_start..string_end];

    // Convert to UTF-8 string with lossy conversion
    let string_value = String::from_utf8_lossy(string_bytes).into_owned();

    Ok(Value::String(string_value))
}

Update read_typed_value function (around line 353) to handle the new variant:

match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::PString { max_length } => read_pstring(buffer, offset, *max_length),
}

Add comprehensive unit tests in the test module (around line 1326):

• Test reading valid pstring with various lengths (0, 1, 5, 255)
• Test pstring with max_length limit
• Test buffer overrun when length byte exceeds available data
• Test buffer overrun when offset is at buffer boundary
• Test empty buffer
• Test UTF-8 handling (valid and invalid sequences)
• Test length byte of 0 (empty string)
• Test length byte exceeding max_length
• Test edge cases: offset at end of buffer, length byte at last position

4. Verify String Comparison Operators

File: file:src/evaluator/operators.rs

No changes needed. The existing apply_equal, apply_not_equal, and other operators already handle Value::String correctly (around lines 9-237). Since read_pstring returns Value::String, all string comparison operators will work automatically.

Optionally add integration tests to verify pstring values work with operators:

• Test equality comparison with pstring values
• Test inequality comparison
• Test string pattern matching if supported

5. Add Integration Tests

File: file:tests/parser_integration_tests.rs or create file:tests/pstring_tests.rs

Add end-to-end tests for pstring functionality:

• Test parsing magic rules with pstring type
• Test evaluating files with pstring patterns
• Test pstring with child rules
• Test pstring with various operators (=, !=)
• Test real-world Pascal string formats (if any common file formats use them)
• Test error cases: malformed pstring data, buffer overruns

Example test structure:

#[test]
fn test_pstring_basic_parsing() {
    // Create magic file with pstring rule
    // Parse and verify TypeKind::PString is created
}

#[test]
fn test_pstring_evaluation() {
    // Create test file with Pascal string data
    // Evaluate and verify correct string extraction
}

#[test]
fn test_pstring_bounds_checking() {
    // Test files with invalid length prefixes
    // Verify proper error handling
}

6. Update Documentation

Files: file:src/parser/grammar.rs, file:src/evaluator/types.rs, file:README.md (if exists)

Update function documentation to include pstring in supported types:

• Update parse_type_and_operator doc comment to list pstring
• Update module-level documentation if present
• Add pstring examples to relevant documentation

If there's a README or user guide, add pstring to the list of supported magic file types with a brief explanation of Pascal string format.

Sequence Diagram

sequenceDiagram
    participant Parser as Parser (grammar.rs)
    participant AST as AST (ast.rs)
    participant Evaluator as Evaluator (types.rs)
    participant Buffer as File Buffer

    Parser->>Parser: parse_type_and_operator("pstring")
    Parser->>AST: Create TypeKind::PString
    
    Note over Evaluator: During file evaluation
    Evaluator->>Buffer: Read length byte at offset
    Buffer-->>Evaluator: Return length (0-255)
    Evaluator->>Evaluator: Validate bounds (offset + 1 + length)
    Evaluator->>Buffer: Read 'length' bytes
    Buffer-->>Evaluator: Return string data
    Evaluator->>Evaluator: Convert to UTF-8 String
    Evaluator-->>AST: Return Value::String

Loading

Testing Checklist

• Unit tests for TypeKind::PString variant (serialization, equality)
• Parser tests for "pstring" type specifier recognition
• Parser tests for pstring with operators and in complete rules
• Evaluator tests for read_pstring with valid inputs (various lengths)
• Evaluator tests for buffer overrun scenarios
• Evaluator tests for max_length limiting
• Evaluator tests for UTF-8 handling
• Integration tests for end-to-end pstring parsing and evaluation
• Tests for pstring with comparison operators
• Edge case tests: empty strings, maximum length (255), boundary conditions

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase implements a magic file parser and evaluator for file type detection. The type system is defined in `file:src/parser/ast.rs` with a `TypeKind` enum, parsed in `file:src/parser/grammar.rs` using nom combinators, and evaluated in `file:src/evaluator/types.rs`. Currently, string types include only null-terminated strings. The `pstring` (Pascal string) type uses a length-prefix byte (0-255) followed by that many bytes of string data, without null termination. String comparison operators in `file:src/evaluator/operators.rs` already handle `Value::String`, so pstring values will work automatically once implemented.

## Approach

The implementation follows the existing pattern for type variants: extend the AST enum, add parser recognition, implement the evaluator function, and add comprehensive tests. The `pstring` type will be added as a new `TypeKind::PString` variant with an optional `max_length` field (similar to `String`). The parser will recognize "pstring" as a type specifier. The evaluator will implement `read_pstring()` that reads one length byte, validates bounds, then reads that many bytes as a UTF-8 string. This approach maintains consistency with existing string handling while adding Pascal string support.

## Implementation Steps

### 1. Extend TypeKind Enum in AST

**File**: `file:src/parser/ast.rs`

Add a new variant to the `TypeKind` enum (around line 96, after the `String` variant):

```rust
/// Pascal string (length-prefixed)
PString {
    /// Maximum length to read (limits the length byte value)
    max_length: Option<usize>,
},
```

Update the `TypeKind` tests section (around line 514) to include tests for the new `PString` variant:
- Test `PString` with `max_length: None`
- Test `PString` with `max_length: Some(value)`
- Test serialization/deserialization of `PString` variant
- Verify `PString` is distinct from other `TypeKind` variants

### 2. Add Parser Support for pstring Type

**File**: `file:src/parser/grammar.rs`

In the `parse_type_and_operator` function (around line 1432), add "pstring" to the `alt` combinator list:

```rust
let (input, type_name) = alt((
    tag("lelong"),
    tag("belong"),
    tag("leshort"),
    tag("beshort"),
    tag("long"),
    tag("short"),
    tag("byte"),
    tag("pstring"),  // Add this line
    tag("string"),
))
.parse(input)?;
```

In the type_kind match statement (around line 1484), add the pstring case:

```rust
"pstring" => TypeKind::PString { max_length: None },
```

Add parser tests in the test section (around line 1806):
- Test parsing "pstring" returns correct `TypeKind::PString`
- Test "pstring" with attached operators (&, &mask)
- Test "pstring" in complete magic rules
- Test whitespace handling around "pstring"

Update the function documentation (around line 1412) to include pstring in the supported types list.

### 3. Implement read_pstring Function in Evaluator

**File**: `file:src/evaluator/types.rs`

Add a new function after `read_string` (around line 307):

```rust
/// Reads a Pascal-style length-prefixed string from the buffer
///
/// Pascal strings store the length as the first byte (0-255), followed by
/// that many bytes of string data. Unlike C strings, they are not null-terminated.
///
/// # Arguments
///
/// * `buffer` - The byte buffer to read from
/// * `offset` - The offset position to start reading from
/// * `max_length` - Optional maximum length limit (caps the length byte value)
///
/// # Returns
///
/// Returns `Ok(Value::String(string))` if successful, or an error if:
/// - The offset is beyond buffer bounds
/// - The length byte + string data would exceed buffer bounds
/// - The length exceeds max_length (if specified)
///
/// # Examples
///
/// ```
/// // Buffer: [5, 'H', 'e', 'l', 'l', 'o']
/// let buffer = b"\x05Hello";
/// let result = read_pstring(buffer, 0, None).unwrap();
/// assert_eq!(result, Value::String("Hello".to_string()));
/// ```
pub fn read_pstring(
    buffer: &[u8],
    offset: usize,
    max_length: Option<usize>,
) -> Result<Value, TypeReadError> {
    // Check if we can read the length byte
    if offset >= buffer.len() {
        return Err(TypeReadError::BufferOverrun {
            offset,
            buffer_len: buffer.len(),
        });
    }

    // Read the length prefix byte
    let length_byte = buffer[offset];
    let string_length = usize::from(length_byte);

    // Apply max_length limit if specified
    let actual_length = if let Some(max_len) = max_length {
        std::cmp::min(string_length, max_len)
    } else {
        string_length
    };

    // Check if we have enough bytes for the string data
    let string_start = offset + 1;
    let string_end = string_start + actual_length;
    
    if string_end > buffer.len() {
        return Err(TypeReadError::BufferOverrun {
            offset: string_end,
            buffer_len: buffer.len(),
        });
    }

    // Extract string bytes
    let string_bytes = &buffer[string_start..string_end];

    // Convert to UTF-8 string with lossy conversion
    let string_value = String::from_utf8_lossy(string_bytes).into_owned();

    Ok(Value::String(string_value))
}
```

Update `read_typed_value` function (around line 353) to handle the new variant:

```rust
match type_kind {
    TypeKind::Byte => read_byte(buffer, offset),
    TypeKind::Short { endian, signed } => read_short(buffer, offset, *endian, *signed),
    TypeKind::Long { endian, signed } => read_long(buffer, offset, *endian, *signed),
    TypeKind::String { max_length } => read_string(buffer, offset, *max_length),
    TypeKind::PString { max_length } => read_pstring(buffer, offset, *max_length),
}
```

Add comprehensive unit tests in the test module (around line 1326):
- Test reading valid pstring with various lengths (0, 1, 5, 255)
- Test pstring with max_length limit
- Test buffer overrun when length byte exceeds available data
- Test buffer overrun when offset is at buffer boundary
- Test empty buffer
- Test UTF-8 handling (valid and invalid sequences)
- Test length byte of 0 (empty string)
- Test length byte exceeding max_length
- Test edge cases: offset at end of buffer, length byte at last position

### 4. Verify String Comparison Operators

**File**: `file:src/evaluator/operators.rs`

No changes needed. The existing `apply_equal`, `apply_not_equal`, and other operators already handle `Value::String` correctly (around lines 9-237). Since `read_pstring` returns `Value::String`, all string comparison operators will work automatically.

Optionally add integration tests to verify pstring values work with operators:
- Test equality comparison with pstring values
- Test inequality comparison
- Test string pattern matching if supported

### 5. Add Integration Tests

**File**: `file:tests/parser_integration_tests.rs` or create `file:tests/pstring_tests.rs`

Add end-to-end tests for pstring functionality:
- Test parsing magic rules with pstring type
- Test evaluating files with pstring patterns
- Test pstring with child rules
- Test pstring with various operators (=, !=)
- Test real-world Pascal string formats (if any common file formats use them)
- Test error cases: malformed pstring data, buffer overruns

Example test structure:
```rust
#[test]
fn test_pstring_basic_parsing() {
    // Create magic file with pstring rule
    // Parse and verify TypeKind::PString is created
}

#[test]
fn test_pstring_evaluation() {
    // Create test file with Pascal string data
    // Evaluate and verify correct string extraction
}

#[test]
fn test_pstring_bounds_checking() {
    // Test files with invalid length prefixes
    // Verify proper error handling
}
```

### 6. Update Documentation

**Files**: `file:src/parser/grammar.rs`, `file:src/evaluator/types.rs`, `file:README.md` (if exists)

Update function documentation to include pstring in supported types:
- Update `parse_type_and_operator` doc comment to list pstring
- Update module-level documentation if present
- Add pstring examples to relevant documentation

If there's a README or user guide, add pstring to the list of supported magic file types with a brief explanation of Pascal string format.

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Parser as Parser (grammar.rs)
    participant AST as AST (ast.rs)
    participant Evaluator as Evaluator (types.rs)
    participant Buffer as File Buffer

    Parser->>Parser: parse_type_and_operator("pstring")
    Parser->>AST: Create TypeKind::PString
    
    Note over Evaluator: During file evaluation
    Evaluator->>Buffer: Read length byte at offset
    Buffer-->>Evaluator: Return length (0-255)
    Evaluator->>Evaluator: Validate bounds (offset + 1 + length)
    Evaluator->>Buffer: Read 'length' bytes
    Buffer-->>Evaluator: Return string data
    Evaluator->>Evaluator: Convert to UTF-8 String
    Evaluator-->>AST: Return Value::String
```

## Testing Checklist

- [ ] Unit tests for `TypeKind::PString` variant (serialization, equality)
- [ ] Parser tests for "pstring" type specifier recognition
- [ ] Parser tests for pstring with operators and in complete rules
- [ ] Evaluator tests for `read_pstring` with valid inputs (various lengths)
- [ ] Evaluator tests for buffer overrun scenarios
- [ ] Evaluator tests for max_length limiting
- [ ] Evaluator tests for UTF-8 handling
- [ ] Integration tests for end-to-end pstring parsing and evaluation
- [ ] Tests for pstring with comparison operators
- [ ] Edge case tests: empty strings, maximum length (255), boundary conditions

---

Execution Information

Branch: main
Commit: 36adac5

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.