Fix Python 3.12 compatibility for ctypes bitfield structures

[khurrameycon]

Python 3.12 Compatibility Changes

Summary

This document outlines the changes made to make the open-dis-python library compatible with Python 3.12. The primary issue addressed was ctypes structure padding and alignment, which became stricter in Python 3.12.

Problem Statement

When running on Python 3.12, the library was failing due to changes in how ctypes handles bitfield structures. Python 3.12 enforced stricter rules about struct padding, requiring all bitfields within a single ctypes.Structure to use the same underlying C type to avoid unexpected padding bytes.

Files Changed

1. opendis/record/bitfield.py (CRITICAL FIX)

Lines changed: 119 lines (new) vs 101 lines (old)

Key Changes:

OLD Implementation (Pattern Matching Approach):

def _field(name: str, ftype: DisFieldType, bits: int) -> CTypeFieldDescription:
    """Helper function to create the field description tuple used by ctypes."""
    match (ftype, bits):
        case (INTEGER, b) if 0 < b <= 8:
            return (name, c_uint8, bits)
        case (INTEGER, b) if 8 < b <= 16:
            return (name, c_uint16, bits)
        case (INTEGER, b) if 16 < b <= 32:
            return (name, c_uint32, bits)
        case _:
            raise ValueError(f"Unrecognized (ftype, bits): {ftype}, {bits}")

Problem: Each field could potentially use a different base type (c_uint8, c_uint16, c_uint32) based on its individual size. This caused padding issues in Python 3.12 when fields of different sizes were combined in the same structure.

NEW Implementation (Two-Pass Approach):

def _field(name: str,
          ftype: DisFieldType,
          bits: int,
          base_ctype: type[_SimpleCData]) -> CTypeFieldDescription:
    """Helper function to create the field description tuple used by ctypes.

    Args:
        name: Field name
        ftype: Field type (currently only INTEGER supported)
        bits: Number of bits for this field
        base_ctype: The base C type to use for all fields (ensures proper packing)
    """
    if ftype != INTEGER:
        raise ValueError(f"Unrecognized ftype: {ftype}")
    if bits <= 0 or bits > 32:
        raise ValueError(f"Field size must be between 1 and 32: got {bits}")
    return (name, base_ctype, bits)

NEW bitfield() function logic:

# First pass: calculate total bitsize and validate
bitsize = 0
for field_name, ftype, bits in fields:
    # ... validation ...
    bitsize += bits

# Determine base ctype based on TOTAL size (critical for Python 3.12+ compatibility)
# All fields must use the same underlying type to avoid padding issues
if bitsize <= 8:
    base_ctype = c_uint8
elif bitsize <= 16:
    base_ctype = c_uint16
elif bitsize <= 32:
    base_ctype = c_uint32
else:
    raise ValueError(f"Bitfield size {bitsize} exceeds maximum of 32 bits")

# Second pass: create fields with consistent base type
struct_fields = []
for field_name, ftype, bits in fields:
    struct_fields.append(_field(field_name, ftype, bits, base_ctype))

Solution:

• Two-pass approach: First calculates the total bitfield size, then determines a single base_ctype based on the TOTAL size
• Consistent base type: ALL fields in the structure now use the same underlying C type, preventing padding issues
• Explicit parameter: The _field() function now takes base_ctype as a parameter rather than inferring it from individual field sizes

2. .gitignore (Housekeeping)

Changes:

• Fixed pattern: __pycache* → __pycache__* (added missing underscore)
• Added: *.pyc (explicitly exclude compiled Python files)
• Added: *.pyo (exclude optimized bytecode files)

3. Files NOT Changed

The following files remained unchanged as they were already compatible with Python 3.12:

• opendis/dis7.py - Already using modern type annotations (list[...], | None)
• opendis/DataInputStream.py - Binary I/O, version agnostic
• opendis/DataOutputStream.py - Binary I/O, version agnostic
• opendis/PduFactory.py - Factory pattern, compatible
• opendis/RangeCoordinates.py - Math utilities, compatible
• pyproject.toml - Already requires Python >=3.10
• Examples and tests - No changes needed

Technical Details

Why This Fix Was Necessary

In Python 3.12, the ctypes module became stricter about structure layout. When a bitfield structure contains multiple fields with different underlying C types, the C compiler may insert padding bytes to maintain proper alignment. This caused:

1. Size mismatches: The actual struct size differed from the expected size
2. Parsing failures: Binary data couldn't be correctly deserialized
3. Serialization errors: Data written to binary format was malformed

The Solution

By ensuring all bitfields use the same underlying C type determined by the TOTAL structure size:

• No unexpected padding: All fields align naturally within the same base type
• Predictable size: sizeof(Bitfield) equals the expected byte size
• Correct binary representation: Data serializes/deserializes correctly

Example

Consider a bitfield with these fields:

• Field A: 3 bits
• Field B: 10 bits
• Total: 13 bits (requires 2 bytes = 16 bits)

OLD (Python 3.11 and earlier):

• Field A would use c_uint8 (3 bits <= 8)
• Field B would use c_uint16 (10 bits > 8, <= 16)
• Result: Potential padding issues, but Python 3.11 was lenient

NEW (Python 3.12 compatible):

• Total is 13 bits, which fits in 16 bits
• Base type determined: c_uint16
• Both Field A and Field B use c_uint16
• Result: No padding, correct size, works in Python 3.12+

Testing

The changes maintain backward compatibility with Python 3.10+ while fixing compatibility with Python 3.12.

Impact

This change affects all DIS records that use bitfields, including:

• Antenna patterns
• Modulation parameters
• Radio system types
• Other non-octet-aligned binary fields

The fix ensures correct binary serialization/deserialization across all Python 3.10+ versions.

Credits

These changes enable the open-dis-python library to work correctly on Python 3.12 while maintaining compatibility with earlier Python 3.10+ versions.

[leif81]

Thank-you @khurrameycon I'll take a look.

If any other users of the library have comments please feel free to add them here.

[ngjunsiang]

Looks good to me 👍