Skip to content

Conversation

@glopesdev
Copy link
Contributor

@glopesdev glopesdev commented Mar 20, 2025

Although the majority of common analysis use cases requires only the parsing of Harp flat binary files and messages into data frames, it is also desirable to support the converse operation: generating Harp binary files from existing data frames.

This PR enables post-processing and storage of processed datasets using the Harp binary format, for compatibility with existing systems. Both a write and format functions are defined to allow writing data to a Harp binary file, or to a memory array buffer, respectively. Regression tests were added to validate successful round-trip. Example signature for the write function:

def write(
    file: Union[str, bytes, PathLike[Any], BinaryIO],
    data: pd.DataFrame,
    address: int,
    dtype: Optional[np.dtype] = None,
    port: Optional[int] = None,
    epoch: Optional[datetime] = None,
    message_type: Optional[MessageType] = None,
):

Further testing may be required to ensure timestamp roundtrip is not lossy if done over float64. Given the specs of the 8-bit protocol this should be the case, but conversion may require revision in case of the 32-bit protocol.

@glopesdev glopesdev added the feature New planned feature label Mar 20, 2025
@glopesdev glopesdev requested a review from bruno-f-cruz March 20, 2025 01:07
Copy link
Member

@bruno-f-cruz bruno-f-cruz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Mostly just one question to satisfy my curiosity lol

@glopesdev glopesdev merged commit 7f10063 into main Mar 21, 2025
12 checks passed
@glopesdev glopesdev deleted the write-buffer branch March 21, 2025 00:15
bruno-f-cruz added a commit that referenced this pull request Mar 22, 2025
bruno-f-cruz added a commit that referenced this pull request Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New planned feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants