Skip to content

Stop writing trailing zeros out to chunk store #631

@bboreham

Description

@bboreham

We create all chunks in memory as 1K long, and write the full 1K to store every time, even though many chunks will be half-full or less so we are writing a lot of trailing zero bytes.

AWS then base64-encodes all those zeros.

There doesn't seem to be an API on Prometheus (v1) chunks to find out how many bytes are used, but we could simply look for the last non-zero byte in the Marshal() output. We already include the data length in Chunk.Encode().

It's not clear to me whether we need to add back all the zeros when we read back the data, for double-delta encoding.

Slightly related: #300

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions