auto_partial_merge doesn't work in kubernetes source

[abanaev]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We use the vector configuration attached below

According to the vector documentation :

Partial message merging
By default, Vector merges partial messages that are split due to the Docker size limit. For everything else, we recommend that you use the reduce\transform, which enables you to handle custom merging of things like stacktraces.

Our applications in k8s can have logs larger than 1MB. They are split by container runtime(docker) into partial. In kafka we already see a split message.

We also tried to use a configuration without transformations

Configuration

sources:
  k8sLogs_source:
    type: kubernetes_logs
    auto_partial_merge: true
    data_dir: /vector-data-dir
    delay_deletion_ms: 60000
    fingerprint_lines: 1
    glob_minimum_cooldown_ms: 60000
    ignore_older_secs: 600
    max_line_bytes: 1048576
    oldest_first: true
    read_from: beginning

transforms:
  k8sLogs_transform_RAW_log:
    inputs:
      -k8sLogs_source
    type: remap
    source: |
        parse .........
        .....
        ....

sinks:
  i16nodes_k8sLogs_sink:
    type: "kafka"

Version

0.34.2-alpine

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[Alexander-177]

same problem

[cyberjihadxyz]

Same problem please up

[sc0rp10]

hello, @jszwedko any updates? it seems to be a serious bug that prevents the long log lines being stored. Our vector installation loses 10% of audit logs, as it was discovered recently.

[jszwedko]

hello, @jszwedko any updates? it seems to be a serious bug that prevents the long log lines being stored. Our vector installation loses 10% of audit logs, as it was discovered recently.

No updates from me unfortunately :( As always, we prioritize issues regularly and are open to contributions.

[jszwedko]

#22582 may be related.

[ganelo]

At a guess, it could be that you're setting max_line_bytes too low to account for the overhead - in our case, our logs are split into 2.5 MiB chunks by crio but we still had to set max_line_bytes to ~2637824 (16 KiB of buffer over the 2.5 MiB size) to account for some padding (was our best guess). Alternately, it could be that you're mistaken as to the size of the chunks that docker is splitting them into.

#22582 is a slightly different problem - if you do actually want to drop very large log lines (eg > 50 MiB or some such) but keep smaller merged lines, there's no way to do that today. My PR would add a new config field to allow specifying a limit on the size of merged lines, since there's no way to achieve the described behavior with the current implementation of max_line_bytes and auto_partial_merge alone.

[ganelo]

Another possibly relevant issue for some people experiencing this issue (albeit not the OP, from their pasted config): #9790 (see also #8705 which is linked indirectly from #9790).

The thinking would be that if you have batch.max_bytes set on your source (or somewhere in your upstream pipeline if your source is another vector instance) then that might be dropping those larger lines before they get a chance to reach you.