Trying to increase log buffer size fails miserably

[zwoop]

... The lesson is, never try!

This is likely same / similar to the old TS-777 issue (Craig reminded me):

https://issues.apache.org/jira/browse/TS-777

I tried again with 32KB, which works fine on Docs / CI, but fails in our (much more busy) production environment. When it fails, logging stops, and we start seeing the following errors:

Sep 27 16:29:58 s_sys@xxx traffic_server[17236]: {0x7ffb39f0c700} NOTE: Skipping the current log entry for tcpinfo.log because its size (56) exceeds the maximum payload space in a log buffer
Sep 27 16:29:58 s_sys@xxx traffic_server[17236]: {0x7ffb3590c700} NOTE: Skipping the current log entry for tcpinfo.log because its size (56) exceeds the maximum payload space in a log buffer
Sep 27 16:29:58 s_sys@xxx traffic_server[17236]: {0x7ffb3950c700} NOTE: Skipping the current log entry for foo.log because its size (560) exceeds the maximum payload space in a log buffer

Note how it not only affects regular access logs, but also plugin logs (tcpinfo in this case).

[zwoop]

I wonder now if there's some odd mismatch between these two settings:

proxy.config.log.log_buffer_size=9216
proxy.config.log.ascii_buffer_size=36864

Where the default ascii_buffer_size is exactly 4x the size of the log_buffer_size. @bneradt you've been knee deep in this, what do you think?

[bneradt]

HI @zwoop.

I don't think there should be a mismatch between those, per se. Basically, log_buffer_size is the most "fundamental" size setting: it applies to both binary and ascii logs. As such, I'd expect all logs to be limited to 9K given the configuration you mention. log_buffer_size will trump ascii_buffer_size and max_line_size.

As an aside, the various log configurations were confusing to me so I tried to document them here (most were simply not documented before):

https://docs.trafficserver.apache.org/en/latest/admin-guide/files/records.config.en.html?highlight=ascii_buffer_size#proxy-config-log-max-line-size

Please let me know if something seems wrong or could be described better.

[maskit]

I looked into the code, and the only finding was that the way to allocate a buffer for LogBuffer depends on log_buffer_size. With the default value the buffer is allocated by IOBufferAllocator, but with the size of 32KB the buffer is allocated by ats_malloc. The threshold for switching allocators is proxy.config.log.io.max_buffer_index (the default size is 32KB).

If use of ats_malloc is the trigger of the issue, increasing the threshold may be a workaround.

[ywkaras]

Most Yahoo prod proxies are using 26624 for log_buffer_size config var.

[ywkaras]

Related PR: #8507