Today conmon is engaging in a byte-capture of stdout/stderr pipes, recording data as a series of JSON documents in a log file. All bytes from both file descriptors are captured without discrimination. We also see that conmon tries to write to disk as fast as possible (but sequentially) with what it reads from both pipes.
There is a natural back-pressure presented by conmon to the processes in the container writing to stdout/stderr that results from the rate the disk can accepts write()s. It is usually the case that all conmon processes server containers write to the same disk. Which means it is possible for "noisy" containers to take up much of the available disk bandwidth leading to unwanted impacts of one container on another.
Further, the rate at which containers write to disk is independent of the rate at which log file readers (or scrapers; e.g. fluentd, fluent-bit, rsyslog, syslog-ng, filebeat, etc.) can read. Because log file rotation is performed independent of the readers and conmon on most platforms, it is possible for conmon to write more data than what a reader can take in before a log file is deleted behind the reader's back with it ever seeing it.
These two problems suggest that log behavior policies applied by conmon to the log stream will give administrators the chance to solve them.
There are three policy behaviors we can begin to consider, each applied to stdout/stderr independently:
back-pressure
- Given a rate specified in "bytes per interval", stop reading from the given pipe if the number of bytes read so far exceeds the limit for that interval, continuing to read again at the beginning of the next interval
- Suggested interval could just be 1 second by default, and/or allow it to be configured along with the # of bytes for that interval
drop
- Given a rate like specified for
back-pressure, drop bytes read over the limit for the remainder of the interval, accepting them again at the start of the next interval
ignore
- Ignore all bytes read from a given pipe without writing them to disk.
This will allow an administrator to apply a policy to all conmon processes. Since conmon processes don't "know" about each other, coordination of the setting of that policy would have to be managed externally to conmon.
It is possible that conmon processes could periodically poll for configuration changes, or accept some sort of signal to re-evaluate the change.
Today
conmonis engaging in a byte-capture ofstdout/stderrpipes, recording data as a series of JSON documents in a log file. All bytes from both file descriptors are captured without discrimination. We also see thatconmontries to write to disk as fast as possible (but sequentially) with what it reads from both pipes.There is a natural back-pressure presented by
conmonto the processes in the container writing tostdout/stderrthat results from the rate the disk can acceptswrite()s. It is usually the case that allconmonprocesses server containers write to the same disk. Which means it is possible for "noisy" containers to take up much of the available disk bandwidth leading to unwanted impacts of one container on another.Further, the rate at which containers write to disk is independent of the rate at which log file readers (or scrapers; e.g. fluentd, fluent-bit, rsyslog, syslog-ng, filebeat, etc.) can read. Because log file rotation is performed independent of the readers and
conmonon most platforms, it is possible forconmonto write more data than what a reader can take in before a log file is deleted behind the reader's back with it ever seeing it.These two problems suggest that log behavior policies applied by
conmonto the log stream will give administrators the chance to solve them.There are three policy behaviors we can begin to consider, each applied to
stdout/stderrindependently:back-pressuredropback-pressure, drop bytes read over the limit for the remainder of the interval, accepting them again at the start of the next intervalignoreThis will allow an administrator to apply a policy to all
conmonprocesses. Sinceconmonprocesses don't "know" about each other, coordination of the setting of that policy would have to be managed externally toconmon.It is possible that
conmonprocesses could periodically poll for configuration changes, or accept some sort of signal to re-evaluate the change.