filebeat

[jordotech]

After experimenting with this app it dawned on me - this is what filebeat is for. Filebeat lives on the server you're shipping logs FROM and operates on an interval so it does not interfere with request times, etc.

So my question is why is it preferable to store messages in a sqlite db and async ship them as opposed to using filebeat?

[eht16]

First, request times should not be affected that much as the shipping happens in a separate thread asynchronously, i.e. after the request has been handled already.
Of course, we are talking about Python and its GIL but a separate thread still should be better than nothing.

About FileBeat:
there are a few differences which make this package worth using:

• FileBeat needs to be installed and configured on the server independently from your application - this is an advantage and disadvantage - depending on your environment and deployment strategy.
  Deploying FileBeat server-side can be useful if you host multiple applications on a server and one FileBeat instance can handle the log files of all those applications. However, in a Docker / Kubernetes / whatever container world, you likely rather have an application as the single process running in the container and you do not want to have sidecar containers or so just to send your log events to Logstash.

• FileBeat process log files, i.e. you need to write your log messages to a log file with some format and then have FileBeat or Logstash parse this format again to get a structured log event again - python-logstash-async sends a structured log event already to Logstash, no need for additional parsing and gives you much more flexibility to add custom data to your log events without struggling with a logfile format

• when exceptions are to be logged with stack traces included, FileBeat or Logstash needs to be configured carefully to properly detect and handle multiline log messages - with python-logstash-async no special handling of multiline log events is necessary as it cleanly integrates with Python's logging framework.

After all, I think both approaches are valid and have their own use cases. It's your decision which fits best to your situation and requirements.

[jordotech]

Thank you for the detailed answer, this could go in the docs!