Is it recommended to ship logs directly from a PHP app to ElasticSearch? Or is it always better to store to file and use filebeat to ship the logs? I’m familiar with C# and Serilog implementation sending directly to ES and we’ve never had an issue with it, but was wondering if the same could be done with a PHP app using Monolog without performance being severely impacted. I’m assuming Serilog is able to asynchronously send the logs, I’m not sure if the Monolog implementation on PHP is able to do that or if it would only be OK to ship directly as long as they’re infrequent error logs but info/debug logging would cripple performance. Anybody have any experience doing this? For additional background, we’d be using AWS OpenSearch and the PHP app would also be running at AWS in ECS/Fargate containers, so it would be inside the same network.
Advertisement
Answer
personally I would recommend logging to a file and then shipping, no matter the end datastore you end up chosing
the reason is that otherwise you need to handle a tonne of logic in your client – load balancing/failover, retries, error handling, back pressure – and while another client library for that datastore may abstract some of this out, it’s probably not the best place for it
the biggest issue there would be back pressure. what happens if something happens that delays, or stops, logging to your datastore? do you keep the log in memory and retry later? do you block till the log is stored? how do you manage those aspects
if you log to a file (or even stdout for containers) you have a pretty clear delineation point for your code. you will likely have a backup of the log incase you ever need to check it, and you can use task specific tooling to ship your logs