[Monolog] Added ElasticsearchLogstashHandler

[lyrixx]

Q	A
Branch?	4.4
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets
License	MIT
Doc PR

This PR was initially submitted on Monolog.
It has been refused , but Jordi suggested to add it to the symfony bridge. So here we go :)

---

ATM, there are few options to push log to Elastic Stack in order to play with them:

• install logstash and use a Gelf handler. It works but you have to install logstash and configure it. Not an easy task. More over, it need an extra PHP package
• use the ES handler: It does not play well with context and extra: Kibana is not able to filter on nested object. And this handler is tightly coupled to the ElasticaFormatter formater. More over, it need an extra PHP package
• use something to parse file logs. This is really a bad idea since it involves a parsing... More over a daemon is needed to do that (file beat / logstash / you name it)

This is why I'm introducing a new Handler.

• There is not need to install anything (expect ES, of course)
• It play very well with Kibana, as it uses the Logstash format
• It requires symfony/http-client, but in a modern PHP application (SF 4.3) this dependency is already present
• It slow down a bit the application since it trigger an HTTP request for each logs. But symfony/http-client is non-blocking. If you want to use it in production, I recommend to wrap this handler in a buffer handler or a cross-finger handle to have only one HTTP call.

---

Some performance consideration en a prod env with a buffer handler + this one

• with push to ES: https://blackfire.io/profiles/f94ccf35-9f9d-4df1-bfc5-7fa75a535628/graph
• with push to ES commented: https://blackfire.io/profiles/6b66bc18-6b90-4341-963f-797f7a7a689c/graph

As you can see, as requests are made synchronously, there is no penalty on AppKernel::Handler() 😍! But the PHP worker has more work to do, and it's busy much more time (about X2)

I explained everything in the PHP Doc Block

---

This is what you can expect out of the box

[nicolas-grekas]

Note: This handler is not aimed to be used in productio

I'd suggest telling this in the docblock + explain how to do it in prod

[lyrixx]

I'd suggest telling this in the docblock + explain how to do it in prod

Actually, I could work a bit on it to make it available on production:
It it's used in combinaison with a buffer handler, or a finger cross handler, all logs could be sent by batch, and so only one HTTP call.

WDYT?

[fabpot]

Your suggestion for a prod mode makes sense to me.

[lyrixx]

• I updated the PHPdoc.
• I added blackfire profile to the PR description
• I enhanced the main dockbloc
• I added an entry to the readme
• I changed the default host to 127.0.0.1:9200
• I added a timeout to 1 sec

[lyrixx]

@nicolas-grekas How can I disable all exceptions in the http-client? Or at least catch them all? (But I think this is not do-able since I use the __destruct feature)

[lyrixx]

Hmm, I managed to do something. Could you look at the last commit? Thanks

[lyrixx]

I rebased this PR. It's now ready and green 💚

[noniagriconomie]

minor doc review :)

edit : do you plan to add real doc, or a blog post to explain the config for dev/vs prod ?
i think it is a must have, to not degrade perf on prod :)

[lyrixx]

edit : do you plan to add real doc, or a blog post to explain the config for dev/vs prod ?
i think it is a must have, to not degrade perf on prod :)

Not really. There is already many blog post on internet about symfony + ELK

[lyrixx]

I addressed your comments

[lyrixx]

my comment was more for configuring this new handler in the proper way inside a symfony dev|prod stack :)

This will be done in the monolog-bundle :)

[B-Galati]

Do you plan to add some doc ?

[fabpot]

Thank you @lyrixx.

[nicolas-grekas]

@lyrixx could you please have a look at failures on branch master? It looks like the new class is not compatible with Monolog 2. Thanks.

[lyrixx]

👍 I will look at it. Thanks

[lyrixx]

#33241

[DavidBadura]

thanks @lyrixx for this contribution :)

I tried it and found the problem that the logging recrudes. The handler uses the HttpClient, which logs every request (POST http://127.0.0.1:9200/_bulk). I had to exclude the channel http_client, otherwise I hit the memory limit.

monolog:
    handlers:
        es:
            type: service
            id: Symfony\Bridge\Monolog\Handler\ElasticsearchLogstashHandler
            channels: ["!http_client"]

Do you have any idea how to still log the http client requests without landing in a recrusion?
And I think, the problem should also be documented.

[DavidBadura]

I bypassed this problem by not using the global http client:

    Symfony\Bridge\Monolog\Handler\ElasticsearchLogstashHandler:
        arguments:
            $client: null

so I can remove the http_client channel exclude in the mongo handler config :)

[lyrixx]

Hello. I did not anticipate this 😂
I'm working on a fix.

[lyrixx]

Hmm, I don"t see how Symfony could fix this issue since it's "normal" to have this behavior.
So instead I submitted a PR to the doc repo symfony/symfony-docs#12642
and I fixed my article https://jolicode.com/blog/how-to-visualize-symfony-logs-in-dev-with-elasticsearch-and-kibana

Thanks for the feedback 👍