Feature Improvement: Custom Key Names for Structured Logging Support

[SadiHassan]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...
This is an extension of Structured logging feature which was merged in this PR #39277.
In this extension, user will be able to modify log key names. User will also be able to remove unwanted keys from log.
Consider this log below:
{"date_time":"1650918987.180175","thread_name":"#1","thread_id":"254545","level":"Trace","query_id":"","logger_name":"BaseDaemon","message":"Received signal 2","source_file":"../base/daemon/BaseDaemon.cpp; virtual void SignalListener::run()","source_line":"192"}.

To enable JSON logging support, user need to uncomment the entire <formatting> tag below taken from config.xml.

User will be able to modify key names by changing values under tag values inside tag. For example, to change date_time to MY_DATE_TIME, user can do like:
<date_time>MY_DATE_TIME</date_time>

User can stop unwanted log properties to appear in logs. To do so, they can simply comment out that property from this file.
For example, if you do not want your log to print query_id, you can comment out only <query_id> tag.However, if you comment out all the tags under , the program will print default values for as below.

    <!-- <formatting>
        <type>json</type>
        <names>
            <date_time>date_time</date_time>
            <thread_name>thread_name</thread_name>
            <thread_id>thread_id</thread_id>
            <level>level</level>
            <query_id>query_id</query_id>
            <logger_name>logger_name</logger_name>
            <message>message</message>
            <source_file>source_file</source_file>
            <source_line>source_line</source_line>
        </names>
    </formatting> -->

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

[evillique]

It seems like although the .xml file in the test has changed we do not check whether the labels that we assigned are in the logs or not.

[SadiHassan]

It seems like although the .xml file in the test has changed we do not check whether the labels that we assigned are in the logs or not.

Yes, we don't. Last time Alexey asked to make a simple test case that will only check if a valid JSON log is the output or not. However, I am open to add more checking if you require.

[SadiHassan]

Hi @evillique , I updated the integration test code to cover different scenarios of custom keys. Please kindly have a look and give me feedback for any changes required. Thanks!

[SadiHassan]

Hi @evillique , once you get a chance, can you please review my last updates and confirm?

[evillique]

Usually, tests with separate instances are made for testing communication between them, but the overhead does not seem to be large so it is probably ok.

[evillique]

Failed check seem unrelated