Add LogRecordCompatibleDictRenderer processor

[sky-code]

Add new processor for easy integration with stdlib
basically LogRecordCompatibleDictRenderer forward structlog event_dict to stdlib logger

[codecov-io]

Current coverage is 100% (diff: 100%)

Merging #98 into master will not change coverage

@@           master   #98   diff @@
===================================
  Files          13    13          
  Lines         753   766    +13   
  Methods         0     0          
  Messages        0     0          
  Branches       95    96     +1   
===================================
+ Hits          753   766    +13   
  Misses          0     0          
  Partials        0     0          

Powered by Codecov. Last update 4db913e...2ae3274

[hynek]

This sounds interesting! Is it basically an inverted adapter to funnel structlog into logging and format it there freely? So you could use for example https://github.com/madzak/python-json-logger and have json everywhere? Is it something you’re already using yourself?

[sky-code]

Yes it is basically render event_dict into logging than you can handle it by using any logging handler and formatter. I use it already in my project, here is config:

structlog.configure(
        processors=[
            structlog.stdlib.PositionalArgumentsFormatter(),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            LogRecordDictRenderer(True, 'extra')
        ],
        context_class=dict,
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.BoundLogger,
        cache_logger_on_first_use=True,
    )

and than config for stdlib logging

disable_existing_loggers = false
version = 1

[formatters.colored]
style = '$'
class = 'colorlog.ColoredFormatter'
format = '${log_color}${asctime} - ${name} - ${levelname} - ${message}'

[formatters.colored_extra]
style = '$'
class = 'colorlog.ColoredFormatter'
format = '${log_color}${asctime} - ${name} - ${levelname} - ${message} - ${extra}'


[handlers.console]
class = "logging.StreamHandler"
formatter = "colored"
level = "DEBUG"
stream = "ext://sys.stdout"

[handlers.console_extra]
class = "logging.StreamHandler"
formatter = "colored_extra"
level = "DEBUG"
stream = "ext://sys.stdout"

[handlers.logstash]
level = 'INFO'
class = 'logstash.TCPLogstashHandler'
host = 'localhost'
port = 9100 # Default value: 5959
version = 1 # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
message_type = 'logstash' # 'type' field in logstash message. Default value: 'logstash'.
fqdn = false # Fully qualified domain name. Default value: false.
# tags = ['tag1', 'tag2'] # list of tags. Default: None.

[loggers.natrix]
handlers = ["console_extra", "logstash"]
level = "DEBUG"
propagate = false

[root]
handlers = ["console", "logstash"]
level = "INFO"

Here you can see 2 formatter, one for root logger and one extended to display extra field which contains all data from event_dict, this give same result as using KeyValueRenderer, just for development and debug you will see all what you have. Than I use handler from python-logstash package to send logs into logstash with all data, but this logstash logger handler filter out extra field which duplicate data at general for debugging and not needed in logs storage
In general with this config you can use any stdlib handler and formatter to do what you want and at same time using features from structlog

[sky-code]

I was confused that structlog not have builtin way to forward log message with data what you have to stdlib logging for handling it here by using all handlers available already.
In my opinion structlog should have some builtin processor for integration with stdlib logging

If you accept this pull, consider next things

• move LogRecordCompatibleDictRenderer to stdlib module or leave it in processors module?
• rename LogRecordCompatibleDictRenderer to some more human friendly and pythonic
• rename add_extra_event_dict argument
• renmae add_event_dict_with_key argument

[hynek]

The lacking stdlib supporting is simply due to the fact that I try to avoid to use it myself as well as I can (it’s painfully slow and complex) and nobody stepped up so far to tackle the harder problems. :)

[hynek]

First, please accept my apology, that it took so long for me to take care of this.

As you may see in the final commit, I have changed quite a bit especially because this function is a logging hotspot and should be as fast as possible. I believe the options you had are better handled with additional processors when you configure structlog for development. Of course, I’ve let you still be the author.

Many thanks though for this simply yet genius idea. I have promoted your approach as the main approach in the docs because that’s what many people asked for. This should have been in structlog from day 1!