render_to_log_kwargs does not handle all kwargs

[last-partizan]

Hi, i want to use standard logging for formatting, to use formatting from https://github.com/elastic/ecs-logging-python

But, following configuration:

structlog.configure(
    processors=[
        # Transform event dict into `logging.Logger` method arguments.
        # "event" becomes "msg" and the rest is passed as a dict in
        # "extra". IMPORTANT: This means that the standard library MUST
        # render "extra" for the context to appear in log entries! See
        # warning below.
        structlog.stdlib.render_to_log_kwargs,
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

Causes errors when using structlog in the following way:

logger = structlog.getLogger()
try:
    assert False
except:
    logger.error("Error happened", exc_info=True)

... 
KeyError: "Attempt to overwrite 'exc_info' in LogRecord"

This is because not all kwargs to logging should go to extra.

I fixed this by rewriting render_to_log_kwargs in the following way:

def _render_to_log_kwargs(_, __, event_dict: EventDict) -> EventDict:
    required = {
        "msg": event_dict.pop("event"),
    }
    optional = {
        k: event_dict.pop(k)
        for k in
        ["exc_info", "stack_info", "stackLevel"]
        if k in event_dict
    }
    return required | optional | {
        "extra": event_dict,
    }

But i think this should be merged upstream. I can make PR if you want, or you can add those changes yourself.

[eoinmorgan]

+1, I also hit the exact same issue. I'm trying to use the stdlib.LoggerFactory and structlog.stdlib.ProcessorFormatter to defer all of the log record manipulation into the stdlib logging, but using the structlog processors (so that 3rd party libs that use logging.getLogger() have consistent formatting with my own code's logs). The stdlib.BoundLogger() calls with exc_info in the extra kwarg :/ Thanks for your help

[hynek]

Hm this never popped up, because you'd usually format exceptions using format_exc_info before handing them off to standard library.

I guess we could sieve out keywords that must not be passed on, are there more than exc_info?

[last-partizan]

Yeah, adding format_exc_info fixes this.

I don't think sieving out is correct, we need to pass actual arguments. Three of them: "exc_info", "stack_info", "stackLevel".