Skip to content

Commit

Permalink
Logging Attributes (#1033)
Browse files Browse the repository at this point in the history 
* Log Forwarding User Attributes (#682)

* Add context data setting

Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: Hannah Stepanek <[email protected]>
Co-authored-by: Lalleh Rafeei <[email protected]>

* Update record_log_event signature with attributes

* Logging attributes initial implementation

* Fix settings attribute error

* Update logging instrumentation with attributes

* Update log handler API

* Add loguru support for extra attrs

* Add more explicit messaging to validator

* Expanding testing for record_log_event

* Expand record log event testing

* Fix settings typo

* Remove missing loguru attributes from test

* Adjust safe log attr encoding

* Correct py2 issues

* Fix missing record attrs in logging.

Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: Hannah Stepanek <[email protected]>
Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Hannah Stepanek <[email protected]>

* Log Attribute Filtering (#1008)

* Expand validator for log events

* Add settings for context data filtering

* Add attribute filtering for log events

* Linting

* Apply suggestions from code review

* Remove none check on attributes

* Squashed commit of the following:

commit 3962f54
Author: Uma Annamalai <[email protected]>
Date:   Thu Jan 4 12:50:58 2024 -0800

    Remove case sensitive check in ASGIBrowserMiddleware check. (#1017)

    * Remove case sensitive check in should_insert_html.

    * [Mega-Linter] Apply linters fixes

    * Remove header decoding.

    ---------

    Co-authored-by: umaannamalai <[email protected]>

commit c3314ae
Author: Lalleh Rafeei <[email protected]>
Date:   Tue Jan 2 17:17:20 2024 -0800

    Temporarily pin hypercorn version in tests (#1021)

    * Temporarily pin hypercorn to <0.16

    * Temporarily pin hypercorn to <0.16

    * Add comment to tox.ini

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit 1357145
Author: Uma Annamalai <[email protected]>
Date:   Tue Jan 2 16:17:08 2024 -0800

    Drop py27 from memcache testing. (#1018)

commit 23f969f
Author: Timothy Pansino <[email protected]>
Date:   Wed Dec 20 17:01:50 2023 -0800

    Nonced CSP Support (#998)

    * Add nonce to CSP in browser agent

    * Adjust nonce position

    * Add testing for browser timing nonces

commit 8bfd2b7
Author: Uma Annamalai <[email protected]>
Date:   Mon Dec 18 13:58:10 2023 -0800

    Remove RPM config workflow. (#1007)

* Add Dictionary Log Message Support (#1014)

* Add tests for logging's json logging

* Upgrade record_log_event to handle dict logging

* Update logging to capture dict messages

* Add attributes for dict log messages

* Implementation of JSON message filtering

* Correct attributes only log behavior

* Testing for logging attributes

* Add logging context test for py2

* Logically separate attribute tests

* Clean out imports

* Fix failing tests

* Remove logging instrumentation changes for new PR

* Add test for record log event edge cases

* Update record_log_event for code review

* Fix truncation

* Move safe_json_encode back to api.log as it's unused elsewhere

* Black formatting

* Add missing import

* Fixup warning message

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Logging Attribute Instrumentation (#1015)

* Add tests for logging's json logging

* Upgrade record_log_event to handle dict logging

* Update logging to capture dict messages

* Add attributes for dict log messages

* Implementation of JSON message filtering

* Correct attributes only log behavior

* Testing for logging attributes

* Add logging context test for py2

* Logically separate attribute tests

* Clean out imports

* Fix failing tests

* Linting

* Ignore path hash

* Fix linter errors

* Fix linting issues

* Apply suggestions from code review

* StructLog Attribute Instrumentation (#1026)

* Add tests for logging's json logging

* Upgrade record_log_event to handle dict logging

* Update logging to capture dict messages

* Add attributes for dict log messages

* Implementation of JSON message filtering

* Correct attributes only log behavior

* Testing for logging attributes

* Add logging context test for py2

* Logically separate attribute tests

* Clean out imports

* Fix failing tests

* Structlog cleanup

* Attempting list instrumentation

* Structlog attributes support

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Remove other frameworks changes

* Bump tests

* Change cache to lru cache

* Linting

* Remove TODO

* Remove unnecessary check

---------

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Loguru Attribute Instrumentation (#1025)

* Add tests for logging's json logging

* Upgrade record_log_event to handle dict logging

* Update logging to capture dict messages

* Add attributes for dict log messages

* Implementation of JSON message filtering

* Correct attributes only log behavior

* Testing for logging attributes

* Add logging context test for py2

* Logically separate attribute tests

* Clean out imports

* Fix failing tests

* Structlog cleanup

* Attempting list instrumentation

* Structlog attributes support

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Loguru instrumentation refactor

* New attribute testing

* Move exception settings

* Clean up testing

* Remove unneeded option

* Remove other framework changes

* [Mega-Linter] Apply linters fixes

* Bump tests

---------

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: TimPansino <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Temporarily pin starlette tests

* Update web_transaction.py

---------

Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: Hannah Stepanek <[email protected]>
Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Hannah Stepanek <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: TimPansino <[email protected]>
  • Loading branch information
@TimPansino @umaannamalai
@hmstepanek @lrafeei @mergify
7 people authored Jan 12, 2024
1 parent 9990e71 commit aebf47f
Show file tree
Hide file tree
Showing 33 changed files with 1,208 additions and 503 deletions.
7 changes: 4 additions & 3 deletions newrelic/api/application.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@


class Application(object):

_lock = threading.Lock()
_instances = {}

Expand Down Expand Up @@ -162,9 +161,11 @@ def record_transaction(self, data):
if self.active:
self._agent.record_transaction(self._name, data)

def record_log_event(self, message, level=None, timestamp=None, priority=None):
def record_log_event(self, message, level=None, timestamp=None, attributes=None, priority=None):
if self.active:
self._agent.record_log_event(self._name, message, level, timestamp, priority=priority)
self._agent.record_log_event(
self._name, message, level, timestamp, attributes=attributes, priority=priority
)

def normalize_name(self, name, rule_type="url"):
if self.active:
Expand Down
53 changes: 41 additions & 12 deletions newrelic/api/log.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@
from newrelic.api.time_trace import get_linking_metadata
from newrelic.api.transaction import current_transaction, record_log_event
from newrelic.common import agent_http
from newrelic.common.encoding_utils import json_encode
from newrelic.common.object_names import parse_exc_info
from newrelic.core.attribute import truncate
from newrelic.core.config import global_settings, is_expected_error
from newrelic.packages import six


def format_exc_info(exc_info):
Expand All @@ -42,8 +44,30 @@ def format_exc_info(exc_info):
return formatted


def safe_json_encode(obj, ignore_string_types=False, **kwargs):
# Performs the same operation as json_encode but replaces unserializable objects with a string containing their class name.
# If ignore_string_types is True, do not encode string types further.
# Currently used for safely encoding logging attributes.

if ignore_string_types and isinstance(obj, (six.string_types, six.binary_type)):
return obj

# Attempt to run through JSON serialization
try:
return json_encode(obj, **kwargs)
except Exception:
pass

# If JSON serialization fails then return a repr
try:
return repr(obj)
except Exception:
# If repr fails then default to an unprinatable object name
return "<unprintable %s object>" % type(obj).__name__


class NewRelicContextFormatter(Formatter):
DEFAULT_LOG_RECORD_KEYS = frozenset(vars(LogRecord("", 0, "", 0, "", (), None)))
DEFAULT_LOG_RECORD_KEYS = frozenset(set(vars(LogRecord("", 0, "", 0, "", (), None))) | {"message"})

def __init__(self, *args, **kwargs):
super(NewRelicContextFormatter, self).__init__()
Expand Down Expand Up @@ -76,17 +100,12 @@ def log_record_to_dict(cls, record):
return output

def format(self, record):
def safe_str(object, *args, **kwargs):
"""Convert object to str, catching any errors raised."""
try:
return str(object, *args, **kwargs)
except:
return "<unprintable %s object>" % type(object).__name__

return json.dumps(self.log_record_to_dict(record), default=safe_str, separators=(",", ":"))
return json.dumps(self.log_record_to_dict(record), default=safe_json_encode, separators=(",", ":"))


class NewRelicLogForwardingHandler(logging.Handler):
DEFAULT_LOG_RECORD_KEYS = frozenset(set(vars(LogRecord("", 0, "", 0, "", (), None))) | {"message"})

def emit(self, record):
try:
# Avoid getting local log decorated message
Expand All @@ -95,10 +114,20 @@ def emit(self, record):
else:
message = record.getMessage()

record_log_event(message, record.levelname, int(record.created * 1000))
attrs = self.filter_record_attributes(record)
record_log_event(message, record.levelname, int(record.created * 1000), attributes=attrs)
except Exception:
self.handleError(record)

@classmethod
def filter_record_attributes(cls, record):
record_attrs = vars(record)
DEFAULT_LOG_RECORD_KEYS = cls.DEFAULT_LOG_RECORD_KEYS
if len(record_attrs) > len(DEFAULT_LOG_RECORD_KEYS):
return {k: v for k, v in six.iteritems(vars(record)) if k not in DEFAULT_LOG_RECORD_KEYS}
else:
return None


class NewRelicLogHandler(logging.Handler):
"""
Expand Down Expand Up @@ -126,8 +155,8 @@ def __init__(
"The contributed NewRelicLogHandler has been superseded by automatic instrumentation for "
"logging in the standard lib. If for some reason you need to manually configure a handler, "
"please use newrelic.api.log.NewRelicLogForwardingHandler to take advantage of all the "
"features included in application log forwarding such as proper batching.",
DeprecationWarning
"features included in application log forwarding such as proper batching.",
DeprecationWarning,
)
super(NewRelicLogHandler, self).__init__(level=level)
self.license_key = license_key or self.settings.license_key
Expand Down
65 changes: 55 additions & 10 deletions newrelic/api/transaction.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
create_attributes,
create_user_attributes,
process_user_attribute,
resolve_logging_context_attributes,
truncate,
)
from newrelic.core.attribute_filter import (
Expand Down Expand Up @@ -1524,7 +1525,7 @@ def set_transaction_name(self, name, group=None, priority=None):
self._group = group
self._name = name

def record_log_event(self, message, level=None, timestamp=None, priority=None):
def record_log_event(self, message, level=None, timestamp=None, attributes=None, priority=None):
settings = self.settings
if not (
settings
Expand All @@ -1537,18 +1538,62 @@ def record_log_event(self, message, level=None, timestamp=None, priority=None):

timestamp = timestamp if timestamp is not None else time.time()
level = str(level) if level is not None else "UNKNOWN"
context_attributes = attributes # Name reassigned for clarity

if not message or message.isspace():
_logger.debug("record_log_event called where message was missing. No log event will be sent.")
return
# Unpack message and attributes from dict inputs
if isinstance(message, dict):
message_attributes = {k: v for k, v in message.items() if k != "message"}
message = message.get("message", "")
else:
message_attributes = None

if message is not None:
# Coerce message into a string type
if not isinstance(message, six.string_types):
try:
message = str(message)
except Exception:
# Exit early for invalid message type after unpacking
_logger.debug(
"record_log_event called where message could not be converted to a string type. No log event will be sent."
)
return

# Truncate the now unpacked and string converted message
message = truncate(message, MAX_LOG_MESSAGE_LENGTH)

# Collect attributes from linking metadata, context data, and message attributes
collected_attributes = {}
if settings and settings.application_logging.forwarding.context_data.enabled:
if context_attributes:
context_attributes = resolve_logging_context_attributes(
context_attributes, settings.attribute_filter, "context."
)
if context_attributes:
collected_attributes.update(context_attributes)

if message_attributes:
message_attributes = resolve_logging_context_attributes(
message_attributes, settings.attribute_filter, "message."
)
if message_attributes:
collected_attributes.update(message_attributes)

# Exit early if no message or attributes found after filtering
if (not message or message.isspace()) and not context_attributes and not message_attributes:
_logger.debug(
"record_log_event called where no message and no attributes were found. No log event will be sent."
)
return

message = truncate(message, MAX_LOG_MESSAGE_LENGTH)
# Finally, add in linking attributes after checking that there is a valid message or at least 1 attribute
collected_attributes.update(get_linking_metadata())

event = LogEventNode(
timestamp=timestamp,
level=level,
message=message,
attributes=get_linking_metadata(),
attributes=collected_attributes,
)

self._log_events.add(event, priority=priority)
Expand Down Expand Up @@ -2062,7 +2107,7 @@ def record_ml_event(event_type, params, application=None):
application.record_ml_event(event_type, params)


def record_log_event(message, level=None, timestamp=None, application=None, priority=None):
def record_log_event(message, level=None, timestamp=None, attributes=None, application=None, priority=None):
"""Record a log event.
Args:
Expand All @@ -2073,12 +2118,12 @@ def record_log_event(message, level=None, timestamp=None, application=None, prio
if application is None:
transaction = current_transaction()
if transaction:
transaction.record_log_event(message, level, timestamp)
transaction.record_log_event(message, level, timestamp, attributes=attributes)
else:
application = application_instance(activate=False)

if application and application.enabled:
application.record_log_event(message, level, timestamp, priority=priority)
application.record_log_event(message, level, timestamp, attributes=attributes, priority=priority)
else:
_logger.debug(
"record_log_event has been called but no transaction or application was running. As a result, "
Expand All @@ -2089,7 +2134,7 @@ def record_log_event(message, level=None, timestamp=None, application=None, prio
timestamp,
)
elif application.enabled:
application.record_log_event(message, level, timestamp, priority=priority)
application.record_log_event(message, level, timestamp, attributes=attributes, priority=priority)


def accept_distributed_trace_payload(payload, transport_type="HTTP"):
Expand Down
Loading

0 comments on commit aebf47f

Please sign in to comment.