Random Atribute or Type errors when sending a message in local tests

[butla]

I have a local test that checks if we really do send a message to Sentry after an error in one place. Mountebank is impersonating the Sentry server in that test, I'm just checking that the server got a message from my AioHTTP service. The problem is that sometimes Raven fails to send the message. There are two possible errors (that I've observed):

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/raven/handlers/logging.py", line 97, in emit
    return self._emit(record)
  File "/usr/local/lib/python3.6/site-packages/raven/handlers/logging.py", line 190, in _emit
    **kwargs)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 641, in capture
    **kwargs)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 476, in build_msg
    data.update(processor.process(data))
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 33, in process
    self.filter_stacktrace(value['stacktrace'])
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 107, in filter_stacktrace
    frame['vars'] = varmap(self.sanitize, frame['vars'])
  File "/usr/local/lib/python3.6/site-packages/raven/utils/__init__.py", line 52, in varmap
    for k, v in iteritems(var))
  File "/usr/local/lib/python3.6/site-packages/raven/utils/__init__.py", line 52, in <genexpr>
    for k, v in iteritems(var))
  File "/usr/local/lib/python3.6/site-packages/raven/utils/__init__.py", line 49, in varmap
    ret = func(name, var)
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 182, in sanitize
    value = super(SanitizePasswordsProcessor, self).sanitize(item, value)
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 97, in sanitize
    for key in self.sanitize_keys:
  File "/usr/local/lib/python3.6/site-packages/raven/breadcrumbs.py", line 150, in processor
    data.update({
AttributeError: 'SanitizePasswordsProcessor' object has no attribute 'update'

and

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/raven/handlers/logging.py", line 97, in emit
    return self._emit(record)
  File "/usr/local/lib/python3.6/site-packages/raven/handlers/logging.py", line 190, in _emit
    **kwargs)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 641, in capture
    **kwargs)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 476, in build_msg
    data.update(processor.process(data))
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 33, in process
    self.filter_stacktrace(value['stacktrace'])
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 107, in filter_stacktrace
    frame['vars'] = varmap(self.sanitize, frame['vars'])
  File "/usr/local/lib/python3.6/site-packages/raven/utils/__init__.py", line 52, in varmap
    for k, v in iteritems(var))
  File "/usr/local/lib/python3.6/site-packages/raven/utils/__init__.py", line 52, in <genexpr>
    for k, v in iteritems(var))
  File "/usr/local/lib/python3.6/site-packages/raven/utils/__init__.py", line 49, in varmap
    ret = func(name, var)
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 182, in sanitize
    value = super(SanitizePasswordsProcessor, self).sanitize(item, value)
  File "/usr/local/lib/python3.6/site-packages/raven/processors.py", line 97, in sanitize
    for key in self.sanitize_keys:
TypeError: 'collections.OrderedDict' object is not callable

I'm setting up Raven this way:

    sentry_client = raven.Client(
        app_config.sentry_url,
        transport=functools.partial(
            raven_aiohttp.QueuedAioHttpTransport,
            workers=4,
            qsize=1000,
        ),
        environment=app_config.environment,
        site='api',
        release=app_config.version,
        # I tried to do that to disable breadcrumbs, since I saw that the error had something to do with that.
        hook_libraries=[],
    )
    sentry_handler = raven.handlers.logging.SentryHandler(sentry_client)
    sentry_handler.setLevel(logging.WARNING)
    root_logger = logging.getLogger()
    root_logger.addHandler(sentry_handler)

Like I said, the errors don't appear 100% of the time. On my beefy computer I saw that only two times, let's say in a 100. On the CI server (Jenkins), however, it happens around 50% of the times, so it's very annoying.

It's maybe worth knowing that because of the way I've set up my tests, the application's container comes alive before the fake Sentry server is alive, but it's definitely alive when the first error occurs in the application. Don't know if Raven tries to call home when the client is created and that might mess some stuff up.

Any suggestions about how to debug this or work around it?

[hellysmile]

From logs that I see it more related to raven itself, does it happens same with requests transport?

[butla]

@hellysmile I can check when I have some more time to play with this issue.