Built-in TypeError exception from Connection on_inbound_method()

[juhamkilpinen]

There are several reports from the other projects getting this exception when using Connection class:

   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 472, in on_inbound_method
     return self.channels[channel_id].dispatch_method(
 TypeError: 'NoneType' object has no attribute '__getitem__'

See for example:
https://bugs.launchpad.net/oslo.messaging/+bug/1504725
celery/kombu#857
celery/kombu#463

Does this mean that Connection class is misused or should there be similar check in on_inbound_method() as there is in channel() for self.channels being None, for example:

def on_inbound_method(self, channel_id, method_sig, payload, content):
    if self.channels is None:
        raise RecoverableConnectionError('Connection already closed.')

I have seen this problem e.g. in OpenStack Pike when creating heat stacks and stack creation can fail for that TypeError exception. If I add above check it seems that client can re-connect and stack creation does not fail (when RecoverableConnectionError is thrown instead of TypeError).

[juhamkilpinen]

Sorry I cannot provide a simplified test case to point the issue. I think it is somehow related to caching the connections with multiple process / threads involved, I.e. some kind of misuse of the API. I am happy for now that my workaround seems to work with the OpenStack env.

[levsh]

Hi! I got the similar error.

  File "/opt/.../embedded/lib/python3.5/site-packages/celery/app/base.py", line 745, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/opt/.../embedded/lib/python3.5/site-packages/celery/app/amqp.py", line 552, in send_task_message
    **properties
  File "/opt/.../embedded/lib/python3.5/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/opt/.../embedded/lib/python3.5/site-packages/kombu/connection.py", line 494, in _ensured
    return fun(*args, **kwargs)
  File "/opt/.../embedded/lib/python3.5/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/opt/.../embedded/lib/python3.5/site-packages/amqp/channel.py", line 1743, in basic_publish_confirm
    self.wait(spec.Basic.Ack)
  File "/opt/../embedded/lib/python3.5/site-packages/amqp/abstract_channel.py", line 79, in wait
    self.connection.drain_events(timeout=timeout)
  File "/opt/.../embedded/lib/python3.5/site-packages/amqp/connection.py", line 491, in drain_events
    while not self.blocking_read(timeout):
  File "/opt/.../embedded/lib/python3.5/site-packages/amqp/connection.py", line 497, in blocking_read
    return self.on_inbound_frame(frame)
  File "/opt.../embedded/lib/python3.5/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/opt/.../embedded/lib/python3.5/site-packages/amqp/connection.py", line 500, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
TypeError: 'NoneType' object is not subscriptable

Python 3.5.5
Python libs with some minor patches
amqp (2.3.2)
kombu (4.2.1)
celery (4.2.1)

[4383]

I facing the same issue with openstack/oslo.messaging on queens/OSP13.

I can observe some client deconnection/reconnection just before this error will throw.

In my case, these errors seems related to vm creations and to heat too... I can't reproduce it merely outside of the openstack scope. I'll try to provide some minimalist POC to test it easily.

For further reading about this topic you can take a look to:

• https://bugs.launchpad.net/oslo.messaging/+bug/1781876
• https://bugs.launchpad.net/oslo.messaging/+bug/1504725
• Reset self.channels to emtpy dict instead of None when closing connection. #70
• When py-amqp tries to publish to nonexistent exchange crash with AttributeError: 'NoneType' object has no attribute 'drain_events' #218
• https://bugzilla.redhat.com/show_bug.cgi?id=1511041
• https://bugzilla.redhat.com/show_bug.cgi?id=1747226

[auvipy]

@4383 Thanks for the detailed report & fix!! can we chat in IRC about the usage of celery in OpenStack?

[4383]

@auvipy my pleasure.
Do not hesitate to ping me on IRC, I'm hberaud on freenode.

[Perepyolka]

@4383 @thedrow Hi guys, any updates on this bug? We have encountered this issue and thinking about updating our amqp to have this fix #289.

[4383]

@Perepyolka seems that my issue was fixed by #289