AttributeError: 'int' object has no attribute 'endswith'

[thenewguy]

Seeing this traceback:

[2020-06-12 12:49:04,535: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'int' object has no attribute 'endswith'",)
 Traceback (most recent call last):
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/worker/worker.py", line 208, in start
     self.blueprint.start(self)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
     step.start(parent)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
     return self.obj.start()
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
     blueprint.start(self)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
     step.start(parent)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 599, in start
     c.loop(*c.loop_args())
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/celery/worker/loops.py", line 83, in asynloop
     next(loop)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 364, in create_loop
     cb(*cbargs)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/kombu/transport/redis.py", line 1083, in on_readable
     self.cycle.on_readable(fileno)
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/kombu/transport/redis.py", line 354, in on_readable
     chan.handlers[type]()
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/kombu/transport/redis.py", line 688, in _receive
     ret.append(self._receive_one(c))
   File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/kombu/transport/redis.py", line 704, in _receive_one
     if bytes_to_str(payload['type']).endswith('message'):
 AttributeError: 'int' object has no attribute 'endswith'

Versions:
kombu-4.6.10
celery-4.4.5

[thedrow]

How do I reproduce this error?

[kaiix]

We just caught the same exception, and luckily it had a sentry log.

It looks like in some cases redis-py pubsub leaks part of the health check response (redis-py/client.py at 3.5.3 · andymccurdy/redis-py) to the caller.

# https://github.com/celery/kombu/blob/v5.1.0/kombu/transport/redis.py#L713
r = b'redis-py-health-check'
type, pattern, channel, data = r[0], None, r[1], r[2]
# type == r[0] == 114
# channel == r[1] == 101
# data == r[2] == 100

the complete health check response should be [b'PONG', b'redis-py-health-check']

I think for this issue, check whether the response type is a list should fix the problem and also makes sense. (although the root cause comes from redis-py/hiredis)

---

Versions:

• kombu==5.1.0
• redis==3.5.3
• hiredis==2.0.0