Fix 1063 on 4.x branch

[hsophie-sf]

Fixes #1063
Details on #1063 (comment)

First commit is a failing test which is fixed in second commit.

[hsophie-sf]

Unable to build successfully because of two issues, both unrelated to my changes:

1. Unable to build on python 2.7 because of lzma. The problem I'm getting is similar to: backports/lzma/_lzmamodule.c(115) : fatal error C1083: Cannot open include file: 'lzma.h': No such file or directory peterjc/backports.lzma#22

backports/lzma/_lzmamodule.c(32) : warning C4273: 'PyErr_NewExceptionWithDoc' : inconsistent dll linkage
C:\projects\kombu.tox\2.7\include\pyerrors.h(226) : see previous definition of 'PyErr_NewExceptionWithDoc'
backports/lzma/_lzmamodule.c(115) : fatal error C1083: Cannot open include file: 'lzma.h': No such file or directory

2. Flaky test t/unit/transport/test_redis.py::test_Redis.test_publish__consume fails when
   run via tox -r -e 3.7-linux-unit but passes when
   run via tox -r -e 3.7-linux-unit t/unit/transport/ or more specifically with filename, class, or test method name.

Seems like it's using the test_redis.Client object which doesn't have lock() and other methods implemented. Don't know enough about the code to fix it.

The error is:

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_Redis.test_publish__consume ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <t.unit.transport.test_redis.test_Redis object at 0x117317320>

    def test_publish__consume(self):
        connection = Connection(transport=Transport)
        channel = connection.channel()
        producer = Producer(channel, self.exchange, routing_key='test_Redis')
        consumer = Consumer(channel, queues=[self.queue])
    
        producer.publish({'hello2': 'world2'})
        _received = []
    
        def callback(message_data, message):
            _received.append(message_data)
            message.ack()
    
        consumer.register_callback(callback)
        consumer.consume()
    
        assert channel in channel.connection.cycle._channels
        try:
>           connection.drain_events(timeout=1)

t/unit/transport/test_redis.py:971: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
kombu/connection.py:324: in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
kombu/transport/virtual/base.py:963: in drain_events
    get(self._deliver, timeout=timeout)
kombu/transport/redis.py:381: in get
    self.maybe_restore_messages()
kombu/transport/redis.py:340: in maybe_restore_messages
    num=channel.unacked_restore_limit,
kombu/transport/redis.py:197: in restore_visible
    self.unacked_mutex_expire):
../../.pyenv/versions/3.7.3/lib/python3.7/contextlib.py:112: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

client = <t.unit.transport.test_redis.Client object at 0x11733c080>, name = 'unacked_mutex', expire = 300

    @contextmanager
    def Mutex(client, name, expire):
        """Acquire redis lock in non blocking way.
    
        Raise MutexHeld if not successful.
        """
>       lock = client.lock(name, timeout=expire)
E       AttributeError: 'Client' object has no attribute 'lock'

kombu/transport/redis.py:114: AttributeError

 t/unit/transport/test_redis.py::test_Redis.test_publish__consume

[thedrow]

So why are you backporting this?
Is this not relevant for master?

[hsophie-sf]

So why are you backporting this?
Is this not relevant for master?

We just upgraded to the 4.x branch of Celery and noticed this issue. Since 4.x is LTS, we want it fixed there.
I can also submit an additional PR against master, but a fix on 4.x addresses my needs as well as others on that branch.

[lambacck]

@thedrow you locked #1063 ( #1063 (comment) ) and asked for a PR that reproduces and fixes it. But the PR already exists and is this one.

[auvipy]

in any case this should target master branch. in kombu we do not have proper 4.6.x branch like in celery

[thedrow]

So what exactly happens here?
Why reporting this error is good enough?

[hsophie-sf]

We're not raising the error if it's got a certain message.

[thedrow]

in any case this should target master branch. in kombu we do not have proper 4.6.x branch like in celery

@hsophie-sf You should target master anyway. I've done so for you.
We will backport this fix ourselves.

[thedrow]

I'm pretty convinced that this is the wrong place to fix this. @lambacck Your analysis is 100% correct though.
This is a good workaround but the actual fix should be in the Redis transport as it only happens there right?
Also, I don't understand why this error happens yet.

[hsophie-sf]

CI is failing due to unrelated reason now: https://github.com/hsophie-sf/kombu/runs/2667819031?check_suite_focus=true on test t/unit/transport/test_redis.py::test_Redis::test_publish__consume as noted before

[auvipy]

@matusvalo

[auvipy]

please rebase

[matusvalo]

I agree with @thedrow feedback:

I'm pretty convinced that this is the wrong place to fix this. @lambacck Your analysis is 100% correct though.
This is a good workaround but the actual fix should be in the Redis transport as it only happens there right?
Also, I don't understand why this error happens yet.

Moreover, I miss kind of warning when the exception is ignored.

[matusvalo]

Closing in favor of #1394 and #1404

Users affected by #1063 can do hotfix of their environment based on following comment:

#1063 (comment)

[hsophie-sf]

Closing in favor of #1394 and #1404

Users affected by #1063 can do hotfix of their environment based on following comment:

#1063 (comment)

Thanks. What release will the fixes for 1394 and 1404 be available in?

[auvipy]

5.2.0