PubSub: Subscriber keeps failing with 503

[tameem92]

Hey guys,

I am seeing the pubsub client receive a 503 unavailable and instead of retrying it just exits. And it can't receive any more messages. The subscriber starts and works for a few minutes before throwing the error. I can reproduce it consistently with multiple different pubsub subscriptions.

Package: google-cloud-pubsub==1.4.2
OS: Mac Catalina, also tried on an Ubuntu VM.
Python: Python 3.7.3

Here is an example test case where I see the issue followed by a stack trace:

import os
import json
from time import sleep
from datetime import datetime
import logging
from google.cloud import pubsub_v1
from config import Config

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)

def _callback(message):
    try:
        print('message', message)
    except Exception:
        print('exception')

if __name__ == '__main__':

    subscription = pubsub_v1.SubscriberClient()
    subscription_path = subscription.subscription_path(
        Config.PROJECT_ID,
        Config.PUBSUB_SUB_NAME)


    future = subscription.subscribe(
        subscription_path,
        _callback,
        pubsub_v1.types.FlowControl(max_messages=Config.MAX_MESSAGES))

    try:
        future.result()
    
    except KeyboardInterrupt:
        future.cancel()
        logging.error("Keyboard Interrupt")
    
    except Exception:
        raise

And stack trace -

DEBUG:google.cloud.pubsub_v1.subscriber._protocol.leaser:Snoozing lease management for 8.538204 seconds.
DEBUG:google.api_core.bidi:Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.
Traceback (most recent call last):
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 144, in error_remapped_callable
    return _StreamingResponseIterator(result)
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 72, in __init__
    self._stored_first_result = six.next(self._wrapped)
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/grpc/_channel.py", line 416, in __next__
    return self._next()
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "The service was unable to fulfill your request. Please try again. [code=8a75]"
	debug_error_string = "{"created":"@1587006128.267570000","description":"Error received from peer ipv6:[]:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"The service was unable to fulfill your request. Please try again. [code=8a75]","grpc_status":14}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/bidi.py", line 637, in _thread_main
    self._bidi_rpc.open()
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/bidi.py", line 280, in open
    call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 1059, in streaming_pull
    requests, retry=retry, timeout=timeout, metadata=metadata
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 146, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]
INFO:google.api_core.bidi:Thread-ConsumeBidirectionalStream exiting
INFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting.
INFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting.

[tameem92]

Hey,

I just found this is happening only with google-api-core v1.17.0. Reverting back to v1.16.0 fixes the issue.

googleapis/python-api-core#25

[plamut]

Could it be related to #74? The new release of API core might have caused this one, too.

Edit: Based on the new comment, this indeed appears to be the case. #76 will fix this until the problem in grpcio is resolved.

Edit 2: I was not able to reproduce the problem exactly as described, but it's clear that grpcio==1.17.0 causes problems and we have two confirmations that reverting back to 1.16.0 fixes the problem.

[cyubachi]

Hey,

I just found this is happening only with google-api-core v1.17.0. Reverting back to v1.16.0 fixes the issue.

googleapis/python-api-core#25

Thanks!
Your solution helped me!