RetryErrors obscure actual errors - PubSub API

[ensonic]

Similar to #3083, #3900

2017-08-31T11:27:05.702393618Z   File "main.py", line 274, in start
2017-08-31T11:27:05.702404302Z     self.add_topic(topic, params)
2017-08-31T11:27:05.702413945Z   File "main.py", line 292, in add_topic
2017-08-31T11:27:05.702472288Z     self, self.robot_id, self.pubsub_client, self.network, topic, params)
2017-08-31T11:27:05.702501499Z   File "main.py", line 128, in __init__
2017-08-31T11:27:05.702572732Z     if not self.topic.exists():
2017-08-31T11:27:05.702583408Z   File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub/topic.py", line 200, in exists
2017-08-31T11:27:05.702634134Z     api.topic_get(topic_path=self.full_name)
2017-08-31T11:27:05.702713982Z   File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub/_gax.py", line 132, in topic_get
2017-08-31T11:27:05.702723634Z     topic_pb = self._gax_api.get_topic(topic_path)
2017-08-31T11:27:05.702729859Z   File "/usr/local/lib/python2.7/dist-packages/google/cloud/gapic/pubsub/v1/publisher_client.py", line 341, in get_topic
2017-08-31T11:27:05.702735747Z     return self._get_topic(request, options)
2017-08-31T11:27:05.702754679Z   File "/usr/local/lib/python2.7/dist-packages/google/gax/api_callable.py", line 428, in inner
2017-08-31T11:27:05.702802263Z     return api_caller(api_call, this_settings, request)
2017-08-31T11:27:05.70281738Z   File "/usr/local/lib/python2.7/dist-packages/google/gax/api_callable.py", line 416, in base_caller
2017-08-31T11:27:05.702884253Z     return api_call(*args)
2017-08-31T11:27:05.702898662Z   File "/usr/local/lib/python2.7/dist-packages/google/gax/api_callable.py", line 376, in inner
2017-08-31T11:27:05.703026405Z     return a_func(*args, **kwargs)
2017-08-31T11:27:05.703084222Z   File "/usr/local/lib/python2.7/dist-packages/google/gax/retry.py", line 127, in inner
2017-08-31T11:27:05.703115599Z     ' classified as transient', exception)
2017-08-31T11:27:05.703271957Z google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNIMPLEMENTED, unknown service google.pubsub.v1.Publisher)>)

[lukesneeringer]

@jonparrott Dropping your name on this since your fixes to google.api.core will fix it.

[jlewi]

I hit the same problem with the Cloud Spanner API.

Specifically I was getting an INVALID_ARGUMENT trying to create an instance but no useful information about what was invalid. I ran the same command using gcloud and got a very helpful error message.

ERROR: (gcloud.spanner.instances.create) INVALID_ARGUMENT: Invalid CreateInstance request.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: 'Display name must be between 4-30 characters long. Got: Spanner
      instance used by jlewi@ for TensorBoard.'
    field: instance.display_name

I'm guessing the problem is in the gRPC layer and so all APIs will be affected. Would it make sense to change the title of the issue?

[theacodes]

The new exception handling code will only raise a RetryError if retires retries are exhausted, otherwise, it will raise the real error.

Basically, this won't be an issue once we get gapic to use the new error code.

[robhaswell]

For near-future historians, this is how this programmer did meaningful error handling:

from google.gax.errors import RetryError
from grpc import StatusCode

try:
  leap_without_looking()
except RetryError as exc:
    if exc.cause.code() is not StatusCode.ALREADY_EXISTS:
        raise

[chemelnucfin]

@jonparrott May I ask if this is fixed now? I'm only asking because I think it might have been fixed...

[theacodes]

Yes gax is dead for pubsub