AttributionError raised When Google Ads Server Error Occurred

[holyachon]

Describe the bug:

AttributeError: 'NoneType' object has no attribute 'Call'	

Stacktrace

status = StatusCode.INTERNAL
details = "Internal error encountered....
grpc._channel._SingleThreadedRendezvous: <_SingleThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered."
debug_error_string = "    "created": "@1638903791.733258205",
    "description": "Error received from peer ipv4:216.58.197.202:443",
    "file": "src/core/lib/surface/call.cc",
    "file_line": 1069,
    "grpc_message": "Internal error encountered.",
    "grpc_status": 13
}
"
debug_error_string = "{"created":"@1638903791.733258205","description":"Error received from peer ipv4:216.58.197.202:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Internal error encountered.","grpc_status":13}"	


File "/var/task/google/ads/googleads/v9/services/services/google_ads_service/client.py", line 3282, in search_stream
response = rpc(
└ <google.api_core.gapic_v1.method._GapicCallable object at 0x7fb798f9a430>
File "/var/task/google/api_core/gapic_v1/method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
│             │       └ {'metadata': [('x-goog-request-params', 'customer_id=**MASKING**'), ('x-goog-api-client', 'gl-python/3.8.11 grpc/1.41.1 gax/2....
│             └ (customer_id: "**MASKING**"
│               query: "SELECT \n            customer.id,\n            customer.descriptive_name,\n            cus...
└ <function _wrap_stream_errors.<locals>.error_remapped_callable at 0x7fb798f9eaf0>
File "/var/task/google/api_core/grpc_helpers.py", line 160, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
│          └ <function from_grpc_error at 0x7fb79d25a700>
└ <module 'google.api_core.exceptions' from '/var/task/google/api_core/exceptions.py'>
File "/var/task/google/api_core/exceptions.py", line 532, in from_grpc_error
if isinstance(rpc_exc, grpc.Call) or _is_informative_grpc_error(rpc_exc):
│        │             │                          └ <_SingleThreadedRendezvous of RPC that terminated with:
│        │             │                            	status = StatusCode.INTERNAL
│        │             │                            	details = "Internal error encountered....
│        │             └ <function _is_informative_grpc_error at 0x7fb79d25a5e0>
│        └ None
└ <_SingleThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered....
AttributeError: 'NoneType' object has no attribute 'Call'	

(sorry for formatting)

Steps to Reproduce:
it is not easy to reproduce because it needs server error to reproduce the bug.
And as a rule of thumb, not all server errors raise AttributeError. In some cases, ServerError exception is normally raised.

Expected behavior:
ServerError exception raised

Client library version and API version:
Client library version: 14.1.0 (latest)
Google Ads API version: v9

aiohttp @ file:///Users/sym/Library/Caches/pypoetry/artifacts/05/07/33/33717ed661d3ff4fefae447a53297894172e5164474c91063085d43330/aiohttp-3.8.0-cp38-cp38-macosx_10_9_x86_64.whl
aiosignal @ file:///Users/sym/Library/Caches/pypoetry/artifacts/4c/98/1b/ee90d29611d2aaeb4a2b40c1509ca578cccf552d05a53563f4f8bf9bb3/aiosignal-1.2.0-py3-none-any.whl
appdirs @ file:///Users/sym/Library/Caches/pypoetry/artifacts/68/c7/33/ef0f6af8d1aa906f1a5641762fe5d6f47d565c5a4b3482f1c6fb0f5f20/appdirs-1.4.4-py2.py3-none-any.whl
argcomplete @ file:///Users/sym/Library/Caches/pypoetry/artifacts/96/70/57/5875e2d640c712de1319caa20ce7168ed5a211a3c87554b04e123b347b/argcomplete-1.12.3-py2.py3-none-any.whl
arrow @ file:///Users/sym/Library/Caches/pypoetry/artifacts/27/b2/c5/96652e26f28b571e8c677545083f3d8e295914b90344709c00f7ad2ca8/arrow-1.1.1-py3-none-any.whl
async-timeout @ file:///Users/sym/Library/Caches/pypoetry/artifacts/84/cd/e4/c9346fd7aaa2b90be31afd8151ca9fc849baf4f1dc93714e23202c2a92/async_timeout-4.0.0-py3-none-any.whl
attrs @ file:///Users/sym/Library/Caches/pypoetry/artifacts/1a/aa/39/10d6d07084f186f8cf6963cb033440402ad5088bb94d712239170f2ef6/attrs-21.2.0-py2.py3-none-any.whl
aws-embedded-metrics @ file:///Users/sym/Library/Caches/pypoetry/artifacts/cc/61/7c/f721bc7db7a765043ccf5491a0b0af4b2aa8d843e2d260cd0746104fa4/aws_embedded_metrics-1.0.7-py3-none-any.whl
backports.entry-points-selectable @ file:///Users/sym/Library/Caches/pypoetry/artifacts/87/38/bc/010a813c45cba13501d5134886066f1a3dfe52f3ae0fe28996a8035dcf/backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl
black @ file:///Users/sym/Library/Caches/pypoetry/artifacts/38/f4/61/550fd050005dc1c64850374a7f98dd941a4633d72e87f147bebf26bdc3/black-20.8b1.tar.gz
boto3 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ae/4a/f7/bec32731294b3f6bc791e2947b1015d0cb28c7d4cd505e65c812abbc9a/boto3-1.16.63-py2.py3-none-any.whl
botocore @ file:///Users/sym/Library/Caches/pypoetry/artifacts/58/6e/f5/be863909d3ea80cc6665ccf7b99324ab0d8a08681ecea2815149f35fd2/botocore-1.19.63-py2.py3-none-any.whl
cachetools @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ca/68/fb/a94522dd8c8d081803f16d6ec74ce9a678a827247431e2a8339f25506f/cachetools-4.2.4-py3-none-any.whl
certifi @ file:///Users/sym/Library/Caches/pypoetry/artifacts/77/74/3e/daf57610ff13227e3b3f50a78487055f6ac94cc9363abd2463be34c5e9/certifi-2021.10.8-py2.py3-none-any.whl
cffi @ file:///Users/sym/Library/Caches/pypoetry/artifacts/e6/3a/ed/211634ffd1f9ed0a6957a6981ef59f3af9934e62dc2ee15819d9623196/cffi-1.15.0-cp38-cp38-macosx_10_9_x86_64.whl
chardet @ file:///Users/sym/Library/Caches/pypoetry/artifacts/47/b7/82/19c2b887f87f3adbaf4e34c55189388e5132c78f6929d7001a78b0209b/chardet-4.0.0-py2.py3-none-any.whl
charset-normalizer @ file:///Users/sym/Library/Caches/pypoetry/artifacts/23/43/96/9c5bffd9765f723eb96925cec608e36125667e8f31b6b3152f0b51716f/charset_normalizer-2.0.7-py3-none-any.whl
click @ file:///Users/sym/Library/Caches/pypoetry/artifacts/3c/f6/a3/2262df8f5a6f3de5bbc78cc13803c60524c8384dce76661fdfe3df975f/click-8.0.3-py3-none-any.whl
colorlog @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f5/a7/3e/1f1413ec92fc0a7c37b13f3d879599c82cd789f81c76e18089b6925b80/colorlog-6.6.0-py2.py3-none-any.whl
coverage @ file:///Users/sym/Library/Caches/pypoetry/artifacts/0e/17/32/a937f84a598e3a99f4e437f42df4b8e891e02c39aeefca49fdee9e4c86/coverage-6.1.1-cp38-cp38-macosx_10_9_x86_64.whl
cryptography @ file:///Users/sym/Library/Caches/pypoetry/artifacts/65/42/84/99e7508590fbc4796163f2f564ab4caf7c7a512127f55f9e0ef5b96d6d/cryptography-3.4.8-cp36-abi3-macosx_10_10_x86_64.whl
distlib @ file:///Users/sym/Library/Caches/pypoetry/artifacts/03/f1/f5/36347a1de773fbd629afb90a04cd682421c383092f800cd4b0ee90379a/distlib-0.3.3-py2.py3-none-any.whl
filelock @ file:///Users/sym/Library/Caches/pypoetry/artifacts/1c/30/62/c8f5fef72567efad2e65fd88a591a98c265774421bbd6a8e8ca4424c9b/filelock-3.3.2-py3-none-any.whl
frozenlist @ file:///Users/sym/Library/Caches/pypoetry/artifacts/c7/97/b2/2f3e0bc769592d27f0ff9dfa1d5c4d666a8e19a0ff758560892b13cab8/frozenlist-1.2.0-cp38-cp38-macosx_10_9_x86_64.whl
google-ads @ file:///Users/sym/Library/Caches/pypoetry/artifacts/be/74/8c/68441760850e611b65470fbf5cb0b23856284bbd3023e4ff9a231192f8/google_ads-14.1.0-py3-none-any.whl
google-api-core @ file:///Users/sym/Library/Caches/pypoetry/artifacts/6c/9f/cb/f29b967235fcf45553f3707e3b4075af2d12f393b250e9262e595cbc8f/google_api_core-2.2.2-py2.py3-none-any.whl
google-auth @ file:///Users/sym/Library/Caches/pypoetry/artifacts/7f/fe/8c/3b0b80802d984bbf73e991482e6e88c196d8deb2fd45306483de7b67e8/google_auth-1.35.0-py2.py3-none-any.whl
google-auth-oauthlib @ file:///Users/sym/Library/Caches/pypoetry/artifacts/35/f1/0a/8a2ac196dbf26e110b0c838bbbaeb3707a1475bae0e456bdf49b0dbb7b/google_auth_oauthlib-0.4.6-py2.py3-none-any.whl
googleapis-common-protos @ file:///Users/sym/Library/Caches/pypoetry/artifacts/8c/2e/ad/efdce99c62dbe56fda156dd62f807ea79a43f83dea45bd9f9ff4bc6240/googleapis_common_protos-1.53.0-py2.py3-none-any.whl
grpcio @ file:///Users/sym/Library/Caches/pypoetry/artifacts/7f/26/38/939f4ababe22e4d92e9206b0514c35e0803a063c66da76c941caca8c02/grpcio-1.41.1-cp38-cp38-macosx_10_10_x86_64.whl
idna @ file:///Users/sym/Library/Caches/pypoetry/artifacts/9e/f9/03/6066b92d35486e7f0d4f310126e3c60a6619726e43ef98e32e105b5c52/idna-2.10-py2.py3-none-any.whl
iniconfig @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ac/f9/da/e990ffcd9ec361a68676a5916e391286e1ea5d1b8907ae887e141a71f5/iniconfig-1.1.1-py2.py3-none-any.whl
jmespath @ file:///Users/sym/Library/Caches/pypoetry/artifacts/43/d5/a2/f83573231324de7f5b61f5c607fbbe82ca535359a452de4852d2e25e8d/jmespath-0.10.0-py2.py3-none-any.whl
loguru @ file:///Users/sym/Library/Caches/pypoetry/artifacts/d0/16/90/b85b179e5e32051e734faa1aea1335c15d5c74447de12006487c70f02d/loguru-0.5.3-py3-none-any.whl
multidict @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ce/07/97/96c8c0778bcefb790a0e0509f8f3114fc02f75a074477cac59c5174797/multidict-5.2.0-cp38-cp38-macosx_10_9_x86_64.whl
mypy-extensions @ file:///Users/sym/Library/Caches/pypoetry/artifacts/b6/a0/b0/a5dc9acd6fd12aba308634f21bb7cf0571448f20848797d7ecb327aa12/mypy_extensions-0.4.3-py2.py3-none-any.whl
nox @ file:///Users/sym/Library/Caches/pypoetry/artifacts/57/03/a0/2d9d2709981a7c7d46d9b81bc87f4487e266d5b5b7b6d26f1fdfbdc4ef/nox-2021.10.1-py3-none-any.whl
oauthlib @ file:///Users/sym/Library/Caches/pypoetry/artifacts/e6/64/ef/a43b939f04c9ee17174fb3657dba149ed15ba473ef44adf939f5032b85/oauthlib-3.1.1-py2.py3-none-any.whl
packaging @ file:///Users/sym/Library/Caches/pypoetry/artifacts/97/a9/40/465a52d3ec4e9927b21488ac9bf7c5fa59e9ee44c337de47d197b804e4/packaging-21.2-py3-none-any.whl
pathspec @ file:///Users/sym/Library/Caches/pypoetry/artifacts/04/91/14/294ed2ee6c852b0d466bdd15d393127eff4168b35ae81cedf5a03fe348/pathspec-0.9.0-py2.py3-none-any.whl
pep562 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/e1/15/52/2820b854d72fc1a2f0816aae9ee4f803435ffcdae9a07aa30760afebdc/pep562-1.1-py2.py3-none-any.whl
platformdirs @ file:///Users/sym/Library/Caches/pypoetry/artifacts/c2/ba/64/b6513f98bf6524a0ec2316e6f9326829bf25522e915b9965b0e55b969b/platformdirs-2.4.0-py3-none-any.whl
pluggy @ file:///Users/sym/Library/Caches/pypoetry/artifacts/38/05/d7/70e2b0d553c780097b692ed658b0f553a28f16028d848a98174b1a2249/pluggy-1.0.0-py2.py3-none-any.whl
proto-plus @ file:///Users/sym/Library/Caches/pypoetry/artifacts/72/27/84/cb70b1de7be5e0fcb6fbf2f83f0b2dcf6b7c269401c84348de2446be52/proto_plus-1.18.1-py3-none-any.whl
protobuf @ file:///Users/sym/Library/Caches/pypoetry/artifacts/da/16/58/39025383d35aa8857945fb78e0716001e527e7dee801db8994234468da/protobuf-3.17.3-cp38-cp38-macosx_10_9_x86_64.whl
py @ file:///Users/sym/Library/Caches/pypoetry/artifacts/6b/b2/2b/e6686e7d0183dbd36bd66921efa3e77ce26260a3671524cd86614290e0/py-1.10.0-py2.py3-none-any.whl
pyasn1 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/a6/f7/5e/59a43ec23ade0888631b3f24244da7f5d5a0b6b40849c86b8c6b4c54d1/pyasn1-0.4.8-py2.py3-none-any.whl
pyasn1-modules @ file:///Users/sym/Library/Caches/pypoetry/artifacts/09/f6/56/0e33158234b3e9b9e5d8eb01e7a96670a99ab43b3dbe89b0f129477a59/pyasn1_modules-0.2.8-py2.py3-none-any.whl
pycparser @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f1/03/25/40eb46f7bede64f78ba073e2141b8216e611cbcde72e3117c326560101/pycparser-2.20-py2.py3-none-any.whl
pydantic @ file:///Users/sym/Library/Caches/pypoetry/artifacts/af/47/17/f8f37bb7abd91fed95dd4126981e6192c373440fdb36b85365c1917ac8/pydantic-1.7.4-cp38-cp38-macosx_10_9_x86_64.whl
PyJWT @ file:///Users/sym/Library/Caches/pypoetry/artifacts/7c/13/04/3e4aa78f6ee3ee721315e24346fe7226d1fc7330485447a1ce5d1151c7/PyJWT-2.3.0-py3-none-any.whl
PyMySQL @ file:///Users/sym/Library/Caches/pypoetry/artifacts/95/eb/3e/086ab3187a3284af82f8239d098217d060bca74ce9da2651bc5576a9c5/PyMySQL-0.10.1-py2.py3-none-any.whl
pyparsing @ file:///Users/sym/Library/Caches/pypoetry/artifacts/da/e7/3d/1780282f558e5fd157bf708b28b8ba0d08323ef6bc5b6396139ce38a0b/pyparsing-2.4.7-py2.py3-none-any.whl
pytest @ file:///Users/sym/Library/Caches/pypoetry/artifacts/90/e1/d5/985c1b371486cf5749ce1ec1beac04524dacb0642a52d916fd1aa82671/pytest-6.2.5-py3-none-any.whl
pytest-cov @ file:///Users/sym/Library/Caches/pypoetry/artifacts/b9/58/55/db0439b8e3e9c6ad7790a801ddb71803fd3791b5d6fd7f65efe27016f5/pytest_cov-2.10.1-py2.py3-none-any.whl
pytest-mock @ file:///Users/sym/Library/Caches/pypoetry/artifacts/30/d9/c2/bb39c4b3c8494e50e2dfa5a17991686a54be0256d6a30893fe796dc9fe/pytest_mock-3.4.0-py3-none-any.whl
python-dateutil @ file:///Users/sym/Library/Caches/pypoetry/artifacts/0f/f1/e2/e1d3399cea26388e2ed5b93ea3e7c137d2b4027c5ba14c64ab839294ed/python_dateutil-2.8.2-py2.py3-none-any.whl
pytz @ file:///Users/sym/Library/Caches/pypoetry/artifacts/07/c1/43/6f22a533e7bc5e277f10b4c6033e936ff7bbeb6464e5f27dbfb6c641d7/pytz-2021.3-py2.py3-none-any.whl
PyYAML @ file:///Users/sym/Library/Caches/pypoetry/artifacts/99/c6/60/2baceb8eb743927ffda4f0184e9aea5339ca72c9a463509feeeb0d1668/PyYAML-5.4.1-cp38-cp38-macosx_10_9_x86_64.whl
regex @ file:///Users/sym/Library/Caches/pypoetry/artifacts/d7/6f/34/ca74ab99f0547627cd097e864e92476bc96b3286da62684dd5b02dddbf/regex-2021.11.2-cp38-cp38-macosx_10_9_x86_64.whl
requests @ file:///Users/sym/Library/Caches/pypoetry/artifacts/fe/f0/74/d691c7619cd645682c495ff78949b9c284e88d3f2ee85124df5cec5b0a/requests-2.25.1-py2.py3-none-any.whl





tenacity @ file:///Users/sym/Library/Caches/pypoetry/artifacts/44/71/21/397279f8fca199d048595a0942e6b22859b07f13d4e7496ddf932be65d/tenacity-8.0.1-py3-none-any.whl
toml @ file:///Users/sym/Library/Caches/pypoetry/artifacts/9f/8c/c3/7b4f6778d60f3c9fa11f8fd0e48243bbad25a04975e0a01006b6350594/toml-0.10.2-py2.py3-none-any.whl.3.0-py2.py3-none-any.whl
typed-ast @ file:///Users/sym/Library/Caches/pypoetry/artifacts/2a/97/67/6a4e444fede37f31a09025fea88c54d4a40d739c4640e72d4c8d355661/typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl
typing-extensions @ file:///Users/sym/Library/Caches/pypoetry/artifacts/63/00/66/294e7d75d34398c27c7ed9b154822f99422b0f11548b11c569c188eb2b/typing_extensions-3.10.0.2-py3-none-any.whl
urllib3 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/d8/12/e2/1b499db5e41e88fd0f31d0e056a4dc7b53824f63f027fffc1702969c57/urllib3-1.26.7-py2.py3-none-any.whly.whl
virtualenv @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f0/72/e5/03355e512a823b9fe1f8905fca1d21e958e08ee2501c03106460ae69a1/virtualenv-20.10.0-py2.py3-none-any.whl
yarl @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f1/71/2d/1439e8877c108b7d845e8e4199efa1fc8c8f30c3d25cdaa912722eba93/yarl-1.7.2-cp38-cp38-macosx_10_9_x86_64.whl_14_x86_64.whl

Request/Response Logs:

[WARNING] 2021-12-07T19:03:11.733Z e853bb65-dd7d-5b95-9b9c-e14c6f59508a Request made: ClientCustomerId: **MASKING**, Host: googleads.googleapis.com, Method: /google.ads.googleads.v9.services.GoogleAdsService/SearchStream, RequestId: wPV3zIKhwoN0VcBuXFLZxA, IsFault: True, FaultMessage: Internal error encountered.	

Anything else we should know about your project / environment:

[BenRKarl]

@holyachon How often are you receiving INTERNAL errors for this query? Is it intermittent or consistent?

It looks like in your application google-api-core is having a problem importing the grpcio dependency. If you look at this line it sets grpc to None if it can't import it, then later here the error is thrown. Would it be possible for you to inspect this file at runtime, either with a breakpoint or print statement, to see what the state of the grpc library is when it's executed?

[holyachon]

@holyachon How often are you receiving INTERNAL errors for this query? Is it intermittent or consistent?

It looks like in your application google-api-core is having a problem importing the grpcio dependency. If you look at this line it sets grpc to None if it can't import it, then later here the error is thrown. Would it be possible for you to inspect this file at runtime, either with a breakpoint or print statement, to see what the state of the grpc library is when it's executed?

@BenRKarl Hello. Thank you for your response.

It seems to happen once or twice a day. I think adding a breakpoint is difficult because this error cannot be reproduced at my own will.
Then would it be helpful to log the state of grpc when an error occurs like the code below?

        try:
            response = self.ads_service.search_stream(self.stream_request)
        except AttributeError as e:
            from google.api_core.exceptions import grpc
            logger.info(grpc)

[androkon]

I think, this problem from this line, module grpc_status does not exists, and even if importing of module grpc is success, in except block the variable grpc becomes None.
May be, code must be as this?
from grpc import Status as rpc_status

[androkon]

It seems like helps explicit package setup
pip install grpcio-status

[holyachon]

It seems like helps explicit package setup pip install grpcio-status

I tried it and I think it works well. (because after install grpcio-status, I can reference rpc_status)
I'll have to take a closer look, but I think this dependency should be added to googleapis/python-api-core.
There seems to be a bit of a related issue. googleapis/python-api-core#301

Thank you for your help @androkon :)

[holyachon]

It seems like helps explicit package setup pip install grpcio-status

I'll have to take a closer look, but I think this dependency should be added to googleapis/python-api-core. There seems to be a bit of a related issue. googleapis/python-api-core#301

The Issue says grpcio-status is optional dependency in python-api-core.
So I think it should be added to google-ads-python.
May I ask what you think? @BenRKarl

[BenRKarl]

@holyachon thanks for working on this and sharing these details. I'm hesitant to make grpcio-status a dependency of this project because it should be resolved within python-api-core since that's the library where it's used, and from where the error originates. I'm going to close this issue and the related PR and will ping the maintainers of api core to try to find a resolution in their library via the issue here: googleapis/python-api-core#301

In the meantime I recommend the temporary workaround of manually installing grpcio-status.