Incorrect error message for "RESOURCE_OPERATION_RATE_EXCEEDED" error

[m-strzelczyk]

Environment details

• OS type and version: Linux
• Python version: python --version: 3.7
• pip version: pip --version:
• google-cloud-compute version: pip show google-cloud-compute: main branch, it's happening in our automated tests.

Steps to reproduce

1. Run test samples tests in test_images.py
2. Sometimes you will get an error, for example in this build: https://source.cloud.google.com/results/invocations/f54330fa-48d1-4d96-8372-183e481afd50/targets/cloud-devrel%2Fclient-libraries%2Fpython%2Fgoogleapis%2Fpython-compute%2Fsamples%2Fpython3.7%2Fpresubmit/log
3. The error reported by the library is: google.api_core.exceptions.Forbidden: 403 FORBIDDEN
4. The error I find in the Cloud Logging is: RESOURCE_OPERATION_RATE_EXCEEDED and in human terms: Too frequent operations from the source resource. See: img_error.txt

Code example

In samples/snippets/tests/test_images.py

Stack trace

=================================== FAILURES ===================================
____________________________ test_image_from_image _____________________________

autodelete_image_name = 'test-img-8ea88455df'

    def test_image_from_image(autodelete_image_name):
        src_img = get_image_from_family('debian-cloud', 'debian-11')
        new_image = create_image_from_image(PROJECT, src_img.name, autodelete_image_name, 'debian-cloud',
                                            guest_os_features=[compute_v1.GuestOsFeature.Type.MULTI_IP_SUBNET.name],
>                                           storage_location='eu')

snippets/tests/test_images.py:121:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
snippets/images/create_from_image.py:122: in create_image_from_image
    wait_for_extended_operation(operation, "image creation from image")
snippets/images/create_from_image.py:58: in wait_for_extended_operation
    result = operation.result(timeout=timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <google.cloud.compute_v1.services.images.client.ImagesClient.insert.<locals>._CustomOperation object at 0x7f720ca795d0>
timeout = 300, retry = <google.api_core.retry.Retry object at 0x7f72094dec10>

    def result(self, timeout=None, retry=DEFAULT_RETRY):
        """Get the result of the operation, blocking if necessary.

        Args:
            timeout (int):
                How long (in seconds) to wait for the operation to complete.
                If None, wait indefinitely.

        Returns:
            google.protobuf.Message: The Operation's result.

        Raises:
            google.api_core.GoogleAPICallError: If the operation errors or if
                the timeout is reached before the operation completes.
        """
        kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
        self._blocking_poll(timeout=timeout, **kwargs)

        if self._exception is not None:
            # pylint: disable=raising-bad-type
            # Pylint doesn't recognize that this is valid in this case.
>           raise self._exception
E           google.api_core.exceptions.Forbidden: 403 FORBIDDEN

.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py:137: Forbidden

[vam-google]

@m-strzelczyk The text log you provided (img_error.txt) - that has json response result in it. From which specific call is that?

I'm asking because there are basically two "ways" to get an error:

1. on a protocol-level - HTTP response as it is (i.e. "raw" error), which youare getting as 403
2. api-specific way, which comes in an api-specific response message (which looks like what you have in img_error.txt.

To me it seems that the actual returned error response is correct (this is what is returned by compute). The error details in img_error.txt are from audit log, not compute response. WE can return extra details for compute call only if that message is actually returned by compute call itself. Is it returned in some form together with the 403 error? If it is, then in which form and how is it encoded in the response message? I was under impression that our library in addition to outputing the HTTP error code would also dump the rest of the payload of the response (unprocessed, but it will let a user to read the json dump and reason about it).

[vchudnov-g]

We have a similar issue in gapic-generator-python#1408, though it's not clear it's the same root cause, since there it manifests as an exception rather than an uninformative error code.

It probably makes sense to solve that one first, and then see whether the solution addresses this one, which sounds trickier to reproduce.

[parthea]

I'm going to close this as duplicate of googleapis/gapic-generator-python#1408. I also haven't been able to reproduce the issue.