fix: populate default timeout and retry after client-side timeout

[tswast]

This addresses internal issue 195337762 where sometimes query job creation can
take longer than expected and retrying the API call can be faster than waiting
for the first query job request to fail.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #889
Towards #779

🦕

[tswast]

Removed because for most snippets except for the ones that explicitly use a service account key file, the credentials from gcloud work fine. Downloading service account keys is increasingly restricted on internal projects.

[tswast]

This is the exception I get if I set the timeout to too small of a value for most requests to complete.

[tseaver]

Do we need a test for this exception type in tests/unit/test_retry.py?

[tswast]

Good call. Added.

Looking at the Requests docs, it seems ReadTimeout is just one kind of Timeout error that can happen, so I updated this to the base class and added some tests for other timeouts.

[tswast]

Looks like a bunch of problems with load jobs in the system and sample tests:

capsys = <_pytest.capture.CaptureFixture object at 0x7f56cfd37320>
random_table_id = 'python-docs-samples-tests.python_bigquery_samples_snippets_20210824153103_865de1.python_bigquery_samples_snippets_20210824153103_c9ee5a'

    def test_load_table_uri_firestore(capsys, random_table_id):
>       load_table_uri_firestore.load_table_uri_firestore(random_table_id)

load_table_uri_firestore_test.py:19:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
load_table_uri_firestore.py:51: in load_table_uri_firestore
    load_job.result()  # Waits for the job to complete.
../../google/cloud/bigquery/job/base.py:708: in result
    return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <google.cloud.bigquery.job.load.LoadJob object at 0x7f56cfcd2710>
timeout = None, retry = <google.api_core.retry.Retry object at 0x7f56e6aefb38>

    def result(self, timeout=None, retry=DEFAULT_RETRY):
        """Get the result of the operation, blocking if necessary.

        Args:
            timeout (int):
                How long (in seconds) to wait for the operation to complete.
                If None, wait indefinitely.

        Returns:
            google.protobuf.Message: The Operation's result.

        Raises:
            google.api_core.GoogleAPICallError: If the operation errors or if
                the timeout is reached before the operation completes.
        """
        kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
        self._blocking_poll(timeout=timeout, **kwargs)

        if self._exception is not None:
            # pylint: disable=raising-bad-type
            # Pylint doesn't recognize that this is valid in this case.
>           raise self._exception
E           google.api_core.exceptions.InternalServerError: 500 Error encountered during execution. Retrying may solve the problem.

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:135: InternalServerError

=================================== FAILURES ===================================
___________________________ test_load_table_uri_json ___________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7fb817ea1940>
random_table_id = 'precise-truck-742.python_dataset_sample_20210824155155_176f3c90.example_table_20210824155155_ab9701c1'

    def test_load_table_uri_json(capsys, random_table_id):

>       load_table_uri_parquet.load_table_uri_parquet(random_table_id)

samples/tests/test_load_table_uri_parquet.py:20:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
samples/load_table_uri_parquet.py:33: in load_table_uri_parquet
    load_job.result()  # Waits for the job to complete.
google/cloud/bigquery/job/base.py:708: in result
    return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <google.cloud.bigquery.job.load.LoadJob object at 0x7fb817ea19d0>
timeout = None, retry = <google.api_core.retry.Retry object at 0x7fb8183a19d0>

    def result(self, timeout=None, retry=DEFAULT_RETRY):
        """Get the result of the operation, blocking if necessary.

        Args:
            timeout (int):
                How long (in seconds) to wait for the operation to complete.
                If None, wait indefinitely.

        Returns:
            google.protobuf.Message: The Operation's result.

        Raises:
            google.api_core.GoogleAPICallError: If the operation errors or if
                the timeout is reached before the operation completes.
        """
        kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
        self._blocking_poll(timeout=timeout, **kwargs)

        if self._exception is not None:
            # pylint: disable=raising-bad-type
            # Pylint doesn't recognize that this is valid in this case.
>           raise self._exception
E           google.api_core.exceptions.InternalServerError: 500 Error encountered during execution. Retrying may solve the problem.

.nox/prerelease_deps-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:135: InternalServerError

______ TestBigQuery.test_load_table_from_local_parquet_file_decimal_types ______

target = functools.partial(<bound method PollingFuture._done_or_raise of <google.cloud.bigquery.job.load.LoadJob object at 0x7fabf9596880>>)
predicate = <function if_exception_type.<locals>.if_exception_type_predicate at 0x7fac42cb3310>
sleep_generator = <generator object exponential_sleep_generator at 0x7fabe8b6b970>
deadline = 120, on_error = None

    def retry_target(target, predicate, sleep_generator, deadline, on_error=None):
        """Call a function and retry if it fails.

        This is the lowest-level retry helper. Generally, you'll use the
        higher-level retry helper :class:`Retry`.

        Args:
            target(Callable): The function to call and retry. This must be a
                nullary function - apply arguments with `functools.partial`.
            predicate (Callable[Exception]): A callable used to determine if an
                exception raised by the target should be considered retryable.
                It should return True to retry or False otherwise.
            sleep_generator (Iterable[float]): An infinite iterator that determines
                how long to sleep between retries.
            deadline (float): How long to keep retrying the target. The last sleep
                period is shortened as necessary, so that the last retry runs at
                ``deadline`` (and not considerably beyond it).
            on_error (Callable[Exception]): A function to call while processing a
                retryable exception.  Any error raised by this function will *not*
                be caught.

        Returns:
            Any: the return value of the target function.

        Raises:
            google.api_core.RetryError: If the deadline is exceeded while retrying.
            ValueError: If the sleep generator stops yielding values.
            Exception: If the target raises a method that isn't retryable.
        """
        if deadline is not None:
            deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(
                seconds=deadline
            )
        else:
            deadline_datetime = None

        last_exc = None

        for sleep in sleep_generator:
            try:
>               return target()

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:190:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <google.cloud.bigquery.job.load.LoadJob object at 0x7fabf9596880>
retry = <google.api_core.retry.Retry object at 0x7fac42c47fd0>

    def _done_or_raise(self, retry=DEFAULT_RETRY):
        """Check if the future is done and raise if it's not."""
        kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}

        if not self.done(**kwargs):
>           raise _OperationNotComplete()
E           google.api_core.future.polling._OperationNotComplete

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:87: _OperationNotComplete

The above exception was the direct cause of the following exception:

    def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY):
        """Poll and wait for the Future to be resolved.

        Args:
            timeout (int):
                How long (in seconds) to wait for the operation to complete.
                If None, wait indefinitely.
        """
        if self._result_set:
            return

        retry_ = self._retry.with_deadline(timeout)

        try:
            kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
>           retry_(self._done_or_raise)(**kwargs)

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/future/polling.py:108:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (), kwargs = {}
target = functools.partial(<bound method PollingFuture._done_or_raise of <google.cloud.bigquery.job.load.LoadJob object at 0x7fabf9596880>>)
sleep_generator = <generator object exponential_sleep_generator at 0x7fabe8b6b970>

    @functools.wraps(func)
    def retry_wrapped_func(*args, **kwargs):
        """A wrapper that calls target function with retry."""
        target = functools.partial(func, *args, **kwargs)
        sleep_generator = exponential_sleep_generator(
            self._initial, self._maximum, multiplier=self._multiplier
        )
>       return retry_target(
            target,
            self._predicate,
            sleep_generator,
            self._deadline,
            on_error=on_error,
        )

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:283:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

target = functools.partial(<bound method PollingFuture._done_or_raise of <google.cloud.bigquery.job.load.LoadJob object at 0x7fabf9596880>>)
predicate = <function if_exception_type.<locals>.if_exception_type_predicate at 0x7fac42cb3310>
sleep_generator = <generator object exponential_sleep_generator at 0x7fabe8b6b970>
deadline = 120, on_error = None

    def retry_target(target, predicate, sleep_generator, deadline, on_error=None):
        """Call a function and retry if it fails.

        This is the lowest-level retry helper. Generally, you'll use the
        higher-level retry helper :class:`Retry`.

        Args:
            target(Callable): The function to call and retry. This must be a
                nullary function - apply arguments with `functools.partial`.
            predicate (Callable[Exception]): A callable used to determine if an
                exception raised by the target should be considered retryable.
                It should return True to retry or False otherwise.
            sleep_generator (Iterable[float]): An infinite iterator that determines
                how long to sleep between retries.
            deadline (float): How long to keep retrying the target. The last sleep
                period is shortened as necessary, so that the last retry runs at
                ``deadline`` (and not considerably beyond it).
            on_error (Callable[Exception]): A function to call while processing a
                retryable exception.  Any error raised by this function will *not*
                be caught.

        Returns:
            Any: the return value of the target function.

        Raises:
            google.api_core.RetryError: If the deadline is exceeded while retrying.
            ValueError: If the sleep generator stops yielding values.
            Exception: If the target raises a method that isn't retryable.
        """
        if deadline is not None:
            deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(
                seconds=deadline
            )
        else:
            deadline_datetime = None

        last_exc = None

        for sleep in sleep_generator:
            try:
                return target()

            # pylint: disable=broad-except
            # This function explicitly must deal with broad exceptions.
            except Exception as exc:
                if not predicate(exc):
                    raise
                last_exc = exc
                if on_error is not None:
                    on_error(exc)

            now = datetime_helpers.utcnow()

            if deadline_datetime is not None:
                if deadline_datetime <= now:
>                   raise exceptions.RetryError(
                        "Deadline of {:.1f}s exceeded while calling {}".format(
                            deadline, target
                        ),
                        last_exc,
                    ) from last_exc
E                   google.api_core.exceptions.RetryError: Deadline of 120.0s exceeded while calling functools.partial(<bound method PollingFuture._done_or_raise of <google.cloud.bigquery.job.load.LoadJob object at 0x7fabf9596880>>), last exception:

I don't see how adding a 5-minute timeout could have caused these, but will need some more testing.

[tseaver]

@tswast did you ever track down the source of those 500 errors?

[tswast]

@tswast did you ever track down the source of those 500 errors?

I didn't see anything obvious in the logs, but there was a production issue in the US region yesterday so maybe that's it?