[CT-352] Catch all Requests Exception on Deps Install and Retry

[emmyoop]

We are not retrying for all exceptions from Requests and it's causing more unexpected/seemingly unexplained failures. We should try to mitigate these failures by retrying (up to 5 times). The logic to retry already exists. We just need to make sure we catch the right/entirety of exceptions.

1. Catch all exceptions from Requests by catching just requests.exceptions.RequestException since all exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. Note that the ReadError should remain.

2. Also fire a new event to track what exception we're retrying so that we can track where the failures are in the future.

dbt-core/core/dbt/utils.py

Lines 609 to 620 in ecfd77f

	except (
	requests.exceptions.ConnectionError,
	requests.exceptions.Timeout,
	requests.exceptions.ContentDecodingError,
	ReadError,
	) as exc:
	if attempt <= max_attempts - 1:
	fire_event(RetryExternalCall(attempt=attempt, max=max_attempts))
	time.sleep(1)
	_connection_exception_retry(fn, max_attempts, attempt + 1)
	else:
	raise ConnectionException("External connection exception occurred: " + str(exc))

Related Issues: #4601, #4178

[nave91]

@emmyoop I'm just noting this down here in case it gets lost in our process. We saw another surge in dbt deps command failing because of hub returning 503 errors.

If we are catching and retrying on RequestException, we should be able catch the hub's 503 errors along with other HTTP error codes.

[barberscott]

Saw this on 1.0.5-rc1:

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1295, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 140, in resolve_packages
    _check_for_duplicate_project_names(resolved, config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 106, in _check_for_duplicate_project_names
    project_name = package.get_project_name(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/base.py", line 90, in get_project_name
    metadata = self.fetch_metadata(project, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/base.py", line 86, in fetch_metadata
    self._cached_metadata = self._fetch_metadata(project, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 61, in _fetch_metadata
    return RegistryPackageMetadata.from_dict(dct)
  File "<string>", line 38, in from_dict
ValueError: Argument for dbt.contracts.project.RegistryPackageMetadata.from_dict method should be a dict instance