Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Enable Retries on Http calls for Saas Connectors #1290

Closed
earmenda opened this issue Sep 12, 2022 · 0 comments · Fixed by #1376
Closed

Enable Retries on Http calls for Saas Connectors #1290

earmenda opened this issue Sep 12, 2022 · 0 comments · Fixed by #1376
Assignees
Labels
enhancement New feature or request

Comments

@earmenda
Copy link
Contributor

earmenda commented Sep 12, 2022

Is your feature request related to a specific problem?

Currently our saas connectors fail when hitting a throttle from http servers without any sort of retrying.

Describe the solution you'd like

Saas connectors use the requests library to make http request. This library supports retrying on throttles but we should also consider retries at the Node level.

Describe alternatives you've considered, if any

  • It is possible to handle these exceptions at the Node level
  • We're also planning on working in a client side rate limiter but these can either work together or be exclusive features

Additional context

Example

import logging
import requests

from requests.adapters import HTTPAdapter, Retry

logging.basicConfig(level=logging.DEBUG)

s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))

s.get("http://httpstat.us/503")
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant