Replacing uses of single-request {httr} functions with RETRY()

[jameslamb]

TL;DR

There are a ton HTTP clients on CRAN, and many of them make single pass-or-fail HTTP requests. If they added some retry logic (with httr::RETRY()), they would be more resilient to transient issues like brief network disruptions or service downtime.

Details

{httr} is a super-popular package for making HTTP requests in R and handling their response. It has a LOT of direct reverse imports, and impacts a lot of other projects indirectly.

I have seen many examples of packages where people use functions like httr::GET(), httr::POST(), httr::VERB(), etc. to make HTTP requests. These functions attempt to make a single request and raise an error if anything goes wrong.

I believe (though I don't know for sure) that this all-or-nothing approach is not something package authors have carefully and intentionally chosen, and changing those calls to httr::RETRY() would make those packages more resilient to transient problems like brief network outages or periods where the service(s) it hits are overwhelmed. In my experience, using retry logic can improve the user experience with HTTP clients.

I propose a group spend some time improving the ecosystem of HTTP clients in R by repeating these steps:

1. choose a package from the {httr} reverse imports
2. go to the package's source code on GitHub (if it exists)
3. find all references to httr::GET(), httr::POST(), httr::PUT(), httr::VERB(), httr::DELETE(), httr::HEAD(), and httr::PATCH() and replace them with the appropriate httr::RETRY()
4. create a pull request with the change, including some text in the summary explaining to the package author how this change will make their code more reliable (see added retry logic to HTTP calls ropensci/eia#5 for an example)

I'd be happy to start out with a short introduction to HTTP and what httr::RETRY() actually does.

[emilyriederer]

Thanks for posting, @jameslamb ! I love this idea of taking the time to shore up bits and pieces of many different packages and spreading best practices.

Another good outcome might be seeing if RStudio would take a PR to {httr} 's 'Best Practices for API packages' vignette to share this advice more broadly

[jdblischak]

My workflowr package is one of those reverse dependencies, and I super welcome a PR to make the GitHub API calls more robust.

The function create_gh_repo() makes 3 GET requests and a POST request.

[jameslamb]

I wrote up a 101-level "wtf is HTTP" last night: https://github.com/jameslamb/talks/tree/master/chi-r-collab-httr. Decided to put it in that repo because I might turn it into a lightning talk in the future.

For those who want to do the "go make PRs to existing packages" part of this project, here are some examples:

• added retry logic to HTTP requests plotly/plotly.R#1656
• added retry logic for HTTP calls cloudyr/aws.alexa#15
• added retry logic to HTTP calls ropensci/eia#5
• added retry logic to HTTP requests r-transit/tidytransit#131
• added retry logic to HTTP calls willdebras/ctar#9
• added retry logic for HTTP requests (fixes #160) uptake/uptasticsearch#172
• dmi3kno/polite@7d34280

The basic process is:

1. Go to httr's CRAN page
2. Look at the packages in Reverse Depends and Reverse Imports. Choose one that sounds interesting and click on it.
3. Hopefully the package has a URL: section with a GitHub link. Click it.
4. From that repository, click fork to fork it. (GitHub docs)
5. Clone the repository to your machine.(GitHub docs)
6. Look for uses of httr single-shot functions.
   • These are:
     • DELETE()
     • GET()
     • HEAD()
     • POST()
     • PATCH()
     • PUT()
     • VERB()
   • You can find these with commands like git grep 'POST(' or using Find in Files in RStudio
7. Replace those calls with httr::RETRY(). See the pull requests linked above for some examples.
8. Commit your changes, push them to your fork, and create a pull request (GitHub docs)
   • When you open the PR, be sure to add a link to this issue in the description. That way GitHub will do the hard work of tracking for us, like this!

[jameslamb]

Some resources shared during our first chat:

• https://http.cat
• http://httpbin.org (try http://httpbin.org/status/418 😀 )
• Mozilla has a page for every status code, for example: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/418

[jameslamb]

Cleaning up my "created by me" issue list today, I think this served it's purpose and can be closed.

Thanks for all your help everyone, this was fun!!!