Skip to content

Commit

Permalink
Implement pkg crul for get funcs
Browse files Browse the repository at this point in the history
* get_storms and get_storm_data now utilize crul to help with
asyncrhonous requests and to prohibit too many requests to the NHC
archives within a given period of time.

* load_storm data has been removed. Users are encouraged to download
the rrricanesdata pkg for all data up to the previous month.

* All checks and tests pass. 0 ERRORS, 0 WARNINGS, 0 NOTES
  • Loading branch information
timtrice committed Jul 16, 2017
2 parents 7c9879e + 59b2b01 commit d631a32
Show file tree
Hide file tree
Showing 283 changed files with 8,997 additions and 5,261 deletions.
3 changes: 2 additions & 1 deletion .Rbuildignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,5 @@
^datasets/
^reports/
^\.github$
^_pkgdown.yml$
^_pkgdown\.yml$
^deploy\.sh$
54 changes: 26 additions & 28 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,40 +1,38 @@
language: r

warnings_are_errors: true

addons:
apt:
packages:
- libgdal-dev
- libproj-dev

# Test package on multiple platforms
- libgdal-dev
- libproj-dev
matrix:
include:
- os: linux
dist: trusty
sudo: false
r: 3.3.3
- os: linux
dist: trusty
sudo: false
r: release
- os: linux
dist: trusty
sudo: false
r: devel
- os: osx
r: 3.3.3
- os: osx
r: release

- os: linux
dist: trusty
sudo: false
r: 3.3.3
- os: linux
dist: trusty
sudo: false
r: release
- os: linux
dist: trusty
sudo: false
r: devel
- os: osx
r: 3.3.3
- os: osx
r: release
r_packages:
- covr

r_github_packages:
- timtrice/rrricanesdata@feature/drat
cache: packages

after_success:
- Rscript -e 'library(covr); codecov()'

#r_github_packages:
# - ropenscilabs/rnaturalearthhires
- test $TRAVIS_PULL_REQUEST == "false" &&
test $TRAVIS_BRANCH == "master" &&
bash deploy.sh
env:
global:
secure: t3RJn4gGOnWDc2DSBOHwZ+DkLTZsmo2f9JDfg0H9ugzhZQFvXWy9/+HCak4t00aCpRojkMadqmQPORoeGCvzbdZV6Iu+ueFBb9JPD6CVgBiUYLcBL0aaYtPydKpw5EAFmvoHQo2J/p6bVtVvpBPOf+OVaeLqzcKFD8mxJtVl/sbSuZzCUKhs2KPIEtfpYZuchlZKRV6K+vrDomQytqQJeTlc5TTqY1VMOyY2E1eRgTxpTnN44caWSiqAmMkBx403TmFRpA0XZzsxxmUxTpXrNayT6S0qdn/VClyTTvJb/E92jFjm6c2MxCL6D5Xw7mv51srHf4qIxf3eLyj2ilZUKB4aNeecr5Y1/NeBgmx9K+WD16e9TJYOfR0J8i2HikBLgSgFCvU8D5p3sC6gI/+GCPBXyAEVcndnPUBCaRqfz4eeu3CkxiWgzWwabBoqjas/jp7gyOaj1lwEqOm+WNgA4E/7Klv161CXVHPLyvC7w/7mwVLK6PYN/6yyLAgGA8sumigSPxUvRSnzpDhBRaZH0/39pMFcY+XDjM0tVGj2ku/fdWwKHK8nwTEjNYVWGEjnFuJt9mGnacWOnyeEG9AAMvXYKrLkbKVWh+i4332F0YGqSmtzBn+2X9EQ753fujXd+s3jKkoKaYI58hb+A7BVPeVInDJOfDMmj6KklT6JFHM=
27 changes: 26 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,36 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/).

## [0.2.0-5.1] yyyy-mm-dd
## [0.2.0-6] 2017-07-16

### Added
- NA

### Changed
- `get_storms` and `get_storm_data` now use asynchronous http requests to make data collection faster. (#94)
- Prefaced all built-in data objects with "df"; modified names slightly.

### Removed
- `load_storm_data` has been removed. Archived data can now be accessed
through `rrricanesdata`.

### Deprecated
- NA

### Fixed
- NA

### Security
- NA

## [0.2.0-5.1] 2017-07-10

### Added
- Instructions for Linux users to install `libgdal1-dev`, `libproj-dev`, and `libxml2-dev`. (#95)

### Added
- NA

### Changed
- NA

Expand Down
21 changes: 13 additions & 8 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ Type: Package
Title: Web scraper for Atlantic and east Pacific hurricanes and tropical storms
Description: Get archived data of past and current hurricanes and tropical
storms for the Atlantic and eastern Pacific oceans. Data is available for
storms since 1998.
Version: 0.2.0-5.1
storms since 1998. Datasets are updated via the rrricanesdata package.
Currently, this package is about 6MB of datasets. See the README or view
`vignette("drat")` for more information.
Version: 0.2.0-6
Authors@R: c(person("Tim", "Trice", email = "[email protected]", role = c("aut", "cre")))
Depends:
R (>= 3.3.3)
Expand All @@ -15,28 +17,31 @@ LazyData: TRUE
ByteCompile: TRUE
Imports:
broom (>= 0.4.2),
dplyr (>= 0.7.1),
crul (>= 0.3.4),
dplyr (>= 0.5.0),
ggplot2 (>= 2.2.1),
httr (>= 1.2.1),
lubridate (>= 1.6.0),
magrittr (>= 1.5),
maptools (>= 0.9-2),
purrr (>= 0.2.2.2),
readr (>= 1.1.1),
rgdal (>= 1.2-7),
purrr (>= 0.2.2),
readr (>= 1.1.0),
rgdal (>= 1.2-6),
rgeos (>= 0.3-23),
rlang (>= 0.1.1),
rnaturalearthdata (>= 0.1.0),
rvest (>= 0.3.2),
stringr (>= 1.2.0),
tibble (>= 1.3.3),
tidyr (>= 0.6.3),
tibble (>= 1.3.0),
tidyr (>= 0.6.1),
xml2 (>= 1.1.1)
Suggests:
devtools,
knitr,
rmarkdown,
rrricanesdata (>= 0.0.1),
sp,
testthat
RoxygenNote: 6.0.1
VignetteBuilder: knitr
Additional_repositories: https://timtrice.github.io/drat
1 change: 0 additions & 1 deletion NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ export(gis_prob_storm_surge)
export(gis_windfield)
export(gis_wsp)
export(knots_to_mph)
export(load_storm_data)
export(mb_to_in)
export(saffir)
export(shp_to_df)
Expand Down
7 changes: 4 additions & 3 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
rrricanes 0.2.0-5.1 (yyyy-mm-dd)
rrricanes 0.2.0-6 (2017-07-16)
==================================

### NEW FEATURES
Expand All @@ -7,15 +7,16 @@ rrricanes 0.2.0-5.1 (yyyy-mm-dd)

### MINOR IMPROVEMENTS

* NA
* `get_storms` and `get_storm_data` have been rewritten to utilize pkg `crul`'s asynchronous features. This will not make much of a difference in `get_storms` (and may actually be slightly slower; to be explained). But the difference with `get_storm_data` should be very noticeable. There is a limit to hitting the NHC archives; 80 requests per 10 seconds. Both functions send 4 links through every 0.5 seconds to avoid this limit. Timeout issues should no longer occur so options rrricanes.http_attempts and rrricanes.http_timeout have been removed. The primary cause of long processing now is due to scraping, particularly with the `fstadv` products; the amount of data in these products and the unstructured nature of the products require a number of rules. This can probably be simplified in future releases. (#94)

### BUG FIXES

* NA

### DEPRECATED AND DEFUNCT

* NA
* `load_storm_data` has been removed in favor of loading `rrricanesdata`. See
vignette "installing_rrricanesdata" for more information.

rrricanes 0.2.0-5 (2017-07-08)
==================================
Expand Down
Loading

0 comments on commit d631a32

Please sign in to comment.