Skip to content
This repository has been archived by the owner on Jul 27, 2020. It is now read-only.

Malformed pagination causes problems #3

Closed
CloCkWeRX opened this issue Mar 21, 2016 · 0 comments · Fixed by #4
Closed

Malformed pagination causes problems #3

CloCkWeRX opened this issue Mar 21, 2016 · 0 comments · Fixed by #4

Comments

@CloCkWeRX
Copy link
Member

Dubbo's ATDIS feed does this:

$ bundle exec ruby scraper.rb 
Saving page 1 of 7
Saving page 2 of 7
Saving page 3 of 7
Saving page 4 of 7
Saving page 5 of 8
Saving page 6 of 7
Saving page 7 of 13
/home/clockwerx/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/bundler/gems/atdisplanningalertsfeed-ce20de6801cf/lib/atdisplanningalertsfeed.rb:22:in `save_page': undefined method `current' for nil:NilClass (NoMethodError)
    from /home/clockwerx/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/bundler/gems/atdisplanningalertsfeed-ce20de6801cf/lib/atdisplanningalertsfeed.rb:17:in `save'
    from scraper.rb:6:in `<main>'

For the best chance of success, we should probably ignore the published pagination and proceed until there are no results on the next page.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant