Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
mike442144 committed Aug 24, 2015
1 parent 255104b commit 19efd2f
Show file tree
Hide file tree
Showing 3 changed files with 20 additions and 2 deletions.
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ node-webcrawler is originally a fork of [node-crawler](https://github.com/sylvin
0.5.1 version changelog:
* remove cache feature, it's useless
* add `localAddress`, `time`, `tunnel`, `proxyHeaderWhiteList`, `proxyHeaderExclusiveList` properties to pass to `request`

0.5.2 version changelog:
* terminate all the resources in your pool before their timeouts have been reached
* add a read-only property `queueSize` to crawler

Features:
* server-side DOM & automatic jQuery insertion with Cheerio (default) or JSDOM
Expand Down Expand Up @@ -134,6 +138,19 @@ Other:
* `referer`: String, if truthy sets the HTTP referer header
* `rateLimits`: Number of milliseconds to delay between each requests (Default 0) Note that this option will force crawler to use only one connection (for now)


Class:Crawler
-------------

Instance of Crawler

_crawler.queue(uri|options)_
* `uri` String, `options` is [Options](#options-reference)

_crawler.queueSize_
Size of queue, read-only


Working with Cheerio or JSDOM
-----------------------------

Expand Down
3 changes: 2 additions & 1 deletion lib/crawler.js
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ Crawler.prototype.init = function init (options) {
if (self.options.onDrain) {
self.options.onDrain();
}
self.pool.destroyAllNow();
});
};

Expand Down Expand Up @@ -457,4 +458,4 @@ Object.defineProperty(Crawler.prototype,'queueSize',{
})

module.exports = Crawler;
module.exports.VERSION = '0.5.1';
module.exports.VERSION = '0.5.2';
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "node-webcrawler",
"version": "0.5.0",
"version": "0.5.2",
"description": "Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously",
"main": "./lib/crawler.js",
"directories": {
Expand Down

0 comments on commit 19efd2f

Please sign in to comment.