Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Launch tasks without timeout : option timeout to 0 #282

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ Emitted when a task is queued via [Cluster.queue] or [Cluster.execute]. The firs
- `retryDelay` <[number]> How much time should pass at minimum between the job execution and its retry. Ignored by tasks queued via [Cluster.execute]. Defaults to `0`.
- `sameDomainDelay` <[number]> How much time should pass at minimum between two requests to the same domain. If you use this field, the queued `data` must be your URL or `data` must be an object containing a field called `url`.
- `skipDuplicateUrls` <[boolean]> If set to `true`, will skip URLs which were already crawled by the cluster. Defaults to `false`. If you use this field, the queued `data` must be your URL or `data` must be an object containing a field called `url`.
- `timeout` <[number]> Specify a timeout for all tasks. Defaults to `30000` (30 seconds).
- `timeout` <[number]> Specify a timeout for all tasks. `0` disable, defaults to `30000` (30 seconds).
- `monitor` <[boolean]> If set to `true`, will provide a small command line output to provide information about the crawling process. Defaults to `false`.
- `workerCreationDelay` <[number]> Time between creation of two workers. Set this to a value like `100` (0.1 seconds) in case you want some time to pass before another worker is created. You can use this to prevent a network peak right at the start. Defaults to `0` (no delay).
- `puppeteer` <[Object]> In case you want to use a different puppeteer library (like [puppeteer-core](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteer-vs-puppeteer-core) or [puppeteer-extra](https://github.com/berstend/puppeteer-extra)), pass the object here. If not set, will default to using puppeteer. When using `puppeteer-core`, make sure to also provide `puppeteerOptions.executablePath`.
Expand Down
23 changes: 12 additions & 11 deletions src/Worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,19 @@ export default class Worker<JobData, ReturnData> implements WorkerOptions {

let result: any;
try {
result = await timeoutExecute(
const taskToLaunch = task({
page,
// data might be undefined if queue is only called with a function
// we ignore that case, as the user should use Cluster<undefined> in that case
// to get correct typings
data: job.data as JobData,
worker: {
id: this.id,
},
});
result = timeout === 0 ? await taskToLaunch : await timeoutExecute(
timeout,
task({
page,
// data might be undefined if queue is only called with a function
// we ignore that case, as the user should use Cluster<undefined> in that case
// to get correct typings
data: job.data as JobData,
worker: {
id: this.id,
},
}),
taskToLaunch,
);
} catch (err) {
errorState = err;
Expand Down