Skip to content

Commit

Permalink
Add limit prop to config.
Browse files Browse the repository at this point in the history
  • Loading branch information
lewisdonovan committed Oct 27, 2024
1 parent 30fc290 commit fae39f7
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 2 deletions.
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,11 @@ Whether or not Puppeteer should run in [headless mode](https://www.browserstack.

Defaults to `true`

#### limit
The total number of articles that you would like to be returned. Please note that with higher numbers, the actual returned number may be lower. Typically the max is `99`, but it varies depending on many variables in Puppeteer (such as rate limiting, network conditions etc.).

Defaults to `99`

## TypeScript 💙
Google News Scraper includes full [TypeScript](https://typescriptlang.org/) definitions.

Expand Down
6 changes: 4 additions & 2 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ const googleNewsScraper = async (userConfig: GNSUserConfig) => {
puppeteerHeadlessMode: true,
logLevel: 'error',
timeframe: '7d',
queryVars: {},
queryVars: {},
limit: 99
},
...userConfig,
} as GNSConfig;
Expand Down Expand Up @@ -133,7 +134,8 @@ const googleNewsScraper = async (userConfig: GNSUserConfig) => {
await page.close();
await browser.close()

return results.filter(result => result.title)
const filtered = results.filter(result => result.title);
return config.limit < results.length ? filtered.slice(0, config.limit) : filtered;

}

Expand Down
2 changes: 2 additions & 0 deletions src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ export type GNSUserConfig = SearchParams & {
logLevel?: LogLevel;
queryVars?: QueryVars;
filterWords?: string[];
limit?: number;
};

export type GNSConfig = SearchParams & {
Expand All @@ -29,6 +30,7 @@ export type GNSConfig = SearchParams & {
logLevel: LogLevel;
queryVars: QueryVars;
filterWords?: string[];
limit: number;
};

export type Article = {
Expand Down

0 comments on commit fae39f7

Please sign in to comment.