Pull Request

Add new crawlers:

• Puppeteer: Launch Chrome to load JavaScript. Useful for websites that require JavaScript to display content.
• Cheerio (default): Load only the raw HTML without JavaScript. This method is 18 times faster than the previous default.

Add new scrapers:

• Default: Extract page blocks one by one. Useful for content websites.
• Schema: Extract schema data from the meta description. Useful for e-commerce and media sites.
• Custom: Use selectors to extract data. Useful for highly personalized websites.
• Docsearch: Export a compatible version for Docsearch. Useful for older documents.
• Markdown: Export all page content as a single markdown file. Useful for LLMs.

Add some test and benchmarks: yarn run tests

A full documentation

This is the list of features and fixes it should include.

Features

P1

• Create custom scraper #18
• Create Puppeteer crawling backend option #108
• Create Cheerio crawling backend option #109
• Create Markdown scraping strategy #110
• Keep the previous settings #103
• Index the PDF files #104

P2

• Provide option to slow or rate limit requests #99
• Handle pages not found #48
• Load the sitemap as starter point for crawling.  #102
• Improve errors logs #34
• Handle pages not found #48

P3

• Retrieve page titles from meta tags #53
• Create Playwright crawling backend option #107
• Add possibility to exclude selectors #93
• Detect/handle localized pages #105
• Update Schema scraping strategy to handle microdata #106

Breaking

• Remove the useless headless option #112
• By default use cheerio instead of Puppeteer #113

Fix

• Throw error when redis server is not answering #56