Scrapy, a fast high-level web crawling & scraping framework for Python.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Web Scraping Framework

Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python

🤖 Scrape data from HTML websites automatically by just providing examples

📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告

A list of AI agents and robots to block.

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Scrapy middleware to handle javascript pages using selenium

HTTP API for Scrapy spiders

[Unmaintained] A simple and clean video/music/image downloader 👾

The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

ISP Data Pollution to Protect Private Browsing History with Obfuscation

Scrapy Extension for monitoring spiders execution.

WarcDB: Web crawl data as SQLite databases.

Python 3 script to dump/scrape/extract company employees from LinkedIn API

The simple, easy to use command line web crawler.

Scalable Python web scraping scripts for +40 popular domains

Stop stalking and start StopStalking 😉