Skip to content

Latest commit

 

History

History
20 lines (15 loc) · 424 Bytes

README.md

File metadata and controls

20 lines (15 loc) · 424 Bytes
Raw

scrapy_tianya

A crawler for bbs.tianya.cn, using scrapy as crawler framework

Prerequisite:

  • scrapy
  • mongodb
  • pymongo

PS: Actually you can rewrite tianya/pipelines.py to change the storage backend, instead of mongodb :)

PPS: xpath links are easy to get in Chrome Developer Tool

Instruction

cd path/to/repo
mkdir job
scrapy crawl tianyaSpider -s JOBDIR=/path/to/job/job-1_or_whatever