Skip to content

A complete multi-threaded web crawler in Python3

License

Notifications You must be signed in to change notification settings

HosseinAbedi/Krawler

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

#KRAWLER: A WEB CRAWLER IN PYTHON3

GitHub issues GitHub forks GitHub stars GitHub license

Project Source code:

See inside src/*

Project docs:

See inside docs/*

Installing the system-wide dependencies:

$ bash system-dependencies.sh
$ sudo pip3 install -U -r requirements.txt

Starting the crawler:

$ pushd src; python3 main.py; popd
![WebCrawler Architecture](docs/figures//WebCrawlerArchitecture.svg)

About

A complete multi-threaded web crawler in Python3

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 99.4%
  • Shell 0.6%