Welcome to the Footy Players Data Management repository! This project consists of a set of Python scripts that automate the collection, enhancement, and storage of football player data. Whether you're a football enthusiast or just interested in learning about web scraping, JSON manipulation, and MongoDB integration, this repository has something for you.
In this project, I've created two Python scripts:
-
crawler.py: This script fetches JSON objects from a JSON bin and uses web scraping techniques to find and update each player's image url by searching Google and copying the link. The updated data is then saved to another JSON bin.
-
populater.py: This script reads the updated JSON objects from the JSON bin and populates a MongoDB collection with this enriched player data.
- Automating the boring stuff: There was no need to manually searching for player images and copy the respective image links! The script harnesses the power of web scraping to fetch and update player images automatically. Also, the updated players' data was populated to a MongoDB collection effortlessly!.
-
Clone this repository to your local machine:
git clone https://github.com/{your-username}/Footy-Players.git -
Install the required Python packages:
pip install -r requirements.txt
-
Configure your JSON bin API endpoints and MongoDB connection details in the respective scripts. npoint is recommended
-
Run the scripts:
python crawler.py python populater.py
Feel free to submit bug reports, feature requests, or even pull requests to help make this mini-project even better.