Internship

Dependecies

Project for AdataPro Internship
First start Virtual Environment and then install:

pip install scrapy

pip install peewee

pip install scrapy_jsonschema

pip install django

pip install classla

To start Crawler

To run the spiders generating the databases:

python GovScraper/start.py

This code will run 3 spiders:
First to generate the urls for all the articles in a links.db
Second to generate articles.db with all the necessary information from the articles

Set up the deployment environment

AWS

First, go to https://aws.amazon.com/ and create a free-tier account. Then, choose the EC2 service. After that, you have to pick on what OS the server will run. Choose Debian-11. Then you get the option to configure the specs of the server through the 'instance-type'. We chose t2-micro.

SSH Access

To configure the SSH access to the cloud we used PuTTY. It can be downloaded from https://www.putty.org/. From here on we have to configure the PuTTY with the .pem key we get from AWS.

Installing packages

You can install the packages you need with debian commands through the shell.

To start The Django site

cd internshipProj

python manage.py runserver

Lastly, go to http://127.0.0.1:8000/.