Skip to content

Home

Jump to bottom
RobKraft edited this page Nov 23, 2021 · 2 revisions

Welcome to the local-govt-reporter-scripts wiki!

The "scripts" are actually a C# .Net 6.0 program that uses Selenium WebDriver to parse web sites. It all runs in a EC2 instance set up as follows:

LocalGovtReporter on EC2 (Ohio US-East-2):

  • EC2 instance
    • Security group “launch-wizard-1” (default)
    • IAM Role: “ec2websitescraper”
      • POLICY: “AmazonSQSFullAccess”,“AmazonDynamoDBFullAccess”
    • SIZE: T2.micro
    • Configuration in EC2 with Windows Server 2019:
      • Enable download files from Internet in IE
      • Download and install Chrome
      • Download and install .Net Core 6
      • Disable Chrome Updates because the sdk in the websitescraping tool uses a specific version of Chrome: https://www.minitool.com/news/disable-automatic-chrome-updates.html
      • Download C# .Net app code
      • Create an account named LocalGovtReporter in the instance for running a scheduled task. (Can’t use administrator because AWS will be changing the PWD on that account).
      • Create a scheduled task to run at startup (run by LocalGovtReporter) even if account no logged in to run the WebSiteScraper C# .net program.
  • Created an AWS SQS named “WebSiteScrapingTask” to receive messages from the websitescraping program that runs in the EC2 instance.
    • IAM role: none. Access Policy for IAM Account (I think)
  • Created a Lambda “startec2instancewebsitescraper” that uses a bit of python script to start specific EC2 instance 'i-014d563b4f0773da2'.
    • IAM role: “startstopec2instances”
      • POLICY: “startstopec2instance”
        • Start and Stop EC2 Instances
  • Created a Lambda “stopec2instancewebsitescraper” that uses a bit of python script to stop specific EC2 instance 'i-014d563b4f0773da2'.
    • IAM role: “startstopec2instances”
      • POLICY: “startstopec2instance”
        • CloudWatch Logs and EC2
  • Created an AWS Eventbridge rule “autostartwebsitescraping” to schedule “startec2instancewebsitescraper” Lambda to run every Sunday at 2am Central.
  • Created an AWS Eventbridge rule autostopwebsitescraping to schedule “stopec2instancewebsitescraper” Lambda to run every Sunday at 3am Central.
  • Create a Lambda to be called when SQS message received to send email to interested parties.
    • IAM: default Lambda role.
      • POLICY: Edit the IAM role and add AmazonSQSReadOnlyAccess to it. Also AWSLambdaSQSQueueExecutionRole
  • Create a trigger for the Lambda based on the SQS role to send email to and from [email protected]
Clone this wiki locally