Feat/bulk/all create job

[gweiying]

Context

This PR adds logic to create and send out jobs to the QR code generation Lambda via SQS.

The logic is as follows:

• After the url mappings have been created, url mappings are sent to the JobController
• JobController creates a Job,
• Job Controller batches the url mappings into several JobItems belonging to that Job
  • for batch size of 100, 919 mappings will be split into 10 JobItems belonging to 1 Job, 9 JobItems with 100 mappings each and 1 JobItem with 19
• JobController will start these JobItems by sending it via SQS to QR code generation Lambda

Details

SQS authentication

We have added security rules on the SQS queue so that only the EB IAM role is able to send incoming messages to the queue.

We experimented with VPC endpoints to secure the traffic between EB and SQS, but the VPC endpoint caused the EB deployments to fail. See https://stackoverflow.com/questions/55458492/create-aws-vpc-endpoint-for-sqs. After debugging for hours, we realised it was simpler to use the NAT gateway already attached to our private VPC security group to allow for outbound internet connections to the SQS queue.

Local development

Currently this feature does not work locally as there's no local stack set up yet. This is a lower priority task and will be tackled at the end.

Tests

What tests should be run to confirm functionality?

• Deploy to staging,
• On Health staging (not configured on Edu and Go yet), create a bulk upload via Postman API
• You should be able to see the following
  • New job and job items should be created on health-staging DB jobs table and job_items table. (Check the job uuid, you'll need it for later)
  • SQS health-staging-bulk-qrcode-generate-start should receive the job item messages
  • Lambda health-staging-bulk-qrcode-generation should be triggered (can see invocations via Cloudwatch)
  • S3 bucket should have the newly generated files stored under job uuid from the DB table

Deploy Notes

Notes regarding deployment of the contained body of work. These should note any
new dependencies, new scripts, etc.

New environment variables:

• SQS_BULK_QRCODE_GENERATE_START_URL
• SQS_TIMEOUT
• SQS_REGION
• BULK_QR_CODE_BATCH_SIZE

[halfwhole]

LGTM, staging setup works really nicely! few nits below:

[halfwhole]

the description for BULK_QR_CODE_BATCH_SIZE looks the same as BULK_UPLOAD_MAX_NUM

[gweiying]

fixed cc818c3

[halfwhole]

could these metric names be moved to dogstatsd.ts?

[gweiying]

will do this in one go for all the other metrics too! good idea

[halfwhole]

the readme says this env var is compulsory though it looks like it's optional - am wondering if SQS_REGION could/should be optional too?
also just realised the existence of requiredVars in this file, which could be helpful to add the compulsory ones to

[gweiying]

fixed cc818c3