Amazon Kinesis Firehose SAM template for ingesting website access logs from Amazon API Gateway. The data is stored in a raw bucket, processed by a Lambda function, and then stored in a processed bucket as well. During the processing period, the data is also pushed to an Amazon DynamoDB table for real-time analytics.
This is the initial delivery stream to ingest large amounts of data
This is the first step for the data. Raw data coming in to the Kinesis Dilvery Stream (Firehose) is first saved here in the format it comes in.
After saving the data to the RawDataBucket, Kinesis then triggers this Lambda for any custom processing
The process function pulls data and sends it to the ProcesseddataTable for immediate analytics
This bucket stors the data in it's processed state
Data is then sent to the KinesisAnalyticsApp for more in depth analysis and reporting
This output directs analyzed data from Kinesis Analytics to the Count Function
Count function takes the analyzed data and pushes it to the CountTable
Stores the latest top vied links as a trend