Real Time Framework on AWS using Kinesis,Lambda and DynamoDB

Kinesis

AWS/GCP/Azure are offering real-time message broker systems to publish and subscribe events on real-time. We can consider Kinesis as AWS managed alternative to Kafka. It allows you to ingest high volumes of data(IoT,System Logs, Clickstream) per second and have the flexibility to scale automatically based on the demand.

DynamoDB

Fully Managed, Highly available No SQL distributed database. Scales to Massive workloads. It Can handle Millions of Requests per second and trillions of rows and petabytes of storage. Fast and consistency very low latency retrieval. Widely used in big data world for

Use-Case

AWS EC2 server produces Transaction logs with Customer-order transactions and ingests into Kinesis Data stream, On the other side whenever data occurs in Kinesis data stream , Lambda function picks up and pre-process/transforms into DynamoDB Table.

Steps

  1. Create Kinesis Stream
  2. Create DynamoDB Table
  3. Ingest Data into Kinesis Data Stream
  4. Process data stream records using Lambda
  5. Insert processed records into DynamoDB table

Create Kinesis Data Stream

Let’s logon to EC2 server and create data steam by using AWS CLI

Create Kinesis Data Stream

Create DynamoDB table

A DynamoDB table must have primary Key, A primary Key can be defined in two ways

Create DynamoDB Table

Define Lambda Function

Every time a new event created in Kinesis Data stream would trigger AWS Lambda function. We can write any custom code that suites our requirement . In this case, It’s basic ETL to load Kinesis data stream into DynamoDB table.

AWS Lambda ETL (Kinesis Data Stream To DynamoDB)

Ingest Data into Kinesis

There are many ways to ingest data into Kinesis data stream . Kinesis Producer Library(KPL), Kinesis Agent, Third party applications like Apache Spark

Verify DynamoDB table

Kinesis Data Stream Records are successfully pre-processed and ingested into DynamoDB by Lambda function

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sivaprasad Mandapati

Sivaprasad Mandapati

Google Cloud Certified Data Engineer and Architect,Big Data,Spark,ETL Frameworks