Amazon Kinesis Streaming Data in the Cloud

Amazon AWS rolled out its newest Cloud based solution today. Kinesis is Amazon's answer to Streaming Data challenges. The solution is aimed at use cases in the areas of Accelerated Log/Data Feed Ingest-Tranform-Load, Continual Metrics / KPI Extraction, Near real-time Data Analytics and Complex Event Processing (CEP). Kinesis leverages Amazon's investments in S3, DynamoDB and Redshift so customers can aggregate and store the data, extract information from the data as its in flight and apply business anlytics to the information.

The system is designed to meet needs for low end-to-end latency from data ingest to processing, high scalability, data durablity via replication, system elasticity and openess to support application developers who will integrate it with operational workflows. The system is a highly managed service freeing users to focus on thier businees instead of tech.

Getting data into the system is fairly straight forward using the Amazon PUT function, each update is limited to 50kb in size. Assuming standard enterprise bandwidth and fast souce application speeds Kinesis can ingest the information in under a second. Data repllication takes about the same time and in the end Kinesis can make data available within 7 seconds. This isn't how I would define real-time streaming but for many applications its a huge advantage over traditional batch and will often match the speed of business.

As you would expect Amazon has priced the solution as a pay-as-you-go service. You will need to provision 1 shard for every 1000 transaction per second (TPS) you are processing for each 1MB/sec you ingest the cost is $0.015 the same is true for egress of data at 2MB/sec. Additional costs of $0.28 per 1,000,000 PUT transactions will also apply.

I like this offer it fits well between sub-second real-time streaming and batch. Most companies would be challenged to find processes that demand higher speeds that what Kinesis offers so I suspect it will fill a void for many of their clients. Early adopter for Kinesis have come from the Financial services and digital advertising sectors both industries are leaders in real-time adoption.

The product is still in limited BETA so you will have to apply to give it a try.