Analyzing data in the stream

Streaming de dados com AWS Kinesis e Lambda

Maksim Pecherskiy

Data Engineer

Last lesson

graph

This lesson

graph

Kinesis Data Analytics

graph

Kinesis Data Analytics

graph

Kinesis Data Analytics

graph

Why Kinesis Data Analytics

graph

Kinesis Data Analytics vs transformation Lambdas

Transformational Lambda

Python + Pandas
Filter / aggregate
Fixed window
Great for data transformations per item
Cannot combine multiple streams
Not the best way to send output to another destination

Kinesis Data Analytics

SQL
Filter / aggregate
We control the window
Lets us look at the stream in chunks
Can combine multiple streams
Can send output to another stream or other destinations

Kinesis Data Analytics SQL

graph

Kinesis Data Analytics SQL

-- Create destination SQL stream
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" 
(ITEM VARCHAR(1024), ITEM_COUNT DOUBLE);

-- Create the pump
CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM"

-- Pump the results
SELECT * FROM "SOURCE_SQL_STREAM__001"

Some options

Join multiple streams
Enrich data from a static list using a join
Find anomalies in the data
Continuously filter data
Find the top x repeating items in a timeframe

Finding overpingers

Let's practice!

Streaming de dados com AWS Kinesis e Lambda