Analyzing data in the stream

Streaming Data with AWS Kinesis and Lambda

Maksim Pecherskiy

Data Engineer

Last lesson

graph

Streaming Data with AWS Kinesis and Lambda

This lesson

graph

Streaming Data with AWS Kinesis and Lambda

Kinesis Data Analytics

graph

Streaming Data with AWS Kinesis and Lambda

Kinesis Data Analytics

graph

Streaming Data with AWS Kinesis and Lambda

Kinesis Data Analytics

graph

Streaming Data with AWS Kinesis and Lambda

Why Kinesis Data Analytics

graph

Streaming Data with AWS Kinesis and Lambda

Kinesis Data Analytics vs transformation Lambdas

Transformational Lambda
  • Python + Pandas
  • Filter / aggregate
  • Fixed window
  • Great for data transformations per item
  • Cannot combine multiple streams
  • Not the best way to send output to another destination
Kinesis Data Analytics
  • SQL
  • Filter / aggregate
  • We control the window
  • Lets us look at the stream in chunks
  • Can combine multiple streams
  • Can send output to another stream or other destinations
Streaming Data with AWS Kinesis and Lambda

Kinesis Data Analytics SQL

graph

Streaming Data with AWS Kinesis and Lambda

Kinesis Data Analytics SQL

-- Create destination SQL stream
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" 
(ITEM VARCHAR(1024), ITEM_COUNT DOUBLE);

-- Create the pump
CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM"

-- Pump the results
SELECT * FROM "SOURCE_SQL_STREAM__001"
Streaming Data with AWS Kinesis and Lambda

Some options

  • Join multiple streams
  • Enrich data from a static list using a join
  • Find anomalies in the data
  • Continuously filter data
  • Find the top x repeating items in a timeframe
Streaming Data with AWS Kinesis and Lambda

Finding overpingers

2020-07-21_05-19.png

Streaming Data with AWS Kinesis and Lambda

Let's practice!

Streaming Data with AWS Kinesis and Lambda

Preparing Video For Download...