What is streaming and why does it matter?

Streaming Data with AWS Kinesis and Lambda

Maksim Pecherskiy

Data Engineer

Batch vs stream

chapter1_take_ms_dahkX.png

Streaming Data with AWS Kinesis and Lambda

Batch vs stream

Batch

  • "Better"
  • Larger datasets
  • More complex analysis
  • Slower moving data
  • Ex. Daily sales report
  • Ex. Forecasting next month's sales
  • Ex. Churn prediction

Stream

  • "Cooler"
  • Simpler analysis: aggregation / filtering
  • Individual records / micro batches
  • Data moves FAST
  • Ex. Fraud detection
  • Ex. Monitoring wind turbines
  • Ex. Real time alerting
Streaming Data with AWS Kinesis and Lambda

Cody and the fleet

Cody

Cody next to her bike

The Fleet

a collage of vehicles

Streaming Data with AWS Kinesis and Lambda

Telematics streaming

chapter1_take_ms_IfvAm.png

chapter1_take_ms_tNX4d.png

chapter1_take_ms_frlCB.png

Streaming Data with AWS Kinesis and Lambda

AWS Kinesis

Streaming Data with AWS Kinesis and Lambda

Data Firehose

chapter1_take_ms_UPYak.png

Streaming Data with AWS Kinesis and Lambda

Delivery streams

chapter1_take_ms_MDHnC.png

Streaming Data with AWS Kinesis and Lambda

Delivery streams

chapter1_take_ms_ytekB.png

Streaming Data with AWS Kinesis and Lambda

Delivery streams

chapter1_take_ms_gx6D1.png

Streaming Data with AWS Kinesis and Lambda

Creating a Firehose client

import boto3

firehose = boto3.client('firehose',

                        aws_access_key_id=AWS_KEY_ID, 
                        aws_secret_access_key=AWS_SECRET,

                        region_name='us-east-1')
Streaming Data with AWS Kinesis and Lambda

Working with delivery streams

# Show created delivery streams
response = firehose.list_delivery_streams()

print(response['DeliveryStreamNames'])

['old-delivery-stream1', 'a-test-stream']
Streaming Data with AWS Kinesis and Lambda

Delete streams

# Show created delivery streams
response = firehose.list_delivery_streams()

# Delete them all!
for stream_name in response['DeliveryStreamNames']:

      firehose.delete_delivery_stream(DeliveryStreamName=stream_name)
Streaming Data with AWS Kinesis and Lambda

Review

  • Batch vs stream
  • Cody and telematics collection
  • AWS Kinesis
  • Kinesis Firehose Delivery Streams
  • AWS Kinesis Data Firehose
  • List and delete Firehose delivery streams
  • Producer -> data generator
  • Destination -> where the data is going
Streaming Data with AWS Kinesis and Lambda

Let's practice!

Streaming Data with AWS Kinesis and Lambda

Preparing Video For Download...