Monitoring performance

Streaming Data with AWS Kinesis and Lambda

Maksim Pecherskiy

Data Engineer

Last lesson

Lambda transform

def lambda_handler(event, context):

    comprehend = boto3.client('comprehend', 
                              region_name='us-east-1', 
                              aws_access_key_id = AWS_KEY, 
                              aws_secret_access_key=AWS_SECRET)

    output = []

    for record in event['records']:
        ...

Lambda transform

def lambda_handler(event, context):
    ...
    for record in event['records']:

        dict_data = base64.b64decode(record['data']).decode('utf-8').strip()
        dict_data = json.loads(dict_data)

        sentiment_all = comprehend.detect_sentiment(
          Text=dict_data['text'], 
          LanguageCode=dict_data['lang'])
        dict_data['sentiment'] = sentiment_all['Sentiment']
        ...

Lambda transform

def lambda_handler(event, context):
    ...
    for record in event['records']:
        ...

        output_record = {
            'recordId': record['recordId'],
            'result': 'Ok',
            'data': base64.b64encode(json.dumps(dict_data).encode('utf-8'))
        }

        output.append(output_record)
    return {'records': output}

Wiring it up

Update firehoseDeliveryRole

Create delivery stream

Cloudwatch

The stream should minimize data loss due to downtime.

Logs (raw data)
Metrics (measures of various activities of the service)
Alarms (notifications when a metric is out of a specified range)
Dashboards (metrics visualization)

Failure points

Let's practice!

Streaming Data with AWS Kinesis and Lambda