Streaming data

Developing applications on AWS

Ricardo Sueiras

Principal Technologist

Streaming data fundamentals

 

• Records processed continuously as they arrive.
• Producers write to a stream.
• Consumers read independently and in parallel.
• Absorbs traffic spikes without waiting for batch jobs.

What makes up streaming data?

 

• Producers: applications that emit records.
• Streams: durable, ordered buffers that decouple producers from consumers.
• Consumers: applications that read and process records.
• Storage and Retention: defines how long records stay available for replay.

Shards

 

 

• Shards are units of throughput and parallel processing.
• Partition keys distribute records across shards.
• Ordering is guaranteed only within a shard.
• Consumers use check-pointing to track progress and recover from failures.

AWS managed services for streaming data

• Amazon Kinesis Data Streams.
• Amazon Data Firehose.
• Core building blocks for streaming data applications.

 

Kinesis Data Streams

• Low-latency streaming.
• Configurable retention: 24 hours (default) up to 365 days.
• Durable, replayable streams.
• Multiple consumers can read in parallel.
• Developers manage scaling using shards.

 

When to use Kinesis Data Streams

• Choose it when you need real-time latency.
• You require replay of records.
• You have multiple independent consumers.
• You need custom transformation logic.

 

Writing records

 

• Partition keys control shard placement and ordering.
• PutRecords batches up to 500 records or 5 MB per request.
• Failures are reported per record.
• FailedRecordCount helps you identify failed records.

Consuming records: classic

 

 

• Consumers poll on demand against the Kinesis API.
• Write limits: 1 MB/sec or 1,000 records/sec.
• Read limits: 2 MB/sec or 5 GetRecords calls/sec.

Consuming records: stream start

 

• Consumers specify where in the stream to begin reading.
• TRIM_HORIZON: oldest available record in the shard.
• LATEST: the next new record.
• AT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER: from a specific sequence number.
• AT_TIMESTAMP: at or after a specified timestamp.

Consuming records: enhanced fan-out

• Each registered consumer gets a dedicated 2 MB/sec per shard.
• Records are delivered via HTTP/2 push.
• Use it when multiple independent consumers need low latency.
• They also need high throughput to the same stream.

 

Lambda event source mapping

• Lambda polls Kinesis and invokes in batches.
• BatchSize and MaximumBatchingWindow control batch shape.
• ParallelizationFactor: concurrent invocations per shard.
• BisectBatchOnFunctionError: splits a failing batch to isolate poison pills.
• ReportBatchItemFailures: avoids retrying records that already succeeded.
• OnFailure: sends unprocessable batches to an SQS or SNS DLQ.

 

Data Firehose

• Managed service that delivers streaming data to destinations.
• Targets include Amazon S3, Redshift, and HTTP endpoints.
• Supports buffering, format conversion, and Lambda transformations.
• Higher latency, buffered for seconds to minutes.
• No replay or long-term retention.
• Great for simply getting data to a destination.

 

Pattern: hot-cold

 

 

• Hot-cold pattern combines Data Streams and Data Firehose.
• Data Streams enables real-time processing for producers.
• Firehose archives records from the same source into S3.

Kinesis vs SQS

 

 

• Use Kinesis for ordered, replayable streams with multiple consumers.
• Kinesis suits time-series and ordered data.
• Use SQS for task distribution.

Scaling streaming data

 

• Data streams are divided into shards.
• Partition keys route records to a specific shard.
• The same key always lands on the same shard, preserving order.
• Uneven partition keys lead to hot shards.
• Mitigate with high-cardinality keys.
• Reshard to scale out or in.

Capacity modes: provisioned

• Two capacity modes are available.
• Provisioned mode uses a fixed shard count, managed manually.
• It provides predictable cost and capacity.
• Requires monitoring to avoid throttling and over-provisioning.

 

Capacity modes: on-demand

• On-demand mode scales automatically based on traffic.
• No manual shard management.
• Simplifies operations for unpredictable workloads.
• Higher cost under sustained heavy traffic.

 

Kinesis Producer Library (KPL)

• AWS libraries simplify building scalable producers and consumers.
• The Kinesis Producer Library (KPL) batches and aggregates records.
• It also compresses records and retries on failure.
• Buffers asynchronously to maximize producer throughput.

 

Kinesis Consumer Library (KCL)

 

 

• The Kinesis Consumer Library (KCL) handles shard coordination.
• It manages worker leasing and check-pointing.
• State is stored in a DynamoDB table.
• Consumers use that state to look up where to resume.

Failure handling: idempotency

 

 

• Kinesis delivery is at-least-once, so duplicates are possible.
• Design consumers to be idempotent.
• Deduplicate using the sequence number or a business key in DynamoDB.
• Consumers can reread retained records from a sequence number or timestamp.

Failure handling: throughput errors

 

 

• ProvisionedThroughputExceededException signals throttling.
• Resolve it by increasing the shard count.
• Improve partition key distribution.
• Retry with backoff.
• Switch capacity mode.

• Track IncomingBytes and IncomingRecords in CloudWatch.
• Watch WriteProvisionedThroughputExceeded for throttling.
• Consumer lag occurs when consuming slower than producing.
• Track GetRecords.IteratorAgeMilliseconds: high age indicates lag.

 

Security

• Server-side encryption at rest using AWS KMS.
• Encryption in transit using TLS.
• IAM policies control access.

 

Let's practice!

Developing applications on AWS