Decoupling using AWS Messaging Services

Developing applications on AWS

Ricardo Sueiras

Principal Technologist

Amazon SQS fundamentals

 

• Handles traffic bursts independently of producers.

• Managed message queue for asynchronous messaging.
• Producers send messages into a queue.
• Consumers retrieve messages from the queue.
• Messages retained until a consumer picks them up.
• Retention: 4 days by default, 14 days maximum.
• Consumers poll the queue and process at their own rate.

Message visibility

• A message becomes invisible once a consumer reads it.
• Visibility timeout controls how long it stays hidden.
• If processing fails and the message is not deleted:
  • It becomes visible again.
  • It can be reprocessed.
• Visibility timeout: 30 seconds by default.
• Maximum visibility timeout is 12 hours.

 

Message size

 

• Messages can be up to 256KB in size.
• SQS Extended Client library enables larger messages.
• Stores the large payload in Amazon S3.
• Sends a pointer to S3 in the message payload.

Short polling

• Polling strategy affects cost and efficiency.
• Short polling is the default.
• Queries a subset of SQS servers, returns immediately.
• May return empty, even when messages are available.
• Results in more API calls and higher cost.

 

Long polling

• Long polling queries all SQS servers.
• Waits up to 20 seconds for a message.
• Eliminates unnecessary empty responses.
• Reduces API calls and lowers cost.

 

Standard queues

• SQS offers two types of queues.
• Standard queues:
  • At-least-once delivery.
  • Best-effort ordering.
  • Nearly unlimited throughput.

 

FIFO queues

 

• FIFO queues:
  • Exactly-once processing.
  • Strict message ordering.
  • Built-in message de-duplication.
• Default 300 TPS
  • Increase to 3,000 with batching or more with high-throughput mode.
• Choose the queue type by use case:
  • Standard for scale and throughput.
  • FIFO for ordering and de-duping.

Handling poison messages

 

• Dead letter queues (DLQ) store messages that fail processing.
• A message moves to the DLQ once it exceeds maxReceiveCount.
• This avoids retrying a poison message indefinitely.
• The DLQ must be the same queue type as the source.
• Move messages back to the source queue to reprocess.

Securing queues

 

• Encrypts in transit using HTTPS by default.
• Server-side encryption at rest using AWS KMS.
• Encrypt the payload client-side with the AWS Encryption SDK.
• IAM policies govern who can access SQS queues.
• Resource-based policies give granular, queue-level control.

Amazon SNS fundamentals

• Managed publish-subscribe (pub/sub) service.
• Uses a push model.
• Publishers send messages to a topic.
  • Maximum message size is 256KB.
• Consumers subscribe to topics.
• Subscribers receive messages passively.

 

Scale

• Supports up to 12.5 million subscriptions per topic.
• No message retention:
  • Messages are lost if a subscriber is unavailable.
  • Use DLQ or durable subscriber (SQS queue) to avoid losing messages.
• Distributes a single event to many consumers (fan-out).

 

Message filtering

• SNS supports message filtering policies.
• Subscriptions receive only the messages they care about.
• Without a filter, a subscription receives every message.
• Filter policies apply at the subscription level, not the topic.

 

Filter logic and evaluation

 

• Filters are defined in JSON.
• Matched against message attributes.
• Multiple conditions are evaluated as AND.
• Multiple values within a condition are evaluated as OR.
• Many filter operations are supported:
  • Exact string, numeric, prefix, exclusion, and range matching.

SNS security

 

• Encrypts data in transit with HTTPS by default.
• Server-side encryption at rest using AWS KMS.
  • Grant SNS permission to use the KMS key.
• IAM policies provide access control.
  • Govern who can publish to or subscribe from a topic.

Let's practice!

Developing applications on AWS