28 November 2018

Choosing the Right Messaging Service for Your Distributed App (API305)

by mo

In the cloud, modern apps are decoupled into independent building blocks, called microservices, which are easier to develop, deploy, and maintain. Messaging is a central tool used to connect and coordinate these microservices. AWS offers multiple messaging services, which address a variety of use cases. In this session, learn how to choose the service that’s best for your use case as we present the key technical features of each. We pay special attention to integrating messaging services with serverless technology. We cover Amazon Kinesis, Amazon SQS, and Amazon SNS in detail with discussion of other services as appropriate.

Kuba, Sidhartha

  • importance of messaging
  • different messaging use cases
monolith -> de-couple -> microservices

pillars of modern applications:

computer | storage | database | messaging

producer -> send messages
consumer <- receive messages

messages: anything sent between systems

e.g. json message

payload and attributes

aws messaging services

  • sqs -> queues (modern)
  • sns -> topics (modern)
  • kineses -> streams (modern)
  • MQ -> message brokers (hosted active mq broker) enables you to move to cloud (legacy)


customer use cases

Amazon Fresh

  • durable persisten ha system
  • should scale. 1 message or 1M messages
  • easy to manage.
  • features should not come at the cost of complexity.

SQS (standard queues)

producer -> message into queue -> consumer consume from queue

producer -> send A -> store durable and makes copies
producer -> send B  -> network blip. sqs saved message
  -> producer will resend B.

consumer <- receive message

  • message remains in queue for a timeout
  • sqs wont hand out this message while someone is working on it.
  • when message is consumed, consumer is supposed to call delete message.
  • consumer must acknowledge that the message has been processed.
  • if ack is not provided for message, then timeout expires and message is consumable by another consumer.

no coordination and messages can be processed out of order.

how to handle duplicate messages and message ordering.

Amazon Fresh used SQS

  • SQS FIFO Queues
producer -> specify message group (tag message)
consumer pulls first message in group (tag). no other messages can be processed in that group by any consumer until that message is processed.

messages are processed in order but no guarantee as to which consumer will process the message.

a single fifo has limited throughput. 3K messages/second

Okta - Use Case

user events -> apache storm -> streaming analysis

durable scalable persisten message replay stream analysis

message are not independent

  • need to do analysis to detect anomolies and trigger alerts

kineses data streams (like apache kafka)

  • you need to specify shards

  • producer -> message A (tag with partition key) -> partition key is hashed to determine which partition to send message to.

duplicates will be sent to same shard.

consumers still have to detect duplicates

use KCL (kinesis client library) to track which consumer should process which messages and other things.

okta events -> sent to kinesis stream -> apache storm -> kinesis data firehose -> s3 -> redshirt -> aws lambda -> amazon sns

Edmunds Use Case

  • sells new/used cars

car dealer, oem, franchises -> source systems -> message platform -> target systems -> website


  • durable scalable persistent always available
  • minimal latency

Amazon SNS Topics

send something and deliver to multiple destinations. pub/sub

producer -> SNS topics -> can push to different destinations destinations:

  • SQS queue
  • lambda
  • http/s endpoint (like a webhook)
    • you can control the rate of retries
  • Mobile App
  • SMS
  • E-mail

You can configure filters:

  • f(x)
    • decisions can be made based on attributes in message.

publisher -> A SNS topic (stored across multi-AZ, ack immediately. same low latencty regardless of # of destinations) -> fanout to destinations

  • send copy to each destination
  • f(x) filter can prevent message from going to destination

  • retries can be configured based on the destination
    • assume it will retry forever until success
  • slideshare