In the cloud, modern apps are decoupled into independent building blocks, called microservices, which are easier to develop, deploy, and maintain. Messaging is a central tool used to connect and coordinate these microservices. AWS offers multiple messaging services, which address a variety of use cases. In this session, learn how to choose the service that’s best for your use case as we present the key technical features of each. We pay special attention to integrating messaging services with serverless technology. We cover Amazon Kinesis, Amazon SQS, and Amazon SNS in detail with discussion of other services as appropriate.
- importance of messaging
- different messaging use cases
monolith -> de-couple -> microservices
pillars of modern applications:
computer | storage | database | messaging producer -> send messages consumer <- receive messages
messages: anything sent between systems
e.g. json message
payload and attributes
aws messaging services
- sqs -> queues (modern)
- sns -> topics (modern)
- kineses -> streams (modern)
- MQ -> message brokers (hosted active mq broker) enables you to move to cloud (legacy)
customer use cases
- durable persisten ha system
- should scale. 1 message or 1M messages
- easy to manage.
- features should not come at the cost of complexity.
SQS (standard queues)
producer -> message into queue -> consumer consume from queue
producer -> send A -> store durable and makes copies producer -> send B -> network blip. sqs saved message -> producer will resend B.
consumer <- receive message
- message remains in queue for a timeout
- sqs wont hand out this message while someone is working on it.
- when message is consumed, consumer is supposed to call delete message.
- consumer must acknowledge that the message has been processed.
- if ack is not provided for message, then timeout expires and message is consumable by another consumer.
no coordination and messages can be processed out of order.
how to handle duplicate messages and message ordering.
Amazon Fresh used SQS
- SQS FIFO Queues
producer -> specify message group (tag message) consumer pulls first message in group (tag). no other messages can be processed in that group by any consumer until that message is processed.
messages are processed in order but no guarantee as to which consumer will process the message.
a single fifo has limited throughput. 3K messages/second
Okta - Use Case
user events -> apache storm -> streaming analysis
durable scalable persisten message replay stream analysis
message are not independent
- need to do analysis to detect anomolies and trigger alerts
kineses data streams (like apache kafka)
you need to specify shards
producer -> message A (tag with partition key) -> partition key is hashed to determine which partition to send message to.
duplicates will be sent to same shard.
consumers still have to detect duplicates
use KCL (kinesis client library) to track which consumer should process which messages and other things.
okta events -> sent to kinesis stream -> apache storm -> kinesis data firehose -> s3 -> redshirt -> aws lambda -> amazon sns
Edmunds Use Case
- sells new/used cars
car dealer, oem, franchises -> source systems -> message platform -> target systems -> website
- durable scalable persistent always available
- minimal latency
Amazon SNS Topics
send something and deliver to multiple destinations. pub/sub
producer -> SNS topics -> can push to different destinations destinations:
- SQS queue
- http/s endpoint (like a webhook)
- you can control the rate of retries
- Mobile App
You can configure filters:
- decisions can be made based on attributes in message.
publisher -> A SNS topic (stored across multi-AZ, ack immediately. same low latencty regardless of # of destinations) -> fanout to destinations
- send copy to each destination
f(x) filter can prevent message from going to destination
- retries can be configured based on the destination
- assume it will retry forever until success