shyynuxOne day a team-mate told me during a call to poll some messages from SQS, I was like ‘What’s SQS or polling’, they told me it is an asynchronous queue. I was like whaaat (in my mind), I was a new joinee(shy), I felt so dumb. But i hope you don’t (and you shouldn’t), so today we are going to understand -
- What is a messaging queue?
- An exploration of some popular messaging queues.
- Common terms related to messaging queues (SQS), yes polling as well.
💡 What is a messaging queue?
A messaging queue is a way for services to communicate with each other without having to wait for a response(asynchronously). Messages are stored in a queue until they are processed, and each message is only processed once.
Message queues can be used to break down complex tasks into smaller, more manageable ones, and to make systems more scalable and reliable. They are used in serverless and micro-services architectures.
A component called a producer adds a message to the queue. The message is stored on the queue until another component called a consumer retrieves the message and processes it.
A simple example:
Imagine a website that sells products. When a customer places an order, the website creates a message and puts it in a queue. A separate process then picks up the message from the queue and processes the order.
This allows the website to continue processing other orders while the first order is being processed. It also makes the website more reliable, because if one process fails, another process can pick up the messages from the queue and continue processing them.
💡 Some popular messaging queues:
Amazon SQS: Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components.
Rabbit MQ: An open-source message broker that implements the Advanced Message Queuing Protocol (AMQP).
Azure Service Bus: A cloud-based message broker that provides reliable and scalable messaging services for applications.
Apache Kafka: A distributed streaming platform that allows applications to process streams of data in real time.
Apache ActiveMQ: ActiveMQ is a popular messaging service that facilitates disparate data at scale in enterprise systems.
What and when to use should be decided according to the scale, service, tech stack choice etc. factors.
💡 Useful terms associated with SQS
Amazon SQS(Simple queue Service):
As mentioned above, it is a messaging queue service. Our system has producers, consumers using the queue i.e., SQS. The queue stores each message on multiple Amazon SQS servers to ensure that messages are not lost if a server fails.Producer: The service or application that sends messages to a queue.
Consumer: The service that retrieves and processes messages from a queue, which were placed there by producers.
The Queue: The specific messaging queue being used (e.g., SQS has both simple and FIFO options).
Polling: The process of checking a queue for new messages.
By default, short polling is used, where your application sends a ReceiveMessage request to AWS SQS. SQS checks a subset of servers for messages and sends a response immediately, even if no messages are found. On the other hand, long polling waits for new messages, checking all servers for a specific duration, and returns a response if messages are found, otherwise, it sends an empty response when the polling wait time expires.Dead-letter Queue (DLQ): A queue that stores messages that cannot be processed successfully by other source queues. This can happen for various reasons, such as if the message is invalid, too large, or if the consumer crashes. However, they are not created automatically; we need to create them first.
Queue Attributes: These include maximum message size, message retention period, redrive policy, and visibility timeout, which all affect how the messaging queue functions.
I really hope this information will help in understanding MQs better. If you want to read more on this topic I will recommend a few resources -
- https://aws.amazon.com/message-queue/
- Amazon-SQS-official-documentation
- https://www.rabbitmq.com/getstarted.html
Let me know what messaging queue you used for your service and why.
Have a nice day. 🤍
