RabbitMQ is yet another messaging queue. It is open source, robust, easy, reliable, portable and scalable with high throughput and latency. RabbitMQ is developed with Erlang. RabbitMQ is based on AMQP, Advanced Message Queuing Protocol.
AMQP is an open standard that defines a protocol for system to exchange messages. AMQP defines both the interaction between the parties of the system as well as low level representation of messages that are being exchanged ie: wire format for messages.
Before going into details of RabbitMQ, let’s cover some basic glossary, definitions and use cases. A message is an entity that is transferred between different components of an architecture or infrastructure. Messaging or message queuing is a style of communication between applications or components of a system or systems that enables loose coupling architecture. Message can have several formats, from text to serialized binary. Messaging are sending and receiving messages between different parts of the system.
A messaging infrastructure provides several benefits as follows:
Interoperability: Your infrastructure can have different components in different technologies and languages. Having a messaging queue in the middle provides interoperability so independent components work seamlessly, unaware of other platforms.
Loose coupling: It is one of the best practices both in programming and in software design to implement lose coupling. A messaging queue helps achieve loose coupling easily. You can break your software into components which won’t have any dependencies between and can run independently from each other.
Some of the benefits of loosely coupled producer & consumer paradigm is as follows:
- Producers or consumers fail without impacting each other.
- Performance of each side to have no affect on each other.
- Number of instances of producers and consumers can grow and shrink to enable scalability.
- Producers or consumers can be located on different locations.
Scalability: Loose coupling brings scalability along. If you design your application that you can loose couple your application components, you can scale it easily.
Portable: You can port your messaging queue to any architecture.
Reliability: Most Messaging Queues provides reliable solutions, so that clients won’t lose any data. Messaging queues can work with many data stores such as databases, file systems or caches. It is essential to persist the data which is delivered to queue to a persistent storage in order to avoid data loss. Moreover, there is several persistence schemes implemented in different messaging queues, and interesting one that ActiveMQ uses is temp files via a message dispatcher, if there is no persistence setup. We would like to make sure that when we send a message, messaging queue receives our message, and likewise, we would like to ensure that when we ask for a message, we get a message and this message is removed from the queue. These operations are done with acknowledgements, similar to TCP.
Support for Protocols: most of the message queues support multiple transport protocols such as TCP, HTTP, and STOMP etc.
Enter Producer-Consumer Problem.
Producer-consumer problem is a classical problem in computer science that deals with synchronization between processes. We have a producer which produces and feeds data, and then we have a consumer that consumes data. Producer and consumer share a common buffer to send and receive data respectively. Producer –consumer has a great benefit in parallelization of work, while producers are producing data, consumers can consume at the same time. Moreover, you can have multiple producers as well as multiple consumers. Wrong implementations of this concept can cause several problems such as deadlocks, race conditions. Producer-consumer problem can use a Queue as the buffer storage. We can have a blocking queue for this purpose; also, synchronization should be handled carefully, in order to avoid inconsistent states. Here is the usage of a Blocking queue in a producer – consumer problem, while producers are sending data to a blocking queue, consumers can request messages from the queue and process them for whatever purpose they need to. In order to avoid overflow, queue can block producers from overloading the queue with too much data, this is also called throttling. In order to avoid overflows and overloading of the message queue, producers are blocked until there are more resources available. Likewise for consumers, if the queue is empty, consumers just wait until there is a new message on the queue to be processed.
According the AMQP following are the core concept:
Broker is a middleware application that receives messages produced by publishers and delivers them to consumers or to another broker.
Virtual host provides tenancy to clients and provides security.
Channel is an abstraction layer that provides connectivity between broker and producer/consumer. Channels enable isolation of connectivity so connection don’t interfere with each other. It is possible to have multiple channel within a single connection and this increases performance since creating and destroying TCP connections are expensive.
Exchange is the initial destination that messages flow to. Routing rules are applied in change for their destinations. Routing rules can be direct (point to point), topic (publish-subscribe) and fanout (multicast).
Queue is the final destination of the messages ready to be consumed.
Binding is the virtual connection between exchange and a queue that enables messages to flow.
There can be different topologies for queuing.
Applications that need to use RabbitMQ need to establish persistent connections to it. When connection is established, logical channels can be used for sending and receiving messages. Creating a connection is an expensive operation therefore, one a connection is setup, and multiple channels can be used to communicate with the broker.
Queue and Messages can have the following properties:
Durable: If you want the queue to stay declared even after a broker restart.
AutoDelete: If you want to keep the queue even if it is not being consumed anymore.
Exclusive: If you want the queue to be consumable by other connections.
Message properties are as follows:
contentType: Messages go as byte arrays between systems. Content type of the message can be sent along with the message, such as JSON.
contentEncoding: You can use a specific encoding when serializing messages into byte arrays.
messageId: Message ID is used to enable tracing messages in a system. Usually message IDs should be unique. UUID or GUID can be used for message ID.
deliveryMode: This parameter can have non-persistent : 1 or persistent : 2 values. If the message is persistent, it will be written to the disk and message won’t be lost.
autoAck: Once the consumer receives messages, it will inform the broker of the delivery and messages can be removed safely.