Introduction and Overview of RabbitMQ

As we discussed before in the last Chapter, OpenStack is highly modular—each function (like Nova, Neutron, Cinder, etc.) operates as an independent service.

All services depend on two back end functions:

  • Database (MariaDB)

    Database is used to store the persistent data of each OpenStack component.

  • Message Broker (RabbitMQ)

    Message Broker is used to support the communication between each component. Any message broker if only supports AMQP(Advanced Message Queueing Protocol) protocol could be used as a back end.

    In OpenStack here, we use RabbitMQ as the message broker.

Some common RabbitMQ terms and definitions

Term Description
Binding The key or filter parameter used to route from an exchange to a queue.
Consumer An application that uses the received messages.
Exchange Routes producer-published messages to queues using message metadata.
Publisher/Producer Applications that publish messages.
Queue Stores messages until consumed by an application.
Routing key Producer-specified message metadata used by the exchange for message routing.
Vhost A logical division of a RabbitMQ instance, used to segregate applications, queues, and exchanges.

  • Binding

    Binding determines the connection between the queue and the exchange.

  • Consumer

    Consumer is the application or server side to receive messages, here in OpenStack, it represents those service components.

  • Exchange

    Exchange is like an allocation center. Based on Binding the rule, it will route the message to the corresponding queue.(routing key matches the binding key)

  • Publisher/Produce

    Publisher/Produce is the application or client side to send messages, here in OpenStack, it represents those service components.

  • Queue

    Queue is a temporary space to storage the request messages, until customers fetch messages here.

  • Routing Key

    Routing Key is the information produce send with the message, for specifying the destination, the matched queue that the message should be routed to.(routing key matches the binding key)

  • Vhost

    Vhost provides the isolated environment for each project, it just likes a namespace.

RabbitMQ Message Broker Exchange Concepts

The exchange’s interaction with a queue is based on the match between the routing key included in the message and the binding key associated with the queue on the related exchange.

This is called the Binding rule.

Different kinds of exchange determine the match mode of routing key and binding key.

Messaging Patterns in RabbitMQ

Exchange, Binding and Queue are combined in patterns.

Pattern Name Description One-to-Many Callback
Work Queue A single queue with multiple consumers, messages are distributed in round-robin ❌ ❌
Pub/Sub Each consumer has its own queue and receives all messages ✅ ❌
Routing Exact match on routing key ✅ ❌
Topic Fuzzy match on routing key (wildcards allowed) ✅ ❌
Header Routing based on headers ✅ ❌
RPC Request and response messaging (Remote Procedure Call) ❌ (point-to-point) ✅

1. Publish/Subscribe

Multiple consumers receive the same message.

Characteristics:

  • Uses a fanout exchange
  • Messages are broadcast to all bound queues
  • Each consumer has its own queue, isolated from others

Use Cases:

  • Broadcasting notifications
  • Log collection
  • Multiple subsystems responding to events simultaneously

2. Routing

Messages are delivered based on an exact match of the routing key.

Characteristics:

  • Uses a direct exchange
  • Queues are bound with binding keys that exactly match routing keys
  • Multiple queues can be bound with different keys

Use Cases:

  • Processing different types of tasks in multiple modules
  • Log levels separation (e.g., error, info, debug)

3. Topic

Routing key supports pattern matching with wildcards.

Characteristics:

  • Uses a topic exchange
  • Supports wildcards * and #
  • More flexible and complex routing logic

Use Cases:

  • Microservices event bus
  • Logging systems with hierarchical logs like logs.system.error, logs.user.info

4. Header

Routing based on message header attributes.

Characteristics:

  • Uses a headers exchange
  • Does not use routing keys, but matches based on header attributes

Use Cases:

  • Advanced routing policies
  • Attribute-driven message delivery (e.g., region=us, format=json)

5. Work Queue

Multiple consumers share a single queue, and each message is delivered to only one consumer.

Produce sends the message with the routing key to the exchange (only specify the exchange name), then the exchange will based on its types to determine which queue shoud be routed to. Then, customers subscribe the queue to receive the message from produce.

Many customers subscribe the same queue, even if there is no exchange, queues could aslo hand out work to different multiple customers. The work queue distributes requests in chronological order and follow the rule FIFO(first in first out) in the round-robin way.

work queue

Characteristics:

  • Typically uses a direct or default exchange
  • Multiple consumers compete for messages from the same queue
  • Enables load balancing (the idle consumer takes the message)

Use Cases:

  • Background task processing
  • Batch jobs
  • Data import, etc.

6. RPC (Remote Procedure Call)

Client sends a request, server processes it and replies back (pseudo-synchronous).

Characteristics:

  • Uses a direct exchange for the callback and topic exchange for sending
  • Messages contain reply_to and correlation_id properties
  • Responses are sent back to a specified temporary queue

Use Cases:

  • RPC calls between OpenStack components
  • Asynchronous task tracking in Celery

RPC

reply_to

What is reply_to?

Meaning: It tells the server “after processing, send the result to which queue”.

Type: A string representing the name of the callback queue (reply queue).

Purpose: Allows the client to receive the response message.

Analogy: It’s like sending a parcel and writing “please send the result back to my office address” — this address is the reply_to.

correlation_id

What is correlation_id?

Meaning: A unique identifier (ID) that establishes the correspondence between request and response.

Type: Usually a UUID or any unique string.

Purpose: When the client sends multiple requests, it can use the correlation_id to identify which response corresponds to which request.

Analogy: You send multiple questions (letters), each labelled “Question 1”, “Question 2”, etc. The reply includes the same label, so you know which answer matches which question.

Exchanges

Direct

Direct Exchange: The routing keys are matched against a simple string, like “sales.*“. This means that any message with an exact binding key will be routed to this queue (and vice versa). “Direct” exchange type is used when you want messages sent directly from one application/service into another.

In other words, the routing key must be the same as the binding key, one-to-one.

E.g. taskC_create message is routed to taskC_queue queue.

direct

Topic

Topic Exchange: This type of exchange functions like a “fuzzy match”. The routing key of a message and the binding key of a queue can include wildcards (* and #), allowing for more flexible matching. The * wildcard matches a single word, while # matches multiple words.

E.g. A message can be delivered to one or more queues at the same time. Only compute tasks are routed to the queue named compute_tasks, while all tasks, including compute ones, are routed to the queue named all_tasks.

topic

Fanout

Fanout Exchange: It is like a “broadcast”. It sends messages to all bound queues, regardless of the routing key. Simply put, “if it’s bound, it gets the message”. For example, in a logging system, all queues bound to the exchange will receive the message.

Headers

Headers: This exchange type makes use of the message headers to perform the match against the binding arguments of the queue. This is similar to a topic exchange except that there can be multiple key-value headers, and the queue can be matched against any or all. And headers ignore the routing key at all.

Rabbitmq common command

Check the report summary

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
()[root@controller0 /]# rabbitmqctl report
Reporting server status of node rabbit@controller0 ...

Status of node rabbit@controller0 ...
[{pid,701},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.7.23"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.23"},
      {amqp_client,"RabbitMQ AMQP Client","3.7.23"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.23"},
      {rabbit,"RabbitMQ","3.7.23"},
      {mnesia,"MNESIA  CXC 138 12","4.15.6"},
      {rabbit_common,
          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",
          "3.7.23"},
...output omitted...

List users

1
2
3
4
()[root@controller0 /]# rabbitmqctl list_users
Listing users ...
user tags
guest [administrator]

Create users

1
2
3
4
5
6
7
()[root@controller0 /]# rabbitmqctl add_user shy shypassword
Adding user "shy" ...
()[root@controller0 /]# rabbitmqctl list_users
Listing users ...
user tags
shy []
guest [administrator]

Allocate the permission

  • configure: Define the resources that the user could configure.

  • write: Define the resources that the user could write.

  • read: Define the resources that the user could read.

1
2
3
4
5
6
7
()[root@controller0 /]# rabbitmqctl set_permissions shy '.*' '.*' '.*'
Setting permissions for user "shy" in vhost "/" ...
()[root@controller0 /]# rabbitmqctl list_permissions
Listing permissions for vhost "/" ...
user configure write read
shy .* .* .*
guest .* .* .*

Add the tag to users

1
2
3
4
5
6
7
()[root@controller0 /]# rabbitmqctl set_user_tags shy administrator
Setting tags for user "shy" to [administrator] ...
()[root@controller0 /]# rabbitmqctl list_users
Listing users ...
user tags
shy [administrator]
guest [administrator]

List all exchanges

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
()[root@controller0 /]# rabbitmqctl list_exchanges
Listing exchanges for vhost / ...
name type
heat-engine-listener_fanout fanout
manila-share_fanout fanout
engine_fanout fanout
engine_worker_fanout fanout
cinder-volume_fanout fanout
q-server-resource-versions_fanout fanout
openstack topic
amq.rabbitmq.trace topic
amq.fanout fanout
cinder-scheduler_fanout fanout
compute_fanout fanout
amq.match headers
neutron-vo-Port-1.5_fanout fanout
q-plugin_fanout fanout
octavia-rpc_fanout fanout
amq.topic topic
octavia_provisioning_v2_fanout fanout
amq.direct direct
heat topic
amq.headers headers
manila-scheduler_fanout fanout
nova topic
scheduler_fanout fanout
cinder-volume.hostgroup@tripleo_ceph_fanout fanout
conductor_fanout fanout
neutron topic
octavia topic
q-reports-plugin_fanout fanout
 direct

The last one is an implicit direct type exchange.

Its name is an empty string “”.

Every queue is automatically bound to it with a binding key that matches the queue’s name.

List all available queues

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
()[root@controller0 /]# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name messages
compute.computehci0.overcloud.example.com 0
compute.compute1.overcloud.example.com 0
manila-scheduler.hostgroup 0
engine_worker_fanout_c149d365282c4d7986d35683a679ab43 0
reply_95154a0ef0a7494c91bda27f28a2be56 0
cinder-scheduler 0
q-reports-plugin.controller0.overcloud.example.com 0
scheduler_fanout_76ec3c50ae6447c39559b0311268126b 0
compute.compute0.overcloud.example.com 0
reply_61725f7db23a4e7bacecb41f30be9f72 0
conductor_fanout_49fc039d21894497be4fbfd1ecadcce7 0
octavia-rpc 0
q-server-resource-versions.controller0.overcloud.example.com 0
cinder-volume.hostgroup@tripleo_ceph 0
reply_ccfd22e0f82344bb890ff371a0eaa2f9 0
manila-share_fanout_53d504969d5647319932e92f73bb5773 0
manila-share.hostgroup@cephfs 0
conductor.controller0.overcloud.example.com 0
octavia_provisioning_v2 0
q-reports-plugin 0
heat-engine-listener.34703093-ae25-4a65-80db-203f1ebbad36 0
octavia-rpc.controller0 0
engine_fanout_d9489dad34ab4a2e833e25ba5ac9bd43 0
manila-scheduler_fanout_53141319d79548a3a5a23f472026995c 0
engine 0
q-server-resource-versions 0
scheduler.controller0.overcloud.example.com 0
octavia_provisioning_v2_fanout_516bc8a0d4a04e17b02d7a490ffad7e3 0
q-plugin_fanout_82afdb9f58dd46fd9e12d1b9ff981b57 0
q-plugin.controller0.overcloud.example.com 0
manila-share 0
heat-engine-listener_fanout_092d1af0bf034da0aafbd796ef769ec1 0
compute_fanout_3d54889d98a84dcea09b080c23a255f5 0
cinder-volume 0
engine_worker 0
engine.controller0 0
compute_fanout_ef6b7f891f8b482188a1081436cd99b0 0
engine_worker.34703093-ae25-4a65-80db-203f1ebbad36 0
scheduler_fanout_56d907c013304f0b8ea9ab7ed88b6311 0
compute 0
q-server-resource-versions_fanout_1c237cd6f89549e6873ef5d817441b0f 0
scheduler 0
compute_fanout_d20e7a4456e94533b532e1941594ac6f 0
heat-engine-listener 0
octavia_provisioning_v2.controller0 0
q-reports-plugin_fanout_b04fa13918b749eebcc159bb2520934b 0
cinder-scheduler.controller0 0
cinder-volume_fanout_c4f5af48a98e47f5acda0a01681c16a1 0
octavia-rpc_fanout_a68e626ca08943c986a33ec185f2ad12 0
cinder-volume.hostgroup@tripleo_ceph.hostgroup 0
scheduler_fanout_a19884991bd64a278afed93434054535 0
manila-scheduler 0
cinder-volume.hostgroup@tripleo_ceph_fanout_b7c2b8bbdc494c279d6728ed5fe39f94 0
q-reports-plugin_fanout_3a7fa6f49d9349a0ae6e8143ac075f1f 0
q-plugin 0
conductor 0
cinder-scheduler_fanout_6a11c6037338474a9b3d7ce850b9966a 0

List all customers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
()[root@controller0 /]# rabbitmqctl list_consumers
Listing consumers in vhost / ...
queue_name channel_pid consumer_tag ack_required prefetch_count arguments
compute.computehci0.overcloud.example.com <rabbit@controller0.3.1232.0> 2 true 0 []
compute.compute1.overcloud.example.com <rabbit@controller0.3.1206.0> 2 true 0 []
manila-scheduler.hostgroup <rabbit@controller0.3.725.0> 2 true 0 []
engine_worker_fanout_c149d365282c4d7986d35683a679ab43 <rabbit@controller0.3.1063.0> 3 true []
reply_95154a0ef0a7494c91bda27f28a2be56 <rabbit@controller0.3.1117.0> 1 true 0 []
cinder-scheduler <rabbit@controller0.3.763.0> 1 true 0 []
q-reports-plugin.controller0.overcloud.example.com <rabbit@controller0.3.1624.0> 2 true []
q-reports-plugin.controller0.overcloud.example.com <rabbit@controller0.3.1509.0> 2 true []
scheduler_fanout_76ec3c50ae6447c39559b0311268126b <rabbit@controller0.3.1165.0> 3 true []
compute.compute0.overcloud.example.com <rabbit@controller0.3.953.0> 2 true 0 []
reply_61725f7db23a4e7bacecb41f30be9f72 <rabbit@controller0.3.802.0> 1 true 0 []
conductor_fanout_49fc039d21894497be4fbfd1ecadcce7 <rabbit@controller0.3.829.0> 3 true []
octavia-rpc <rabbit@controller0.3.980.0> 1 true 0 []
q-server-resource-versions.controller0.overcloud.example.com <rabbit@controller0.3.1585.0> 2 true 0 []
cinder-volume.hostgroup@tripleo_ceph <rabbit@controller0.3.930.0> 1 true 0 []
cinder-volume.hostgroup@tripleo_ceph <rabbit@controller0.3.904.0> 2 true 0 []
reply_ccfd22e0f82344bb890ff371a0eaa2f9 <rabbit@controller0.3.856.0> 1 true 0 []
manila-share_fanout_53d504969d5647319932e92f73bb5773 <rabbit@controller0.3.696.0> 3 true []
manila-share.hostgroup@cephfs <rabbit@controller0.3.696.0> 2 true 0 []
conductor.controller0.overcloud.example.com <rabbit@controller0.3.829.0> 2 true 0 []
octavia_provisioning_v2 <rabbit@controller0.3.1008.0> 1 true 0 []
q-reports-plugin <rabbit@controller0.3.1624.0> 1 true 0 []
q-reports-plugin <rabbit@controller0.3.1509.0> 1 true 0 []
heat-engine-listener.34703093-ae25-4a65-80db-203f1ebbad36 <rabbit@controller0.3.1035.0> 2 true 0 []
octavia-rpc.controller0 <rabbit@controller0.3.980.0> 2 true 0 []
engine_fanout_d9489dad34ab4a2e833e25ba5ac9bd43 <rabbit@controller0.3.1090.0> 3 true 0 []
manila-scheduler_fanout_53141319d79548a3a5a23f472026995c <rabbit@controller0.3.725.0> 3 true 0 []
engine <rabbit@controller0.3.1090.0> 1 true 0 []
q-server-resource-versions <rabbit@controller0.3.1585.0> 1 true 0 []
scheduler.controller0.overcloud.example.com <rabbit@controller0.3.1164.0> 2 true 0 []
scheduler.controller0.overcloud.example.com <rabbit@controller0.3.1165.0> 2 true 0 []
scheduler.controller0.overcloud.example.com <rabbit@controller0.3.1149.0> 2 true 0 []
octavia_provisioning_v2_fanout_516bc8a0d4a04e17b02d7a490ffad7e3 <rabbit@controller0.3.1008.0> 3 true 0 []
q-plugin_fanout_82afdb9f58dd46fd9e12d1b9ff981b57 <rabbit@controller0.3.1517.0> 3 true []
q-plugin.controller0.overcloud.example.com <rabbit@controller0.3.1517.0> 2 true 0 []
manila-share <rabbit@controller0.3.696.0> 1 true 0 []
heat-engine-listener_fanout_092d1af0bf034da0aafbd796ef769ec1 <rabbit@controller0.3.1035.0> 3 true 0 []
compute_fanout_3d54889d98a84dcea09b080c23a255f5 <rabbit@controller0.3.1232.0> 3 true 0 []
cinder-volume <rabbit@controller0.3.904.0> 1 true 0 []
engine_worker <rabbit@controller0.3.1063.0> 1 true 0 []
engine.controller0 <rabbit@controller0.3.1090.0> 2 true 0 []
compute_fanout_ef6b7f891f8b482188a1081436cd99b0 <rabbit@controller0.3.1206.0> 3 true 0 []
engine_worker.34703093-ae25-4a65-80db-203f1ebbad36 <rabbit@controller0.3.1063.0> 2 true []
scheduler_fanout_56d907c013304f0b8ea9ab7ed88b6311 <rabbit@controller0.3.1149.0> 3 true []
compute <rabbit@controller0.3.1232.0> 1 true 0 []
compute <rabbit@controller0.3.1206.0> 1 true 0 []
compute <rabbit@controller0.3.953.0> 1 true 0 []
q-server-resource-versions_fanout_1c237cd6f89549e6873ef5d817441b0f <rabbit@controller0.3.1585.0> true 0 []
scheduler <rabbit@controller0.3.1165.0> 1 true 0 []
scheduler <rabbit@controller0.3.1164.0> 1 true 0 []
scheduler <rabbit@controller0.3.1149.0> 1 true 0 []
compute_fanout_d20e7a4456e94533b532e1941594ac6f <rabbit@controller0.3.953.0> 3 true 0 []
heat-engine-listener <rabbit@controller0.3.1035.0> 1 true 0 []
octavia_provisioning_v2.controller0 <rabbit@controller0.3.1008.0> 2 true 0 []
q-reports-plugin_fanout_b04fa13918b749eebcc159bb2520934b <rabbit@controller0.3.1624.0> 3 true 0 []
cinder-scheduler.controller0 <rabbit@controller0.3.763.0> 2 true 0 []
cinder-volume_fanout_c4f5af48a98e47f5acda0a01681c16a1 <rabbit@controller0.3.904.0> 3 true []
octavia-rpc_fanout_a68e626ca08943c986a33ec185f2ad12 <rabbit@controller0.3.980.0> 3 true []
cinder-volume.hostgroup@tripleo_ceph.hostgroup <rabbit@controller0.3.930.0> 2 true 0 []
scheduler_fanout_a19884991bd64a278afed93434054535 <rabbit@controller0.3.1164.0> 3 true []
manila-scheduler <rabbit@controller0.3.725.0> 1 true 0 []
cinder-volume.hostgroup@tripleo_ceph_fanout_b7c2b8bbdc494c279d6728ed5fe39f94 <rabbit@controller0.3.930.0> 3 true 0 []
q-reports-plugin_fanout_3a7fa6f49d9349a0ae6e8143ac075f1f <rabbit@controller0.3.1509.0> 3 true 0 []
q-plugin <rabbit@controller0.3.1517.0> 1 true 0 []
conductor <rabbit@controller0.3.829.0> 1 true 0 []
cinder-scheduler_fanout_6a11c6037338474a9b3d7ce850b9966a <rabbit@controller0.3.763.0> 3 true 0 []

Trace on the RabbitMQ message

This feature can be used to track all messages flowing through the system, whether incoming or outgoing. In other words, it acts like a supervisor, logging every message in motion. This capability is based on the firehose mechanism.

The name firehose comes from its function—it behaves like a firehose, spraying out all messages so you can see each one passing through. This feature is very useful during development and debugging, but as it records all messages, it may impact performance.

Enable the trace

1
rabbitmqctl trace_on

Once enabled, all messages entering the RabbitMQ system are copied to a special exchange called amq.rabbitmq.trace. This allows you to observe the entire message flow within that exchange, which is particularly useful for debugging and monitoring.

Performance impact: Enabling the Firehose Tracer increases system load, as it must handle and replicate every message. Therefore, it is recommended to enable it only when debugging is necessary, and to disable it promptly once debugging is complete.

Monitoring amq.rabbitmq.trace: After enabling tracing, you can monitor system activity by inspecting the messages in the amq.rabbitmq.trace exchange. This exchange contains copies of all messages, making it easier to analyse system behaviour.

Disable the trace

1
rabbitmqctl trace_off