8. Messaging

8.1. Overview

Messaging enables PDC to send out useful informations to message bus so that other systems can subscribe and deal with them.

Current design is based on Django Middleware system, by implementing our MsgMiddleware, we could initialize a message queue for every incoming request during the process_request, and push generated messages into it while processing the request in the view, if no error occurs, before response get returned to user, we pop out all the messages and invoke Messenger to send them to the Message Bus.

PDC Messaging Overview and Dataflow:

                 User Request           User Response
                      +                      ^
+-----+               |                      |
| PDC |               |                      |
+-----+-----------------------------------------------------------------+
|                     v                      |                          |
|   +---------------+ |                      |                          |
|   | MsgMiddleware | |                      ^                          |
|   +---------------+-------------------------------+   +-----------+   |
|   |                 |                      |      |   | Messenger |   |
|   |     +-----------v-+ Init       +-------+----+ |   +-----------+   |      Message
|   |     | Process Req +->+      +-->Process Rsp +----->           +--------> Bus
|   |     +-----------+-+  |      |  +------------+ |   | +send_msg |   |      Server
|   |                 |    |      ^ Dequeue  |      |   |           |   |
|   +----------------------------------------^------+   +-----------+   |
|                     |    |      |          |                          |
|                     | +--v------+---+      |                          |
|                     | |  Msg Queue  |      |                          |
|   +--------+        | +----+--------+      ^                          |
|   |  View  |        |      ^ Enqueue       |                          |
|   +--------+--------------------------------------+                   |
|   |                 |      |               |      |                   |
|   |                 v -----+-------------->+      |                   |
|   |                                               |                   |
|   +-----------------------------------------------+                   |
+-----------------------------------------------------------------------+

8.2. Supported Messengers

                                     +----------------------------------+
                                     |           PDC Messenger          |
                                     +----------------------------------+
                                     |  + __init__()                    |
                                     |  + send_message(topic, msg)      |
                                     +----------------------------------+
                                         ^    ^       ^        ^    ^
                                         |    |       |        |    |
               +-------------------------+    |       |        |    +-------------------------+
               |                   +--------->+       |        +<---------+                   |
               |                   |                  |                   |                   |
+--------------+-+ +---------------+----+ +-----------+--------+ +--------+--------------+ +--+--------------------+
| DummyMessenger | |  FedmsgMessenger   | |  KombuMessenger    | |    ProtonMessenger    | |     StompMessenger    |
+----------------+ +--------------------+ +--------------------+ +-----------------------+ +-----------------------+

There are five messengers that PDC provided, you should choose which one to use according to your messaging infrastructure. Also you could write your own messenger based on your own requirements.

Once you got the answer what left is to configure MESSAGE_BUS item in the settings file accordingly.

Following is the brief introduction of each messenger along with their settings examples, it will help you to know which one to use and how to configure it as well.

1. DummyMessenger (Default)

   If you do not need to send messages out, you could just leave the MLP (Messaging Library Package) to empty string to use the DummyMessenger.

   e.g.

   # Messaging Bus Config
   MESSAGE_BUS = {
       # MLP: Messaging Library Package
       'MLP': '',
   }
   

2. FedmsgMessenger

   If you want to send PDC messages to Fedora Infrastructure Message Bus, you should choose FedmsgMessenger by setting the MLP to ‘fedmsg’.

   fedmsg needs to be installed.

   FedmsgMessenger config example:

   # Messaging Bus Config
   MESSAGE_BUS = {
       # MLP: Messaging Library Package
       'MLP': 'fedmsg',
   }
   

3. KombuMessenger

   Kombu is a messaging library for Python. It supports AMQP(v0.9) and several message server solutions by using pluggable transports.

   NOTE: Not support AMQP(v1.0), so if you’re using some messaging server implements AMQP(v1.0), like ActiveMQ, then

   you should use ProtonMessenger instead.

   kombu and related transport client library(like py-amqp, librabbitmq, or qpid-python) need to be installed.

   To use KombuMessenger, you need to add some more items in the settings, including URL of the broker, EXCHANGE and OPTIONS.

   e.g.

   # Messaging Bus Config
   MESSAGE_BUS = {
       # `kombu` config example:
       'MLP': 'kombu',
       'URL': 'amqp://guest:guest@example.com:5672//',
       'EXCHANGE': {
           'name': 'pdc',
           'type': 'topic',
           'durable': False
       },
       'OPTIONS': {
           # Set these two items to config `kombu` to use ssl.
           'login_method': 'EXTERNAL',
           'ssl': {
               'ca_certs': '',
               'keyfile': '',
               'certfile': '',
               'cert_reqs': 2,  # ssl.CERT_REQUIRED,
           }
       }
   }
   

4. ProtonMessenger

   Qpid-Proton supports AMQP(v1.0), and as if you’re connecting to message server, like ActiveMQ, that supports AMQP(v1.0), ProtonMessenger should be your choice.

   NOTE: current implementation does not support multi endpoints failover.

   qpid-proton needs to be installed.

   ProtonMessenger config example:

   # Messaging Bus Config
   MESSAGE_BUS = {
       'MLP': 'proton',
       'URL': 'amqps://example.com:5671/com.redhat.pdc',
       'CERT_FILE': '',
       'KEY_FILE': '',
   }
   

5. StompMessenger (Recommended)

   STOMP is another message protocol that very simple and easy to implement as a client. So if STOMP over AMQP is acceptable for your application, we recommend you to use StompMessenger as it supports multi-endpoints failover.

   stomp.py needs to be installed.

   StompMessenger config example:

   # Messaging Bus Config
   MESSAGE_BUS = {
       # `stomp` config items:
       'MLP': 'stomp',
       'HOST_AND_PORTS': [
           ('stomp.example1.com', 61613),
           ('stomp.example2.com', 61613),
           ('stomp.example3.com', 61613),
       ],
       'TOPIC': 'pdc',
       'CERT_FILE': '',
       'KEY_FILE': '',
   }
   

8.3. To Be Improved

• Better Error handling
• Message structure refine
• Transaction based Messaging
• Persistent messages that failed to send out
• Non-blocking