Installing RabbitMQ & Celery

RabbitMQ & Celery Tutorials

Before we move on, let's get familiar with a couple of terms.

Picture from slides.com.

1. RabbitMQ - Message broker server built on the Advanced Message Queuing Protocol (AMQP). RabbitMQ is written in Erlang. It's responsible queuing up tasks and scheduling them.
2. Celery - Task queue that is built on an asynchronous message passing system. Celery is written in Python. It can be used as a wrapper for Python API to interact with RabbitMQ.
3. Celeryd - Part of the Celery package and it is the worker that actually runs the task.
4. Producer (Publisher) - A program that sends messages.
5. Consumer - A program that mostly waits to receive messages.

RabbitMQ and Celery work together to execute a code sometime later when resources are available.

This video explains RabbitMQ's basic concepts very well.

1. It starts with Publisher/Message Broker (Queue)/Consumer.
2. It also explains how the Queue works.
3. It addresses how we can guarantee the delivery of message (reliable delivery):
   1. The Broker send confirmation to the Producer when it successfully stored the message received.
   2. The Consumer send Ack to the Broker when it processed the message. At the Acknowledgement from the Consumer, the Broker removes the stored message from the queue.
4. When the message system may fail : potential message duplication.
5. We can scale the Consumers.
6. Then, the video introduces the Exchange which is required by Advanced Message Queuing Protocol (AMQP).
7. We have three types of bindings which tells routing behavior from Exchange to the queues
   1. Fanout Exchange - ALWAYS deliver messages to binded queues REGARDLESS of bindings
   2. Direct Exchange - ONLY delivers messages to binded queues WHEN routing-key MATCHES bindings
   3. Topic Exchange - can use wile card for bindings */#

We'll work on Ubuntu 14.04. RabbitMQ installation on Ubuntu is as simple as this:

$ sudo apt-get install rabbitmq-server

Rabbitmq is set to start automatically after it's installed.

To see the information about the installed RabbitMQ such as version:

$ sudo rabbitmqctl status
Status of node rabbit@k ...
[{pid,5135},
 {running_applications,[{rabbit,"RabbitMQ","3.2.4"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,35396072},
          {connection_procs,2704},
          {queue_procs,5408},
          {plugins,0},
          {other_proc,13384560},
          {mnesia,60688},
          {mgmt_db,0},
          {msg_index,28112},
          {other_ets,761752},
          {binary,8632},
          {code,16522377},
          {atom,594537},
          {other_system,4027302}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1487224832},
 {disk_free_limit,50000000},
 {disk_free,439562670080},
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
 {processes,[{limit,1048576},{used,123}]},
 {run_queue,0},
 {uptime,6387}]
...done.

To stop the rabbitmq:

$ sudo rabbitmqctl stop
[sudo] password for k: 
Stopping and halting node rabbit@k ...
...done.

Issue the status command again to see that it's really stopped.

Error: unable to connect to node rabbit@k: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@k]

hosts, their running nodes and ports:
- k: [{rabbitmqctl8566,53150}]

current node details:
- node name: rabbitmqctl8566@k
- home dir: /var/lib/rabbitmq
- cookie hash: 37CWzJUtjVqK+RSWTjgfqg==

To start it again, the recommended method is

$ sudo invoke-rc.d rabbitmq-server start
 * Starting message broker rabbitmq-server  

If we check the status again:

$ sudo rabbitmqctl status
Status of node rabbit@k ...
[{pid,8683},
 {running_applications,[{rabbit,"RabbitMQ","3.2.4"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,35605480},
          {connection_procs,2704},
          {queue_procs,5408},
          {plugins,0},
          {other_proc,13607584},
          {mnesia,60128},
          {mgmt_db,0},
          {msg_index,20888},
          {other_ets,755712},
          {binary,7488},
          {code,16522377},
          {atom,594537},
          {other_system,4028654}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1487224832},
 {disk_free_limit,50000000},
 {disk_free,439561547776},
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
 {processes,[{limit,1048576},{used,123}]},
 {run_queue,0},
 {uptime,56}]
...done.

We may want to create a directory where we can implement our new system:

$ mkdir ~/TEST/MQ
$ cd ~/TEST/MQ

Now we install Celery:

$ sudo apt-get update
$ sudo pip install celery

$ which celery
/usr/local/bin/celery

$ celery --version
3.1.13 (Cipater)

We're going to use Celery later after we've done some practice with RabbitMQ!

RabbitMQ & Celery Tutorials