Pulling a RabbitMQ out of a ditch

Ok so its a fairly common occurrence for me to end up with my RabbitMQ broker pretty upset. In this entry I am going to cover the high-points of configuring RabbitMQ, as well as the thump  peter cotton-tail on the head and make him work again.

I run allot of different linux stacks older and newer, some are more production worthy than others, some are scaled down vm’s without allot of ram or disk-space very fertile ground for issues cropping up.

The symptom:  I have pika code that connects to the an exchange and publishes. The Error message I get back is AMQPConnectionError

After stopping and restarting the stack numerious times and still getting the same error. I am getting extremely frustrated.

The logfiles

$ cd /var/log/rabbitmq/
$ ls -altrh *.log
-rw-r--r-- 1 rabbitmq rabbitmq    0 Sep  5 04:02 rabbit@localhost-sasl.log
-rw-r--r-- 1 rabbitmq rabbitmq 1.1M Sep  5 17:22 rabbit@localhost.log
$ ls -latrh *_log
-rw-r--r-- 1 root root 23K Sep  5 17:22 startup_log
-rw-r--r-- 1 root root  56 Sep  5 17:22 shutdown_log

Restart the system while watching the logs
Terminal #1:

$ cd /var/log/rabbitmq/
$ tail -f rabbit@localhost.log

Terminal #3:

$ service rabbitmq-server restart

The log output: shutting down

=INFO REPORT==== 5-Sep-2014::19:22:07 ===
Stopping RabbitMQ

=INFO REPORT==== 5-Sep-2014::19:22:07 ===
stopped TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 5-Sep-2014::19:22:07 ===
    application: rabbit
    exited: stopped
    type: temporary

=INFO REPORT==== 5-Sep-2014::19:22:07 ===
    application: mnesia
    exited: stopped
    type: temporary

=INFO REPORT==== 5-Sep-2014::19:22:07 ===
    application: os_mon
    exited: stopped
    type: temporary

=INFO REPORT==== 5-Sep-2014::19:22:07 ===
Halting Erlang VM

As you can see rabbit, mnesia, and os_mon got stopped. These are the 3 things that make up a RabbitMQ application that runs on a erlang node/virtual-machine.
The log output: starting back up

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
Starting RabbitMQ 3.0.0 on Erlang R12B

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
Memory limit set to 404MB of 1010MB total.

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
Disk free limit set to 1000MB

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
Disk free space insufficient. Free bytes:971988992 Limit:1000000000

=WARNING REPORT==== 5-Sep-2014::19:22:10 ===
disk resource limit alarm set on node rabbit@localhost

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 5-Sep-2014::19:22:10 ===
started TCP Listener on 0.0.0.0:5672

Ok, so lets go fix the Disk free space insufficient problem. There are two ways to deal with this issue: one way is to clean up disk space, the other way is to configure RabbitMQ to use less disk space. Here I am going to go after cleaning up the disk, as having less than a gig free is pretty ridiculous anyway.

After restarting RabbitMQ again there were no warnings.

So did that thump-petter-cotton-tail on the butt?

 

The no-sql dbms

Reference this article is written against the following stack.

  • CentOS 5.6
  • RabbitMQ 3.0.0-1
  • erlang-R12B-5.10.el5
  • apache-2.2.27
  • mod_wsgi-3.4
  • django 1.5

 

Full Disclosure: This blog is mostly subsidized by my paycheck, however I also use affiliate links to Amazon for my book reviews.
RabbitMQ in Action: Distributed Messaging for Everyone