Running a RabbitMQ server

[wlanboy]

RabbitMQ is a open source message broker software that implements the AMQP protocol (Advanced Message Queuing Protocol).

If you are not sure what you might do with a message broker you should read their getting started guide.

It is more than just a publish & subscribe pattern that can be implemented through RabbitMQ.

If you need high availability and a high flexibility for routing messages you should consider RabbitMQ as your messaging plattform.

Time to install the server.

1. Add key of RabbitMQ repro:


wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
apt-key add rabbitmq-signing-key-public.asc

2. Add sources:


nano /etc/apt/sources.list

Content:


deb http://www.rabbitmq.com/debian/ testing main

3. Install server


apt-get update && apt-get install rabbitmq-server

4. Add addons:


rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins enable rabbitmq_web_stomp

5. Login into web fronend

Start your browser and call "http://ip-of-your-server:15672"

The GUI is great.

Use the login "guest" with the password "guest" to login.

Please change the login afterwards.



You can manage the whole server through the GUI but you can use the console tool "rabbitmqctl" too.

Some common use cases are:

• rabbitmqctl list_users
  List all users and their attached roles.
• rabbitmqctl add_vhost /test
  Add a new vhost to the server.
  Vhosts are used to separate queues and users.
  You can e.g. limit the access rights.
• rabbitmqctl add_user username password
  Add a user.
• rabbitmqctl set_permissions -p /vhost username ".*" ".*" ".*"
  Grand access to a vhost.

5. Clustering

If you want to cluster RabbitMQ you have to sync the hosts file and the "erlang cookie":


nano /etc/hosts

Check that each hostname does have the right external ip address.


#master
cat /var/lib/rabbitmq/.erlang.cookie

#slaves
echo "the-cookie" > /var/lib/rabbitmq/.erlang.cookie
You have to stop RabbitMQ and copy the conent of the master cookie to all slave cookies

Double check the value because this file should not contain any whitespaces or carriage returns.

And join each slave to the cluster


rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@master
rabbitmqctl start_app

You can check the status of the cluster by running following command


rabbitmqctl cluster_status

6. Clients

And there are tons of clients - from C++ to PHP or Javascript.

I have wrote a little chat application that is based on the JS client lib and using websockets for the communication.

My main usage of the RabbitMQ server are:

• Managing of payloads
  I do have two masters that are generating jobs for my twitter/web archive.
  The RabbitMQ server is distributing the jobs through all workers.
  SSH/keys are not needed because if a worker is ready it is connecting to the RabbitMQ server. 
  So master and slave do not know each other.
  I can add and remove slaves at will and ip changes do not mean anything.
  RabbitMQ is persisting the messages and routing the job messages to free workers (load balancing) too.
• Log handling
  If things go wrong I get a lot of log entries about services/routings/domains/websites that are not available.
  So my RabbitMQ server works as a buffer to ensure that my database or my email servers are not overloaded.
  Perfect tool to queue and load balancing stuff.

If you want to play around you can look at the PHP based master and slave examples (python: M/S).

 

[tchen]

For log buffering, be sure to monitor for high water mark events. Rabbitmq is all fine and good until it starts blocking inputs . Mind you, it's not RMq's fault but it does happen if you're not draining fast enough.

 

[wlanboy]

For log buffering, be sure to monitor for high water mark events. Rabbitmq is all fine and good until it starts blocking inputs . Mind you, it's not RMq's fault but it does happen if you're not draining fast enough.

Yup some clients block if their connection pool is not working as it should.

But you can use a fanout exchange to distribute the message to different queues you won't run into blocking issues, because it helps quite well if each consumer does have its own queue.

 

[tchen]

Fanout?  I'm not entirely sure we're talking about the same thing.  The HWM governs when the flow control kicks in for publishers.  A fanout would merely consume N times more memory, faster.  The default HWM is set to 40% of the system memory, so on a measly lowendbox we're not talking much.  Add a crappy disk IO and it won't be able to page out fast enough.

I recall that some logging systems are designed specifically to drop logs on-the-floor and avoid queueing specifically for this reason.  Just something I figured someone should point out.

 

[splitice]

RabbitMQ is an interesting peice of software, a while ago I investigated it as an alternative to RELP for the transport layer of our system. Unfortunately its got quite a high learning curve and was put on the 'later pile'.

Thanks for the tutorial however, I am going to do a bit more investigation as a result.

For those curious, we use php-resque for job queueing and RELP for log transport (central location). Fortunately a single gbit machine is more than capable of receiving the log stream. Otherwise RabbitMQ would be pretty useful I suspect (for multiple drains).

 

[wlanboy]

Fanout?  I'm not entirely sure we're talking about the same thing.  The HWM governs when the flow control kicks in for publishers.  A fanout would merely consume N times more memory, faster.  The default HWM is set to 40% of the system memory, so on a measly lowendbox we're not talking much.  Add a crappy disk IO and it won't be able to page out fast enough.

I recall that some logging systems are designed specifically to drop logs on-the-floor and avoid queueing specifically for this reason.  Just something I figured someone should point out.

I know that a lot of my tutorials is about doing as much as possible with the lowest invest - like my last series of posts about doing everthing with bash.

But I did not mention any low-end words on this one.

I am running my command RabbitMQ server on a 2GB vps.

But if you only have some million messages per day a 256 MB vps is sufficient.

A small number example: 20 messages per second is 1728000 messages per day.

I am not replacing syslog with RabbitMQ - it is my uptime script so I do have per second and not per millisecond units.

Some words on exchanges:

• Fanout exchanges are very fast because they have no routing to process
• If you have a finite list of routing keys then fanout exchanges might be the right fit because of the 1:1 mapping of exchange per routing  key.
• If you have a quite high number of routing keys topic exchanges are better.
• But for topic routing performance decreases by the number of bindings used
• Direct exchanges are a faster form of topic exchanges if you do not work wild cards on your topics

I know that a high number of exchanges take up memory but it helps to transfer the load.

Exchange-to-exchange bindings improve decoupling which helps increasing the performance too.

RabbitMQ is an interesting peice of software, a while ago I investigated it as an alternative to RELP for the transport layer of our system. Unfortunately its got quite a high learning curve and was put on the 'later pile'.

Thanks for the tutorial however, I am going to do a bit more investigation as a result.

For those curious, we use php-resque for job queueing and RELP for log transport (central location). Fortunately a single gbit machine is more than capable of receiving the log stream. Otherwise RabbitMQ would be pretty useful I suspect (for multiple drains).

You are right, my wording was not that great as I talked about "logging".

I would not recommend RabbitMQ for syslogs because you then have to think about writing your own plugins which is indeed a high learning curve.

But RabbitMQ is an easy start if someone needs a messaging bus for free which is running out of the box.

You get load balancing and a lot of routing options for free which is suitable for any private project.

 

[splitice]

I am pretty sure rsyslog includes a RabitMQ plugin in v7 and for v6 there is a github plugin. Although I could be wrong.

 

[wlanboy]

I am pretty sure rsyslog includes a RabitMQ plugin in v7 and for v6 there is a github plugin. Although I could be wrong.

It was 7.4.0:

contributed module (not project-supported) omrabbitmq was added

 

[peterw]

If you want to play around you can look at the PHP based master and slave examples.

Thank you for accepting PHP. I will look into the examples to play around with fire&forget tasks. I wanted to be able to call functions without the need of knowing ips and logins. RabbitMQ solves this problem because every communication partner is only talking to the server.

 

[tchen]

I know that a lot of my tutorials is about doing as much as possible with the lowest invest - like my last series of posts about doing everthing with bash.

But I did not mention any low-end words on this one.

My bad.  Sometimes I forget which site I'm reading   Thanks for bringing up RMQ.

 

[splitice]

Now using RabbitMQ in the log processing loop to feed some logstash workers

Pretty awesome so far.