Distribute task or commands with bash to different workers

Message bus systems are an easy way to distribute tasks.

RabbitMQ is a message broker and there are command line tools too to interact with queues.

If you need help to install a RabbitMQ service,

To install these tools call:


apt-get install amqp-tools

The two main methods are:

• amqp-consume
• amqp-publish

First one is a blocking call that waits for incoming messages.

Second one is a call to put a message on a queue.

So we do have "workers" that consume a queue and "masters" that are publishing tasks.

A worker would look like:


amqp-consume -s 127.0.0.1:5672 -q "test" -e "amq.topic" --vhost "/" -r "worker1" --username=guest --password=guest -d ~/onmessage.sh

The parameters are:


Usage: amqp-consume [-dA?] [-s|--server=hostnameort] [--vhost=vhost]
[--username=username] [--password=password]
[-q|--queue=queue] [-e|--exchange=exchange]
[-r|--routing-key=routing key] [-d|--declare] [-A|--no-ack]
[-?|--help] [--usage] [OPTIONS]... <command> <args>

So if a message is dropped in the queue "test" and has the routing key "worker1" the command "~/onmessage.sh" is called.

The onmessage script might look like this:


nano ~/onmessage.sh && chmod +x ~/onmessage.sh

With content:


read line
echo "Message: $line"

amqp-consume pipes the content of the message therefore we cannot work with parameters ($1,$2,...) but with read to save the stream into a variable called "line".

To publish a message to the "worker1" following command has to be called:


amqp-publish -e "amq.topic" -r "worker1" -b "this is a test message"

The parameters are:


Usage: amqp-publish [OPTIONS]...
-e, --exchange=exchange the exchange to publish to
-r, --routing-key=routing key the routing key to publish with
-p, --persistent use the persistent delivery mode
-C, --content-type=content type the content-type for the message
-E, --content-encoding=content encoding the content-encoding for the message
-b, --body=body specify the message body

Connection options
-s, --server=hostnameort the AMQP server to connect to
--vhost=vhost the vhost to use when connecting
--username=username the username to login with
--password=password the password to login with

I use this pattern for a download service.

I send urls to workers that wget the target.

You can even put more than one worker on a queue - so the work load is distributed.

Or define a second queue were the workers publish results to implement asynchronous method calls.

I like the amqp-tools because they can be used with any tool or language.

I like the ampq protocol. I use zeromq because it is easier than rabbitmq.

"Easier" depends on the requirements.

RabbitMQ is a service providing AMPQ (with persistence and clustering) where as ZeroMQ provides a Peer-to-Peer implementation of AMPQ.

So it depends on what you want to do with AMPQ.

What exactly are you trying to do?  If you just want to run a bunch of commands in parallel (including on different hosts), GNU Parallel is the simplest way, I've found.  For fancier queuing schemes I've never used any of the MQ things but have used Redis, which is pretty well set up for that.