Monday, 8 September 2014

HA RabbitMQ for OpenStack


Next up is to get HA queues for OpenStack across our servers, this is pretty simple to get working. It does need haproxy and keepalived working, as I've already done that, I just need to add the config to haproxy after rabbitmq is installed.

Again using xcat, we want some extra packages:
epel/erlang
epel/rabbitmq-server

On each node, configure the /etc/rabbitmq/rabbitmq-env.conf file, I'm doing this to bind the rabbitmq to the IP address on the VLAN tagged 40GbE network I'm using for OpenStack management traffic:
RABBITMQ_NODE_IP_ADDRESS=10.30.21.64
RABBITMQ_NODE_PORT=5672

Obviously the IP address is different on different systems.

On one of the nodes start the rabbitmq-server.

Enable queues to be mirrored queued by default:
rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}'

Next you need to sync the erlang cookie to all your nodes (copy /var/lib/rabbitmq/.erlang.cookie to each node).

On the next server node start rabbitmq, stop the queue, add to the cluster and restart the queue:
service rabbitmq-server start
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@server1
rabbitmqctl start_app

Check that the queue policy is visible from the second node:
% rabbitmqctl list_policies
Listing policies ...
/ HA ^(?!amq\\.).* {"ha-mode":"all"} 0

...done.

At this point, we probably want rabbitmq to be able to start at boot time and join the cluster, so create /etc/rabbitmq/rabbitmq.config on all nodes of the form:
[{rabbit,
  [{cluster_nodes, {['rabbit@server1', 'rabbit@server2'], ram}}]}].

haproxy config

Assuming you already have haproxy installed and configured at least basically, something like the following should be added to the haproxy config file:
listen rabbitmq 10.30.20.52:5672
    mode tcp
    balance round robin
    option tcpka
    server server1-osmgmt 10.30.21.64:5672 check inter 5s rise 2 fall 3
    server server2-osmgmt 10.30.21.66:5672 check inter 5s rise 2 fall 3

Once this is setup in haproxy, reload the haproxy config and then point the various OpenStack components at the HA IP address (or preferably the name ;-)).

Well, hopefully it should all work, I don't have quite enough of OpenStack up at the time of writing to test it all!

Rabbitmq authentication

By default there is a guest rabbitmq user, change the default guest password for this to something random:
rabbitmqctl change_password guest `openssl rand -hex 10`

We also want to create a user for OpenStack to use, generate a password (and remember it!) and then create the user:
rabbitmqctl add_user <USERNAME> <PASSWORD>

DR'ing a node

If you lose a node (say DR!), on one of the other rabbit cluster nodes, run something like:
rabbitmqctl forget_cluster_node rabbit@server2

Then follow the steps to add server2 back into the cluster.

If you reinstall a node which was already part of the cluster (or lose the contents of /var/lib/rabbit), it should be possible to rejoin the cluster by creating /etc/rabbitmq/rabbitmq.config, start the daemon, then:
rabbitmqctl stop_app
rabbitmqctl change_cluster_node_type disc
rabbitmqctl start_app

Note that if you don't change the node cluster type, then it will be a ram only cluster node and the queue won't be written to disk. 

No comments:

Post a Comment