Friday, 13 March 2015

Making OpenStack services wait for GPFS with systemd

A little while ago I posted about how OpenStack Cinder and Glance services fail to start properly when using GPFS for the underlying storage layer.

I looked and asked around if anyone had any suggestions on resolving this as GPFS uses a SysV init script. Someone on the GPFS user group suggested that it should be possible to make a systemd service wait on a SvsV init style script.

I've eventually found time to look into this a bit more and found a solution.

Bear in mind that I'm running this system on CentOS 7, so it should work on RHELs, but might be a bit different for other distributions. Basically, we're going to add additional requirements for the systemd manifests to make this work, what we don't want to do is edit the default manifest files as these may get overwritten by upgrade. Luckily, we can add "local" changes as follows:

cd /etc/systemd/system/
mkdir openstack-glance-api.service.d
cd openstack-glance-api.service.d

Now create a file named gpfs.conf with the following contents:
[Unit]
Requires=gpfs.service

After=gpfs.service

Basically what we are saying here is that in addition to the default settings for Glance API, we also require GPFS to be started. Note that using "Requires" on its own doesn't guarantee that GPFS is started, which is why we also have the "After" setting. We're also using Requires rather than Wants, this means if GPFS is stopped, then the Glance API service will also be stopped.

This example obviously only covers the Glance api, you probably also want to do it for:
  • openstack-cinder-api.service.d
  • openstack-cinder-volume.service.d
  • openstack-glance-api.service.d
  • openstack-glance-registry.service.d
  • openstack-glance-scrubber.service.d
  • openstack-swift-object.service.d
  • (possibly some more Swift services as well)
Once you've created these, you need to also call:
systemctl daemon-reload

Incidentally, Incidentally, we also use a custom GPFS init script as well, specifically in the start section we use:
      for i in `seq 1 24` # Wait upto 2 mins for the IB to be ready (24x5s=120s)
      do
                if grep -q gpfs /proc/mounts
                then
                        break
                fi
                sleep 5

      done

And status is rewritten as:
    status)
      lsmod | grep -c mmfs >/dev/null 2>&1
      if [ $? == 0 ]; then
        grep -q gpfs /proc/mounts
        exit $?
      else
        exit $?
      fi

      ;;

Red Hat OpenStack Technical Workshop slides

Yesterday I spoke at the Red Hat OpensStack Technical Workshop held in London, a couple of people asked about my slides, they are available below:

Tuesday, 10 March 2015

What difference does a name make?

As you may have recently heard, IBM have recently announced that GPFS has a new name. Its now Spectrum Storage.

Its been on the cards for a while that it was getting a new name - in fact it became Elastic Storage for a while, but I believe that was the code name for the rebranding project.

In any case, I'm sure most of us who've used it for a while will continue to call it GPFS!

It will be interesting to see if the Power ESS (Elastic Storage Server) will get renamed as well at some point!