Friday, 13 March 2015

Making OpenStack services wait for GPFS with systemd

A little while ago I posted about how OpenStack Cinder and Glance services fail to start properly when using GPFS for the underlying storage layer.

I looked and asked around if anyone had any suggestions on resolving this as GPFS uses a SysV init script. Someone on the GPFS user group suggested that it should be possible to make a systemd service wait on a SvsV init style script.

I've eventually found time to look into this a bit more and found a solution.

Bear in mind that I'm running this system on CentOS 7, so it should work on RHELs, but might be a bit different for other distributions. Basically, we're going to add additional requirements for the systemd manifests to make this work, what we don't want to do is edit the default manifest files as these may get overwritten by upgrade. Luckily, we can add "local" changes as follows:

cd /etc/systemd/system/
mkdir openstack-glance-api.service.d
cd openstack-glance-api.service.d

Now create a file named gpfs.conf with the following contents:
[Unit]
Requires=gpfs.service

After=gpfs.service

Basically what we are saying here is that in addition to the default settings for Glance API, we also require GPFS to be started. Note that using "Requires" on its own doesn't guarantee that GPFS is started, which is why we also have the "After" setting. We're also using Requires rather than Wants, this means if GPFS is stopped, then the Glance API service will also be stopped.

This example obviously only covers the Glance api, you probably also want to do it for:
  • openstack-cinder-api.service.d
  • openstack-cinder-volume.service.d
  • openstack-glance-api.service.d
  • openstack-glance-registry.service.d
  • openstack-glance-scrubber.service.d
  • openstack-swift-object.service.d
  • (possibly some more Swift services as well)
Once you've created these, you need to also call:
systemctl daemon-reload

Incidentally, Incidentally, we also use a custom GPFS init script as well, specifically in the start section we use:
      for i in `seq 1 24` # Wait upto 2 mins for the IB to be ready (24x5s=120s)
      do
                if grep -q gpfs /proc/mounts
                then
                        break
                fi
                sleep 5

      done

And status is rewritten as:
    status)
      lsmod | grep -c mmfs >/dev/null 2>&1
      if [ $? == 0 ]; then
        grep -q gpfs /proc/mounts
        exit $?
      else
        exit $?
      fi

      ;;

1 comment:

  1. Hi Simon,

    Thanks for this. Sometimes it takes a few seconds for the nodes to mount so I found you could also consider adding a block device as a requires like:

    [Unit]
    Requires=gpfs.service dev-mystorage.device
    After=gpfs.service dev-mystorage.device

    I found:

    systemctl show openstack-nova-compute.service

    really useful too.

    ReplyDelete