… What the? Ok, so the basic idea here is that we’re going to be creating a cluster, sharing a block device from one server to the other nodes in a cluster as a Global Network Block Device, putting it into an LVM configuration, and formatting the filesystem using Global File System for file locking between nodes… And we’re gonna do this all natively with Red Hat Clustering Suite. This is a good, low-rent implementation of block device sharing in a cluster, where iSCSI or FCP is not available to the hosts. It’s better than NFS because we get real locking mechanisms, and we get our own fencing mechanism for the cluster (which, unfortunately I won’t be covering in this post). I’ve recently had the opportunity to do this as a proof of concept and this is really cool stuff…

There’s a lot of configuration to get this to work appropriately, but once it’s up and running, you’ll find yourself with a full network device cluster to where you can share a common root for data, configuration, applications, etc… I, personally, found this to be an awesome choice for content-controlled environments, where you need to have the same applications, same configurations, same modules, and without a lot of independent, per-server legwork to make that happen. It also simplifies backups because you’ll really only have the one physical disk that needs to be backed up for an entire farm of servers.

This is a small step toward reaching a private cloud for dynamically expandable services like httpd or clustered java services. I’ve also seen this implementation used on clustered databases that don’t require exclusive locks on datafiles during their normal operational states. Additionally, this can easily be implemented for Xen clusters, VMware clusters, KVM clusters, or pretty much any environment that allows access to local storage, and needs it to be presented to multiple hosts as block devices with locking control between the nodes, and will not work with something flimsy like NFS.

I’m gonna jump right to the brass tacks with this one, because I think that it’s easier to understand it when you see the steps required to get it setup… For the purposes of this article, I have created a test environment consisting of a three node cluster on the example.com domain. The servers are named gnbd1.example.com, gnbd2.example.com, gnbd3.example.com. I have attached a 10GB disk to gnbd1.example.com and made it accessible as /dev/sdb, so gnbd1.example.com will be our static GNBD server (in that it has the physical media directly attached), and the other two nodes will be our GNBD clients.

This target audience for this article is an existing RHEL administrator, with familiarity in RHEL 5.3+ (my example was set up on RHEL 5.5 x64)… I will not explain remedial steps, such as configuring your yum repositories for RHCS packages. Also, a basic working knowledge of clustering and specifically Red Hat Clustering is to be expected when reading this article. There is a great deal of documentation on the Red Hat site regarding clustering, so if you are unfamiliar with it, I would encourage you to stop by their site first.

Enough of that… The bare metal:

1. (All nodes) First step is to install the necessary RPM’s..

# yum -y install modcluster rgmanager gfs2 gfs2-utils lvm2-cluster cman gnbd kmod-gnbd

2. (gnbd1.example.com) Second step is to create a cluster on gnbd1

# ccs_tool create GNBD_CLUSTER1

3. (gnbd1.example.com) Create the fencing device. For the purposes of this example, I will use manual fencing:

# ccs_tool addfence -C manual fence_manual

4. (gnbd1.example.com) Add the nodes with manual fencing. I chose to use manual fencing for this process because, as a paranoid administrator, I want to be able to physically/manually fence a server that is having troubles so that I can see what’s going on with it…

# ccs_tool addnode -C gnbd1 -n 1 -v 1 -f manual
# ccs_tool addnode -C gnbd2 -n 2 -v 1 -f manual
# ccs_tool addnode -C gnbd3 -n 3 -v 1 -f manual

5. (All nodes) On each node, in /etc/hosts, remove the hostname from the assignment, and add the ip addresses and FQDN and short hostname to /etc/hosts  localhost.localdomain localhost gnbd1.example.com gnbd1 gnbd2.example.com gnbd2 gnbd3.example.com gnbd3

6. (gnbd1.example.com) Copy the cluster configuration to the other nodes:

# scp /etc/cluster/cluster.conf root@gnbd2:/etc/cluster/cluster.conf
# scp /etc/cluster/cluster.conf root@gnbd3:/etc/cluster/cluster.conf

7. (All nodes) Enable LVM2 Clustering

# lvmconf --enable-cluster

8. (All nodes) Start the clustering service, clustering manager, and clustered LVM service:

# /sbin/service cman start && /sbin/service rgmanager start && /sbin/service clvmd start

9. (All nodes) Load the GNBD kernel module. This will give you the gnbd block devices in /dev/gnbd

# modprobe gnbd

9. (gnbd1.example.com) The device that you will be exporting must be physically attached to one of the nodes in the cluster and must not have any other Global Network Block Devices imported on it. For the purposes of this example, gnbd1.example.com will host the physical volume and will serve the exported device, and /dev/sdb will be a 10GB device. But, before we begin doing anything with the device, start by enabling the GNBD server daemon on gnbd1.example.com:

# gnbd_serv

10. (gnbd1.example.com) Now, export the device, giving it a unique name and exporting the UUID

# gnbd_export -d /dev/sdb -e alpha -U

11. (gnbd1.example.com) Create the disk as a LVM physical volume, create a Volume Group on it, and create the Logical Volume. We’re going to put a GFS filesystem on this,┬áso we’ll create the names appropriately.

# pvcreate GNBD_PV /dev/sdb
# vgcreate GNBD_VG /dev/sdb
# lvcreate -L 10G -n gnbd_lv GNBD_VG

12. (gnbd1.example.com) Create the Global File System on the Logical Volume. We’ll use the lock_dlm domain.

--- mkfs.gfs2 -p <locking domain> -t <ClusterName>:<Name of the PV> -j <# of nodes in cluster> <mapper path to LV>
# mkfs.gfs2 -p lock_dlm -t GNBD_CLUSTER1:GNBD_PV -j 3 /dev/mapper/GNBD_VG-gnbd_lv

13. (gnbd1.example.com) Since the device is local to gnbd1.example.com, we can simply create a mount point and mount it as a local disk

# mkdir /mnt/gnbd_gfs && mount /dev/mapper/GNBD_VG-gnbd_lv

14. (gnbd2.example.com, gnbd3.example.com) On nodes gnbd2.example.com and gnbd3.example.com, load the gnbd module and import the device from gnbd1.example.com:

# modprobe gnbd
# gnbd_import -i gnbd1.example.com

15. (All nodes) On each of the nodes you should be able to see the physical volume, volume group, and logical volume of the GNBD device — the clvmd service MUST be running in order for this step to work.

# pvdisplay
# vgdisplay
# lvdisplay


16. (All nodes) Build a new initrd with the gnbd module loaded at boot

mkinitrd -v --with=gnbd /boot/initrd-$(uname -r).gnbd.img $(uname -r)

17. (gnbd2.example.com, gnbd3.example.com) Modify the /etc/init.d/cman startup script to also load. Add the following functions prior to the “start” and “stop” cases respectively:

        /sbin/gnbd_import -i $GNBD_SERVER &>/dev/null
        if [ $? -eq 0 ]
        echo "done"
        echo "failed"
        return 1
        return 0

    /sbin/gnbd_import -R &>/dev/null

18. (gnbd2.example.com, gnbd3.example.com) Modify the “start” and “stop” case of /etc/init.d/cman to execute the gnbd_import and gnbd_remove functions respectively:

echo -n "   Importing GNBD Devices... "
if [ $? -eq 0 ]
echo "done"
echo "failed"
return 1

echo -n "   Removing GNBD imports... "
echo "done"

19. (gnbd2.example.com, gnbd3.example.com) Modify /etc/sysconfig/cman (you may need to create this file — it does not exist by default) and create an entry for the variable GNBD_SERVER. This will be the GNBD server exporting the disk (for the purposes of this, gnbd1.example.com):


20. (All nodes) Modify /etc/rc.local on gnbd1.example.com to startup the gnbd_serv daemon after boot, and export the device

echo "/sbin/gnbd_serv && /sbin/gnbd_export -d /dev/sdb -e alpha -U" >>/etc/rc.local

21. (All nodes) Edit /boot/grub/grub.conf to use the new initrd image that you built with the gnbd module. Something like this:

title Red Hat Enterprise Linux Server with GNBD (2.6.18-194.el5.gnbd)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-194.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-194.el5.gnbd.img

22. Reboot

Feel free to email me with questions, comments, or corrections… dan@rhcedan.com


2 Responses to “Clustering GFS on a GNBD with LVM2 and RHCS”

  1. I’m interested to build similar architecture using 3 servers, with some additional redudancy of data.

    My plan is that all 3 servers export GNBD devices (eg. alpha, beta, gamma) and that they are all used as physical devices for LVM volume group. Logical volumes created would have mirroring copies set to 3 (-m 3), which should result in same data being saved on every server trough GNBD.

    This way servers will share the same data, and there will be redudancy in case any of servers fails. I still didn’t try it out, but hope it will work :)

    Do you have any experience with architecture like that?

    I assume that, before LVM vg and lv is created, on one of servers in cluster I should have GNBD imported from other two servers. Every server should have GNBD imported from other two. Because GNBD are exported using -U option, I assume LVM on each server would recognize that local disk and two GNBDs should be used.

    Best Regards,

    • Hi Nikola,

      Of course, I also responded privately via email, but for the benefit of other readers, I will add some of those details into this reply… Essentially, this is probably an outdated way of handling this type of setup. As I stated in my email, your best bet is probably to provision another (fourth) server and slap whatever majority of your physical disks into that, then create iSCSI exports using something like OpenFiler, or even the base iscsi tools that are available with most Linux distributions. If you’re working with inexpensive hardware, OpenFiler can give you the benefit of creating RAID mirrors and stripes for the redundancy that you need, and the block devices will appear as single-disk units to each of your servers. Additionally, your servers can be provisioned and configured to only use the clustered filesystem backend and will not have to have the complicated configurations needed for GNBD. Clustered LVM w/ GFS2 and iSCSI (or FCoE) shared storage in the background will be your best, most stable bet.


Leave a Reply



You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© 2013 Dan's Blog Suffusion theme by Sayontan Sinha