June 2, 2012
Installing and Configuring glusterfs on EL6
I’m always interested in the newest technologies. With the purchase of Gluster by RedHat, I figured it was time to give it a try. We are always looking for new technology that can lower the operations of our sysadmins, maybe Gluster is that option.This guide is heavily based on the administrator’s guide for Gluster.<h3> Installation</h3>All of the gluster packages are in EPEL, so first we need to install that repo on our nodes.<pre class="rootscreen">$ rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-7.noarch.rpm</pre>Then install the glusterfs server:<pre class="rootscreen">$ yum install glusterfs-server -y</pre>Then start the server:<pre class="rootscreen">$ /etc/init.d/glusterd start</pre>For demo purposes only, flush the firewall: <pre class="rootscreen">$ iptables -F</pre><h3> Configuration</h3>And now add the nodes to the gluster system:<pre class="rootscreen">$ gluster peer probe i-0000011aProbe successful$ gluster peer probe i-0000011cProbe successful</pre>Now you can check for the nodes with the status command:<pre class="rootscreen">$ gluster peer statusNumber of Peers: 2Hostname: i-0000011aUuid: 5bdc4f02-4e08-4794-af03-fd624be2d2e0State: Peer in Cluster (Connected)Hostname: i-0000011cUuid: 248be1ba-c5aa-40d1-90e9-ca95a7e31697State: Peer in Cluster (Connected)</pre>In this demo, I decided to make a Distributed Replicated volume. There are many options, but this seemed the best I could see.To create the volume:<pre class="rootscreen">$ gluster volume create test-volume replica 3 transport tcp i-00000119:/exp1 i-0000011a:/exp2 i-0000011c:/exp3</pre>Note, I didn’t make the directories /expX on any of the nodes, they are automatically made for you.To start the volume:<pre class="rootscreen">$ gluster volume start test-volume</pre>To mount the volume, we don’t have to modprobe fuse since it’s built into the 2.6.32 kernel that comes with EL6. You can also use NFS to mount gluster volumes, but I decided to use fuse.<pre class="rootscreen">$ mkdir -p /mnt/glusterfs$ mount -t glusterfs i-0000011a:/test-volume /mnt/glusterfs</pre>YAY! working glusterfs. To confirm that it is working, I copied in a test file, mounted the test-volume on another node in the test cluster as well, and there was my file!<h3> Summary</h3>GlusterFS doesn’t seem too advanced compared to Hadoop or Ceph. If I look in the /expX directories I just see the whole file in there. In the current release, I believe the closest volume configuration we could have to Hadoop or Ceph is Striped Replicated Volumes. But, that volume type is only supported for use as a MapReduce backend.I think GlusterFS would be really cool for a OpenStack back end. Especially since it’s so darn simple. Easily recoverable since the files are stored in plain text. Of course, you would probably want to do striping for the large image sizes of those files. Overall, I feel this was the easiest of the file systems I have tried out. Ceph was a little scary with all the configuration needed. GlusterFS was as simple as just issueing a command to add another server. Of course, does this mean it’ll load balance the files if a server goes away? Don’t really know how that’ll work.
Tags: osg
This guide is heavily based on the administrator's guide for Gluster.
Installation
All of the gluster packages are in EPEL, so first we need to install that repo on our nodes.$ rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-7.noarch.rpm
Then install the glusterfs server:
$ yum install glusterfs-server -y
Then start the server:
$ /etc/init.d/glusterd start
For demo purposes only, flush the firewall:
$ iptables -F
Configuration
And now add the nodes to the gluster system:$ gluster peer probe i-0000011a
Probe successful
$ gluster peer probe i-0000011c
Probe successful
Now you can check for the nodes with the status command:
$ gluster peer status
Number of Peers: 2
Hostname: i-0000011a
Uuid: 5bdc4f02-4e08-4794-af03-fd624be2d2e0
State: Peer in Cluster (Connected)
Hostname: i-0000011c
Uuid: 248be1ba-c5aa-40d1-90e9-ca95a7e31697
State: Peer in Cluster (Connected)
In this demo, I decided to make a Distributed Replicated volume. There are many options, but this seemed the best I could see.
To create the volume:
$ gluster volume create test-volume replica 3 transport tcp i-00000119:/exp1 i-0000011a:/exp2 i-0000011c:/exp3
Note, I didn't make the directories /expX on any of the nodes, they are automatically made for you.
To start the volume:
$ gluster volume start test-volume
To mount the volume, we don't have to modprobe fuse since it's built into the 2.6.32 kernel that comes with EL6. You can also use NFS to mount gluster volumes, but I decided to use fuse.
$ mkdir -p /mnt/glusterfs
$ mount -t glusterfs i-0000011a:/test-volume /mnt/glusterfs
YAY! working glusterfs. To confirm that it is working, I copied in a test file, mounted the test-volume on another node in the test cluster as well, and there was my file!
Summary
GlusterFS doesn't seem too advanced compared to Hadoop or Ceph. If I look in the /expX directories I just see the whole file in there. In the current release, I believe the closest volume configuration we could have to Hadoop or Ceph is Striped Replicated Volumes. But, that volume type is only supported for use as a MapReduce backend.I think GlusterFS would be really cool for a OpenStack back end. Especially since it's so darn simple. Easily recoverable since the files are stored in plain text. Of course, you would probably want to do striping for the large image sizes of those files.
Overall, I feel this was the easiest of the file systems I have tried out. Ceph was a little scary with all the configuration needed. GlusterFS was as simple as just issueing a command to add another server. Of course, does this mean it'll load balance the files if a server goes away? Don't really know how that'll work.