RPi GlusterFS Install

GlusterFS and RPi

Overview

We continue building out our IoT Pi project by adding volume storage that our Kubernetes cluster can use. This is a brief summary of the steps to create a glusterfs endpoint, Mounting the GlusterFS system on our Raspberry PIs and making them available to our Swarm.
For this exercise we are using 1 usb stick in every PI (We have found the same sized stick in each pi, in the same USB slot works best).

Mount USB drives

---

On every Pi

• sudo su - – be careful at this point you are running as root
• mkdir /mnt/usb0 – Create a directory to mount the USB stick
• mount /dev/sda1 /mnt/usb0 – Assuming you don’t have anything else in the RPi’s USB slots, your USB storage should be located in /dev/sda1, if you have more than one usb slot in use, you will need to find the correct device path. You can find where your usb stick is located by running blkid.

Now make them mount on reboot

• echo "/dev/sda1 /mnt/usb0 auto defaults,user 0 1" | tee -a /etc/fstab

Install GlusterFS On all your Pi’s

---

On Each Pi:

• vi /etc/apt/sources.list
• make sure the source repo is uncommented. Typically the last line in the file on Hypriot
• apt-get update && apt-get install -y glusterfs-server

Configure GlusterFS

---

Pick a "master" for glusterfs (one of your Pis)

On the master:

• Probe each of the peers

• Create volume(s) as desired
• gluster volume create data replica 2 transport tcp pimaster:/mnt/usb0/data pinode1:/mnt/usb0/data pinode2:/mnt/usb0/data pinode3:/mnt/usb0/data
• data is the name of the volume
• 2 relicas means the data is written to 2 peers and distributed over 4, so “file1” stored on pi1, pi2 “file2” stored on pi3, pi4
• the mount points pimaster:/mnt/usb0/data etc. are the physical location on disk for the volumes to be located
• In this case, the volume is across all 4 gluster nodes, so the data will be on at least 2 of them, and the volume size is 2x the physical usb stick, as it is 2×2
• Start the volume (on the master)
• gluster volume start data
• Verify the volumes are happy
• gluster volume info

The volume could be any arbitrary directory on the file system, in this case it’s the mounted usb sticks.
It could have just as easily been a common directory on all the Pis or a different location on each Pi.

Mounting Gluster Endpoint for use

To use the gluster volume, we need to mount it to the local system on each of our PI’s. On each Pi in your cluster:

• First we create a directory for the mount point mkdir /gfdata
• For our purposes, we want to make sure we can access no matter who we are: chmod 777 /gfdata
• Then we add it to /etc/fstab: echo "pimaster:/data /gfdata glusterfs defaults,_netdev 0 0" | tee -a /etc/fstab. We point to the glusterfs master which in the example is pimaster.
• Finally we need to reboot

---

Now we can use the /gfdata directory to create volumes in our Docker Swarm cluster. For use in the Kubernetes cluster, we will use the /data glusterfs volume directly. We could also use host mount in the K8s cluster, but that has some dangers with it since not all peers in the glusterfs cluster will have the file as we are replicating 2 times, and joining 2 systems in our example above. We will be adding volumes and moving into Persistent Volumes etc.

Next we will start installing examples of the basic software stack. Check out the plan here.