GlusterFS is a free and open source network distributed storage file system. Network distributed storage file systems are very popular amoung high trafficked websites, cloud computing servers, streaming media services, CDN (content delivery networks) and more. This tutorial shows you how to install GlusterFS on Ubuntu Linux 16.04 LTS server and configure 2 nodes high availability storage for your web server.
Our sample setup
Fig.01: Our scale-out network-attached storage file system for /var/www/html is made of two nodes
What you need to setup GlusterFS with a replicated high availability storage?
- Minimum two separate storage (can be cloud or bare metal boxes)
- Each server needs a separate partition or disk for GlusterFS. For demo purpose I am using /dev/vdb1 as a brick volume.
- A private network (LAN/VLAN) between servers
- Ubuntu Linux 16.04 LTS on both servers
Okay enough talk, lets get started with GlusterFS installation and configuration on a Ubuntu.
For demo purpose I used two nodes. But you must use 4 or more node in production to avoid split brain issues.
Step 1 – Setup /etc/hosts
First setup /etc/hosts files:
$ sudo vi /etc/hosts
Set correct private IP address as per fig.01 or as per your setup:
192.168.1.11 gfs01 192.168.2.12 gfs02
Close and save the file. Test it:
$ ping -c2 gfs01
$ ping -c2 gfs02
Sample outputs:
PING gfs01 (192.168.1.12) 56(84) bytes of data. 64 bytes from gfs01 (192.168.1.12): icmp_seq=1 ttl=64 time=0.033 ms 64 bytes from gfs01 (192.168.1.12): icmp_seq=2 ttl=64 time=0.036 ms --- gfs01 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.033/0.034/0.036/0.006 ms
Step 2 – Configure iptables to accept all traffic between gfs01 and gfs02
Type the following on gfs01 to allow all traffic from lan node gfs02:
$ sudo ufw allow proto any from 192.168.1.12 to 192.168.1.11
Type the following on gfs02 to allow all traffic from lan node gfs01:
$ sudo ufw allow proto any from 192.168.1.11 to 192.168.1.12
Step 3 – Configure glusterfs stable repository
Type the following command on both gfs01 and gfs02 server to install latest stable glusterfs server:
$ sudo add-apt-repository ppa:gluster/glusterfs-3.11
$ sudo apt-get update
Sample outputs:
Fig.02: Use the following above code to install latest stable package
Step 4 – Install glusterfs stable package
Type the following apt-get command/apt command on both gfs01 and gfs02 servers:
$ sudo apt-get install glusterfs-server
Sample outputs:
Fig.03: Install server and client components
Step 4 – Hold glusterfs package upgrade for safety reasons
Your shared storage brick might corrup if glusterfs upgraded while running it. It is best to filter out of automatic updates for safety. Later you learn how to upgrade glusterfs safety. Type the following command on both gfs01 and gfs02 server:
$ sudo apt-mark hold glusterfs*
Sample outputs:
glusterfs-client set on hold. glusterfs-common set on hold. glusterfs-server set on hold. glusterfs-dbg set on hold.
Step 5 – Setup and format disk in each server
You need to type the following commands on gfs01 server (be careful with device names while partition block devices):
$ sudo fdisk /dev/vdb
$ sudo mkfs.ext4 /dev/vdb1
$ sudo mkdir -p /nodirectwritedata/brick1
$ sudo echo '/dev/vdb1 /nodirectwritedata/brick1 ext4 defaults 1 2' >> /etc/fstab
$ mount -a
$ sudo mkdir /nodirectwritedata/brick1/gvol0
$ df -H
You need to type the following commands on gfs02 server:
$ sudo fdisk /dev/vdb
$ sudo mkfs.ext4 /dev/vdb1
$ sudo mkdir -p /nodirectwritedata/brick2
$ sudo echo '/dev/vdb1 /nodirectwritedata/brick2 ext4 defaults 1 2' >> /etc/fstab
$ mount -a
$ df -H
$ sudo mkdir /nodirectwritedata/brick2/gvol0
Sample outputs:
Fig.04: Prepare the bricks by setting up partition block devices and formatting with ext4 file system
Warning: Do not edit or write files directly to a /nodirectwritedata/brick1/ or /nodirectwritedata/brick2/ brick on each server. A direct write will corrupt your volume.
Step 6 – Verify that glusterfs service started
Type the following commands:
$ sudo systemctl status glusterfs-server.service
Sample outputs:
? glusterfs-server.service - LSB: GlusterFS server Loaded: loaded (/etc/init.d/glusterfs-server; bad; vendor preset: enabled) Active: active (running) since Tue 2017-02-28 00:07:22 IST; 26min ago Docs: man:systemd-sysv-generator(8) Tasks: 7 Memory: 16.3M CPU: 1.412s CGroup: /system.slice/glusterfs-server.service ??3429 /usr/sbin/glusterd -p /var/run/glusterd.pid Feb 28 00:07:20 ubuntu-box2 systemd[1]: Starting LSB: GlusterFS server... Feb 28 00:07:20 ubuntu-box2 glusterfs-server[3420]: * Starting glusterd service glusterd Feb 28 00:07:22 ubuntu-box2 glusterfs-server[3420]: ...done. Feb 28 00:07:22 ubuntu-box2 systemd[1]: Started LSB: GlusterFS server.
If not running start it:
$ sudo systemctl start glusterfs-server.service
Enable glusterfs at boot time:
$ sudo systemctl enable glusterfs-server.service
Sample outputs:
glusterfs-server.service is not a native service, redirecting to systemd-sysv-install Executing /lib/systemd/systemd-sysv-install enable glusterfs-server
Step 7 – Configure the trusted pool
From gfs01 server type:
$ sudo gluster peer probe gfs02
peer probe: success. Host gfs02 port 24007 already in peer list
From gfs02 server type:
$ sudo gluster peer probe gfs01
peer probe: success. Host gfs01 port 24007 already in peer list
Step 8 – Set up a GlusterFS volume
From gfs01 (or gfs02) sever type:
# gluster volume create gvol0 replica 2 gfs01:/nodirectwritedata/brick1/gvol0 gfs02:/nodirectwritedata/brick2/gvol0
Sample outputs:
volume create: gvol0: success: please start the volume to access data
# gluster volume start gvol0
Sample outputs:
volume start: gvol0: success
To see status, enter:
# gluster volume status
Status of volume: gvol0 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs01:/nodirectwritedata/brick1/gvol0 49152 0 Y 2621 Brick gfs02:/nodirectwritedata/brick2/gvol0 49152 0 Y 3674 Self-heal Daemon on localhost N/A N/A Y 3694 Self-heal Daemon on gfs01 N/A N/A Y 2641 Task Status of Volume gvol0 ------------------------------------------------------------------------------ There are no active volume tasks
To see info about your volume, enter:
# gluster volume info
Sample outputs:
Volume Name: gvol0 Type: Replicate Volume ID: 5d871eed-182e-461f-9eb6-06798eeb2c04 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gfs01:/nodirectwritedata/brick1/gvol0 Brick2: gfs02:/nodirectwritedata/brick2/gvol0 Options Reconfigured: transport.address-family: inet performance.readdir-ahead: on nfs.disable: on
Step 9 – Client mounts
Now our cluster is up and running. It is the time our clients can mount shared stoage. You can have 1,2,3,4 or as many client you want as per your network resources. You can configure gfs01 and gfs02 as client itself too.
Client mount on gfs01/gfs02 node itself
Type the following command:
$ sudo mkdir /mnt/www/
$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/
Update the /etc/fstab file:
echo 'gfs01:/gvol0 /mnt/www glusterfs defaults,_netdev 0 0' >> /etc/fstab
Save and close the file. Mount it:
$ sudo mount -a
$ df -H
Sample outputs:
Filesystem Size Used Avail Use% Mounted on
udev 981M 0 981M 0% /dev
tmpfs 138M 2.5M 136M 2% /run
/dev/mapper/ubuntu--box--1--vg-root 37G 2.0G 33G 6% /
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 489M 0 489M 0% /sys/fs/cgroup
/dev/vda1 472M 105M 343M 24% /boot
/dev/vdb1 4.8G 11M 4.6G 1% /nodirectwritedata/brick1
tmpfs 98M 0 98M 0% /run/user/0
gfs02:/gvol0 4.8G 11M 4.6G 1% /mnt/www
Client mount on a different server named www1
If www1 is using a Debian/Ubuntu Linux OS, type:
$ sudo add-apt-repository ppa:gluster/glusterfs-3.11
$ sudo apt-get update
$ sudo apt-get install glusterfs-client
$ sudo mkdir -p /mnt/www/
Edit /etc/hosts file and update it as follows:
192.168.2.11 gfs01 192.168.2.12 gfs02
Save and close the file. Update /etc/fstab file to mount a shared storage:
Update the /etc/fstab file:
echo 'gfs01:/gvol0 /mnt/www glusterfs defaults,_netdev 0 0' >> /etc/fstab
Save and close the file. Mount it:
$ sudo mount -a
$ df -H
Step 10 – Test it
Cd /mnt/www/ and create some files on www1 or gfs02:
# cd /mnt/www
# mkdir test
# cd test
# cp /etc/*.conf .
You should see files on both gfs02, gfs01 and www1.
A note about setting network.ping.timeout
Finally run the following command on both gfs02 and gfs01 to set network ping timeout to 5 seconds from default 42:
$ sudo gluster volume set gvol0 network.ping-timeout "5"
Verify it
$ sudo gluster volume get gvol0 network.ping-timeout
Option Value ------ ----- network.ping-timeout 5
Otherwise glusterfs client will hang out when other node goes down for 42 seconds. You just reduced this to 5 seconds.
A note about Apache DocumentRoot
You need to edit httpd.conf and set DocumentRoot to point to /mnt/www/ directory.
References
- How to install GlusterFS on a Ubuntu Linux
- How to mount Glusterfs volumes inside LXC/LXD
- How to enable TLS/SSL encryption with Glusterfs storage
- How to add a new brick to an existing replicated GlusterFS volume on Linux
🐧 Get the latest tutorials on Linux, Open Source & DevOps via:
- RSS feed or Weekly email newsletter
- Share on Twitter • Facebook • 8 comments... add one ↓
Category | List of Unix and Linux commands |
---|---|
File Management | cat |
Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
Network Utilities | dig • host • ip • nmap |
OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
Package Manager | apk • apt |
Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
Searching | grep • whereis • which |
User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
Great article. Somewhat OT question: What did you use to create your ascii-looking architecture diagram?
Hand drawn in a text editor. No special tool used.
Hi! Great article. What kind of dedicated networking do you recommend between Gluster nodes? 1Gbps cooper? 10Gbps?
I’m planning a similar scenario with 4 Proxmox servers sharing their local storage as a unique GlusterFS pool.
Thanks!
Any recomendation for cluster monitoring tool?
Good article, but maybe here is typo
”
Set correct private IP address as per fig.01 or as per your setup:
192.168.2.11 gfs01
192.168.2.12 gfs02
”
is differ from 192.168.1.11 in fig.01 …
Thanks for the heads up. The faq has been updated.
Good Article. One question. How do you take care of split brain condition in a 2 node replicated setup incase if one node goes down ?
Excelent article, I was up and running with 3 nodes in a couple hours, thank you.
You may want to change all of your service calls from “glusterfs-server” to “glusterd”.
ex:
change
sudo systemctl status glusterfs-server.service
to
sudo systemctl status glusterd.service
the former didn’t work for me with ver 3.11