≡ Menu

How to enable TLS/SSL encryption with Glusterfs storage cluster on Linux

I setup GlusterFS cluster on Ubuntu Linux. How do I enable network encryption using TLS/SSL for my Glusterfs storage cluster on Linux for security reasons when using over insecure network such as WAN (wide area network) or the Internet? How do I setup I/O encryption and management encryption with Glusterfs?

GlusterFS allows you to setup TLS (Transport Layer Security) using the OpenSSL. GlusterFS has two option for TLS/SSL. First, it can do encryption of the I/O communication between servers and clients. Second encryption of the glusterd management server i.e. whenever you issue command line status or create new volumes and so on.

Fig.01: Using SSL with Glusterfs

Fig.01: Using SSL with Glusterfs

Files needed for setting up network encryption in GlusterFS

You need to create the following files for TLS/SSL encryption for each node:

  1. /etc/ssl/glusterfs.pem – Your certificate. This file must be created on each node, and you must not share with any other node or any other server in cluster.
  2. /etc/ssl/glusterfs.key – Your private key. This file must be created on each node, and you must not share with any other node or any other server in cluster.
  3. /etc/ssl/glusterfs.ca – Concatenation of all nodes certificate. This file is not unique and must be the identical on all server nodes and clients.
  4. /var/lib/glusterd/secure-access – Enabling TLS encryption on the management path for each node. You need to create this file on each node with the touch command.

Let us say you have a four-node GlusterFS cluster as follows:

IP addressNode hostname used for TLS identities for authorization
192.168.2.45gfs01
192.168.2.46gfs01
192.168.2.47gfs01
192.168.2.48gfs01

The common name (CN), must match the hostname of your node and it must be set using either DNS or /etc/hosts file on each node:
$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ubuntu-box-1
192.168.2.45 gfs01
192.168.2.46 gfs02
192.168.2.47 gfs03
192.168.2.48 gfs04

Using SSL with Glusterfs

Type the following command on each node (gfs01, gfs02, gfs03, and gfs04).

How to generate a private key for each node and client

$ cd /etc/ssl/
$ sudo openssl genrsa -out glusterfs.key 2048

Sample outputs:

Generating RSA private key, 2048 bit long modulus
.......................................................................+++
................................+++
e is 65537 (0x10001)

How to generate a a signed certificate for each node

From gfs01 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs01" -out glusterfs.pem

From gfs02 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs02" -out glusterfs.pem

From gfs03 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs03" -out glusterfs.pem

From gfs04 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs04" -out glusterfs.pem

For client01 and so on:
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=client01" -out glusterfs.pem

How create the certificates of the Certificate Authorities (CA) file

From gfs01 (type command on gfs01 node) copy all the .pem files from all other nodes including clients as follows:
$ mkdir /tmp/ca/
$ cd /tmp/ca/
$ scp root@gfs02:/etc/ssl/glusterfs.pem gfs02.pem
$ scp root@gfs03:/etc/ssl/glusterfs.pem gfs03.pem
$ scp root@gfs04:/etc/ssl/glusterfs.pem gfs04.pem
## copy file from client01 too ##
$ scp root@client01:/etc/ssl/glusterfs.pem client01.pem

Type the following command to concatenate the collected files into a single file:
### this file is for all server nodes only ###
$ cat /etc/ssl/glusterfs.pem gfs02.pem gfs03.pem gfs04.pem client01.pem > glusterfs.ca
### this file is for all clients only ###
$ cat /etc/ssl/glusterfs.pem gfs02.pem gfs03.pem gfs04.pem > glusterfs-client.ca

Now put glusterfs.ca file on all the server nodes:
$ sudo cp glusterfs.ca /etc/ssl/
$ scp glusterfs.ca root@gfs02:/etc/ssl/
$ scp glusterfs.ca root@gfs03:/etc/ssl/
$ scp glusterfs.ca root@gfs04:/etc/ssl/
### now put file on clients only ###
$ scp glusterfs-client.ca root@client01:/etc/ssl/glusterfs.ca
$ rm -rf /tmp/ca/

How to enable management encryption on server nodes

Type the following command on each node (gfs01,gfs02,gfs03,gfs04):
$ sudo touch /var/lib/glusterd/secure-access
Restart the glusterd on all server:
$ systemctl restart glusterfs-server.service

How to enable management encryption on client nodes

Assuming that your volume name is gvol0. Type the following command on all the clients:
$ sudo touch /var/lib/glusterd/secure-access
Unmount the volume on all the clients:
$ sudo umount /var/www
Mount the volume on all the clients:
$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/

How do I enable TLS/SSL I/O encryption for a volume named gvol0?

First stop the volume gvol0 to enable SSL on, run (type all commands on gfs01 node):
$ sudo gluster volume stop gvol0
## You need to nmount the volume on all the clients/servers (if any) ##
## restart glusterfs-server.service on nodes such as gfs01,gfs02,gfs03,gfs04,client01 ##
$ sudo sudo umount /var/www
$ sudo systemctl restart glusterfs-server.service

Next, type the following command that includes names of all the servers and client you want to give access to the volume. The syntax is:
$ sudo gluster volume set {VOLNAME} auth.ssl-allow '{server1,server2,client1,client2,....}'
For example:
$ sudo gluster volume set gvol0 auth.ssl-allow 'gfs01,gfs02,gfs03,gfs04,client01'
In this example, all TLS authenticated clients mount and access your volume:
$ sudo gluster volume set gvol0 auth.ssl-allow '*'
Turn on the client side ssl for the volume:
$ gluster volume set gvol0 client.ssl on
Turn on the server side ssl for the volume:
$ gluster volume set gvol0 server.ssl on
Start the volume, run:
$ sudo gluster vol start gvol0
You need mount the volume on all the clients/servers (if any):
$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/
Verify that SSL is enabled on the I/O path for the volume named gvol0:
$ sudo grep -i ssl /var/log/glusterfs/bricks/nodirectwritedata-brick1-gvol0.log


150:     option auth.login./nodirectwritedata/brick1/gvol0.ssl-allow gfs01,gfs02
152:     option transport.socket.ssl-enabled on
[2017-03-07 08:24:38.893211] I [socket.c:343:ssl_setup_connection] 0-tcp.gvol0-server: peer CN = gfs01
[2017-03-07 08:24:38.893248] I [socket.c:346:ssl_setup_connection] 0-tcp.gvol0-server: SSL verification succeeded (client: 192.168.2.45:49149)
[2017-03-07 08:24:39.146162] I [socket.c:343:ssl_setup_connection] 0-tcp.gvol0-server: peer CN = gfs02
[2017-03-07 08:24:39.146197] I [socket.c:346:ssl_setup_connection] 0-tcp.gvol0-server: SSL verification succeeded (client: 192.168.2.46:49148)

References

Share this tutorial on:

Your support makes a big difference:
I have a small favor to ask. More people are reading the nixCraft. Many of you block advertising which is your right, and advertising revenues are not sufficient to cover my operating costs. So you can see why I need to ask for your help. The nixCraft, takes a lot of my time and hard work to produce. If you use nixCraft, who likes it, helps me with donations:
Become a Supporter →    Make a contribution via Paypal/Bitcoin →   

Don't Miss Any Linux and Unix Tips

Get nixCraft in your inbox. It's free:



{ 0 comments… add one }

Leave a Comment

You can use these HTML tags and attributes: <strong> <em> <pre> <code> <a href="" title="">


   Tagged with: , , , , , , ,