How to enable TLS/SSL encryption with Glusterfs storage cluster on Linux

Posted on in Categories , , , , , , last updated July 21, 2017

I setup GlusterFS cluster on Ubuntu Linux. How do I enable network encryption using TLS/SSL for my Glusterfs storage cluster on Linux for security reasons when using over insecure network such as WAN (wide area network) or the Internet? How do I setup I/O encryption and management encryption with Glusterfs?

GlusterFS allows you to setup TLS (Transport Layer Security) using the OpenSSL. GlusterFS has two option for TLS/SSL. First, it can do encryption of the I/O communication between servers and clients. Second encryption of the glusterd management server i.e. whenever you issue command line status or create new volumes and so on.

Fig.01: Using SSL with Glusterfs
Fig.01: Using SSL with Glusterfs

Files needed for setting up network encryption in GlusterFS

You need to create the following files for TLS/SSL encryption for each node:

  1. /etc/ssl/glusterfs.pem – Your certificate. This file must be created on each node, and you must not share with any other node or any other server in cluster.
  2. /etc/ssl/glusterfs.key – Your private key. This file must be created on each node, and you must not share with any other node or any other server in cluster.
  3. /etc/ssl/glusterfs.ca – Concatenation of all nodes certificate. This file is not unique and must be the identical on all server nodes and clients.
  4. /var/lib/glusterd/secure-access – Enabling TLS encryption on the management path for each node. You need to create this file on each node with the touch command.

Let us say you have a four-node GlusterFS cluster as follows:

IP addressNode hostname used for TLS identities for authorization
192.168.2.45gfs01
192.168.2.46gfs01
192.168.2.47gfs01
192.168.2.48gfs01

The common name (CN), must match the hostname of your node and it must be set using either DNS or /etc/hosts file on each node:
$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ubuntu-box-1
192.168.2.45 gfs01
192.168.2.46 gfs02
192.168.2.47 gfs03
192.168.2.48 gfs04

Using SSL with Glusterfs

Type the following command on each node (gfs01, gfs02, gfs03, and gfs04).

How to generate a private key for each node and client

$ cd /etc/ssl/
$ sudo openssl genrsa -out glusterfs.key 2048

Sample outputs:

Generating RSA private key, 2048 bit long modulus
.......................................................................+++
................................+++
e is 65537 (0x10001)

How to generate a a signed certificate for each node

From gfs01 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs01" -out glusterfs.pem

From gfs02 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs02" -out glusterfs.pem

From gfs03 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs03" -out glusterfs.pem

From gfs04 node (note the common name (CN) must match your hostname as set using DNS or /etc/hosts):
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=gfs04" -out glusterfs.pem

For client01 and so on:
$ cd /etc/ssl/
$ sudo openssl req -new -x509 -key glusterfs.key -subj "/CN=client01" -out glusterfs.pem

How create the certificates of the Certificate Authorities (CA) file

From gfs01 (type command on gfs01 node) copy all the .pem files from all other nodes including clients as follows:
$ mkdir /tmp/ca/
$ cd /tmp/ca/
$ scp [email protected]:/etc/ssl/glusterfs.pem gfs02.pem
$ scp [email protected]:/etc/ssl/glusterfs.pem gfs03.pem
$ scp [email protected]:/etc/ssl/glusterfs.pem gfs04.pem
## copy file from client01 too ##
$ scp [email protected]:/etc/ssl/glusterfs.pem client01.pem

Type the following command to concatenate the collected files into a single file:
### this file is for all server nodes only ###
$ cat /etc/ssl/glusterfs.pem gfs02.pem gfs03.pem gfs04.pem client01.pem > glusterfs.ca
### this file is for all clients only ###
$ cat /etc/ssl/glusterfs.pem gfs02.pem gfs03.pem gfs04.pem > glusterfs-client.ca

Now put glusterfs.ca file on all the server nodes:
$ sudo cp glusterfs.ca /etc/ssl/
$ scp glusterfs.ca [email protected]:/etc/ssl/
$ scp glusterfs.ca [email protected]:/etc/ssl/
$ scp glusterfs.ca [email protected]:/etc/ssl/
### now put file on clients only ###
$ scp glusterfs-client.ca [email protected]:/etc/ssl/glusterfs.ca
$ rm -rf /tmp/ca/

How to enable management encryption on server nodes

Type the following command on each node (gfs01,gfs02,gfs03,gfs04):
$ sudo touch /var/lib/glusterd/secure-access
Restart the glusterd on all server:
$ systemctl restart glusterfs-server.service

How to enable management encryption on client nodes

Assuming that your volume name is gvol0. Type the following command on all the clients:
$ sudo touch /var/lib/glusterd/secure-access
Unmount the volume on all the clients:
$ sudo umount /var/www
Mount the volume on all the clients:
$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/

How do I enable TLS/SSL I/O encryption for a volume named gvol0?

First stop the volume gvol0 to enable SSL on, run (type all commands on gfs01 node):
$ sudo gluster volume stop gvol0
## You need to nmount the volume on all the clients/servers (if any) ##
## restart glusterfs-server.service on nodes such as gfs01,gfs02,gfs03,gfs04,client01 ##
$ sudo sudo umount /var/www
$ sudo systemctl restart glusterfs-server.service

Next, type the following command that includes names of all the servers and client you want to give access to the volume. The syntax is:
$ sudo gluster volume set {VOLNAME} auth.ssl-allow '{server1,server2,client1,client2,....}'
For example:
$ sudo gluster volume set gvol0 auth.ssl-allow 'gfs01,gfs02,gfs03,gfs04,client01'
In this example, all TLS authenticated clients mount and access your volume:
$ sudo gluster volume set gvol0 auth.ssl-allow '*'
Turn on the client side ssl for the volume:
$ gluster volume set gvol0 client.ssl on
Turn on the server side ssl for the volume:
$ gluster volume set gvol0 server.ssl on
Start the volume, run:
$ sudo gluster vol start gvol0
You need mount the volume on all the clients/servers (if any):
$ sudo mount -t glusterfs gfs01:/gvol0 /mnt/www/
Verify that SSL is enabled on the I/O path for the volume named gvol0:
$ sudo grep -i ssl /var/log/glusterfs/bricks/nodirectwritedata-brick1-gvol0.log


150:     option auth.login./nodirectwritedata/brick1/gvol0.ssl-allow gfs01,gfs02
152:     option transport.socket.ssl-enabled on
[2017-03-07 08:24:38.893211] I [socket.c:343:ssl_setup_connection] 0-tcp.gvol0-server: peer CN = gfs01
[2017-03-07 08:24:38.893248] I [socket.c:346:ssl_setup_connection] 0-tcp.gvol0-server: SSL verification succeeded (client: 192.168.2.45:49149)
[2017-03-07 08:24:39.146162] I [socket.c:343:ssl_setup_connection] 0-tcp.gvol0-server: peer CN = gfs02
[2017-03-07 08:24:39.146197] I [socket.c:346:ssl_setup_connection] 0-tcp.gvol0-server: SSL verification succeeded (client: 192.168.2.46:49148)

References

This entry is 3 of 4 in the GlusterFS Tutorial series. Keep reading the rest of the series:
  1. How to install GlusterFS on a Ubuntu Linux
  2. How to mount Glusterfs volumes inside LXC/LXD
  3. How to enable TLS/SSL encryption with Glusterfs storage
  4. How to add a new brick to an existing replicated GlusterFS volume on Linux

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

1 comment

Comments are closed.