How to configure Nginx SSL/TLS passthrough with TCP load balancing

Posted on in Categories , , , , , last updated June 7, 2017

How do I configure SSL/TLS pass through on Nginx load balancer running on Linux or Unix-like system? How do I load balance TCP traffic and setup SSL Passthrough to pass SSL traffic received at the load balancer onto the backend web servers?

Usually, SSL termination takes place at the load balancer and unencrypted traffic sent to the backend web servers. All HTTPS/SSL/TLS and HTTP requests are terminated on the Nginx server itself. Nginx server uses the HTTP protocol to speak with the backend server. You must take great care to make sure no one snoops traffic between your private network. In shared environment, the private network might not be safe, or you may not be able to get VLAN and VPN for the private network. In that case, you can configure Nginx to pass all encrypted traffic directly to the backend end web server. The traffic remains encrypted between all backend web servers:
TCP load balancing with Nginx and SSL Pass-thru Configuration on Linux or Unix
Please note that you must install the SSL/TLS on all backend server such as 192.168.1.100 and 192.168.1.101.

Say hello to stream module

You need to use the ngx_stream_core_module module for TCP load balancing and is available since version 1.9.0. This module is not built by default on a Debian/Ubuntu or any other distro, it should be enabled with the --with-stream configuration parameter. However, you can install the official Nginx package and use this module.

Step 1: Install Nginx LB

Run commands as per your Linux distro:

Ubuntu Linux 16.04 LTS server

First grab PGP key using wget command:
$ cd /tmp/
$ wget https://nginx.org/keys/nginx_signing.key

Install the same using apt-get command:
$ sudo apt-key add nginx_signing.key
Create a config file:
$ sudo vi /etc/apt/sources.list.d/nginx.list
Append the following code:
deb http://nginx.org/packages/ubuntu/ xenial nginx
deb-src http://nginx.org/packages/ubuntu/ xenial nginx

Save and close the file. Update repo and install nginx:
$ sudo apt-get update
$ sudo apt-get install nginx

Debian Linux 9.x server

First grab PGP key using wget command:
$ cd /tmp/
$ wget http://nginx.org/keys/nginx_signing.key

Install the same using apt-get command:
$ sudo apt-key add nginx_signing.key
Create a config file:
$ sudo vi /etc/apt/sources.list.d/nginx.list
Append the following code:
## [ NOTE: Debian 8.x user replace stretch with jessie ] ##
deb http://nginx.org/packages/debian/ stretch nginx
deb-src http://nginx.org/packages/debian/ stretch nginx

Save and close the file. Update repo and install nginx:
$ sudo apt-get update
$ sudo apt-get install nginx

RHEL/CentOS Linux 6.x/7.x server

You must set up the yum repository for RHEL/CentOS, create the file named /etc/yum.repos.d/nginx.repo:
$ sudo vi /etc/yum.repos.d/nginx.repo
Add the the following config:

## Replace "OS" with rhel or centos depending on the distribution used
## Replace "OSRELEASE" with 6 or 7, for 6.x or 7.x versions, respectively
## An example for CentOS 7.x is as follows:
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1

Save and close the file. Run the following yum command:
$ sudo yum update
$ sudo yum install nginx

Sample outputs:

Fig.01: Installing Nginx from the official repo
Fig.01: Installing Nginx from the official repo

Step 2: Configure Nginx LB

Edit the /etc/nginx/nginx.conf file, run:
$ sudo vi /etc/nginx/nginx.conf
Append the following line:

include /etc/nginx/passthrough.conf;

Here is how it looks:

user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
include /etc/nginx/passthrough.conf;

Save and close the file. Create a text file:
$ sudo vi /etc/nginx/passthrough.conf
Update it as follows:

## tcp LB  and SSL passthrough for backend ##
stream {
    upstream cybercitibizapache {
        server 192.168.1.100:443 max_fails=3 fail_timeout=10s;
        server 192.168.1.101:443 max_fails=3 fail_timeout=10s;
    }

log_format basic '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time "$upstream_addr" '
                 '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log /var/log/nginx/www.cyberciti.biz_access.log basic;
    error_log  /var/log/nginx/wwww.cyberciti.biz_error.log;

    server {
        listen 443;
        proxy_pass cybercitibizapache;
        proxy_next_upstream on;
    }
}

Save and close the file. Test it:
$ nginx -t
Restart or reload the nginx server, run:
$ sudo systemctl reload nginx
OR
$ sudo /etc/init.d/nginx reload

Open port 443 and 80 (must type on nginx server running on public IP)

Use the ufw command to open port 443 on Debian/Ubuntu Linux:
$ sudo ufw allow proto tcp from any to 202.54.1.5 port 443
$ sudo ufw allow proto tcp from any to 202.54.1.5 port 80

You can use the following on CentOS7/RHEL7 to open port 80/443:
# firewall-cmd --get-default-zone
# firewall-cmd --get-active-zones
# firewall-cmd --permanent --add-service=http
# firewall-cmd --permanent --add-service=https
# firewall-cmd --reload

Step 3: Backend web server config at 192.168.1.{100,101}

You must use the real/commercial certificate for your production site. I am going to use Nginx with free Let’s Encrypt SSL certificate:
$ sudo apt-get install git bc wget curl
$ cd /tmp/
$ git clone https://github.com/Neilpang/acme.sh.git
$ cd acme.sh/
$ sudo -i
# ./acme.sh --install
$ sudo source ~/.bashrc
# D=/var/www/html
# DOM='www.cyberciti.biz'
# mkdir -vp ${D}/.well-known/acme-challenge/
###---[ NOTE: Adjust permission as per your setup ]---###
# chown -R www-data:www-data ${D}/.well-known/acme-challenge/
# chmod -R 0555 ${D}/.well-known/acme-challenge/
# mkdir -p /etc/nginx/ssl/${DOM}/
# cd /etc/nginx/ssl/${DOM}/
# openssl dhparam -out dhparams.pem -dsaparam 4096
# acme.sh --issue -w $D -d $DOM -k 4096
# vi /etc/nginx/sites-available/default

Edit config for ssl:

## START: SSL/HTTPS www.cyberciti.biz ###
server {
    #------- Start SSL config with http2 support ----#
    listen 443 http2;
    server_name www.cyberciti.biz;
    ssl on;
    ssl_certificate /etc/nginx/ssl/www.cyberciti.biz/www.cyberciti.biz.cer;
    ssl_certificate_key /etc/nginx/ssl/www.cyberciti.biz/www.cyberciti.biz.key;
    ssl_session_timeout 30m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
    ssl_session_cache shared:SSL:10m;
    ssl_dhparam /etc/nginx/ssl/www.cyberciti.biz/dhparams.pem;
    ssl_prefer_server_ciphers on;
     ## Improves TTFB by using a smaller SSL buffer than the nginx default
    ssl_buffer_size 8k;
 
    ## Enables OCSP stapling
    ssl_stapling on;
    resolver 8.8.8.8;
    ssl_stapling_verify on;
 
    ## Send header to tell the browser to prefer https to http traffic
    add_header Strict-Transport-Security max-age=31536000;
 
    ## SSL logs ##
    access_log /var/log/nginx/ssl_access.log;
    error_log /var/log/nginx/ssl_error.log;
    #-------- END SSL config -------##
     # Add rest of your config below like document path and more ##
}
## END SSL www.cyberciti.biz ######

Save and close the file. Create a shell script called /root/hook.sh to copy your /etc/nginx/ config from 192.168.1.100 to second server (192.168.1.101) and reload it. Here is a sample script:

#!/bin/bash
# Purpose: Replicate ssl and nginx settings/certs across all backends
# Author: Vivek Gite {https://www.cyberciti.biz}
# License: GPL v2.0+
# --
# NOTE: SSH keys must be set between all hosts
# The certificate information must be replicated on 
# every server. Modify script as per your needs.
# ---------------------------------------------------------------------
_rsync="/usr/bin/rsync"
_ssh="/usr/bin/ssh"
_lxc="/usr/bin/lxc"
_rsync_opt='-az -H --delete --numeric-ids '
u="root"
## add all IPs/hostnames of backend here ##
servers="192.168.1.101"
S='/etc/nginx/'
D=/etc/nginx
## Do it ##
for b in ${servers}
do
${_rsync} ${_rsync_opt} "$@" ${S} ${u}@${b}:${D}
${_ssh} ${u}@${b} /etc/init.d/nginx reload
done
## Reload 192.168.1.100 too ##
/etc/init.d/nginx reload

Type the following command:
# acme.sh --installcert -d $DOM --keypath /etc/nginx/ssl/$DOM/$DOM.key --fullchainpath /etc/nginx/ssl/$DOM/$DOM.in.cer --reloadcmd '/root/hook.sh'
The cron job will now automatically create certificates and run /root/hook.sh every time to replicate settings:

## default cron installed by acme.sh
## no need to create it ##
33 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
 
## above will renew certificate every 90 days ##

Conclusion

You should be able to use nginx as a load balancer and pass all SSL traffic to backend servers. Your SSL/TSL certificate is getting terminated on the 192.168.1.100 and 192.168.1.101 backend servers rather than the load balancer hosted at public IP address. Please note that you cannot modify the HTTP headers or grab the client’s IP address i.e. info stored in the X-Forwarded* headers. It is the only drawback of Nginx as SSL Passthrough method. For further info see the following urls:

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

7 comment

  1. re “Debain Linux 9.x server” should use `/debian/` instead of `/ubuntu/`


    ## NOTE: Debian 8.x user replace stretch with jessie ##
    deb http://nginx.org/packages/debian/ stretch nginx
    deb-src http://nginx.org/packages/debian/ stretch nginx

Leave a Comment