HowTo: Run Network Service Per System / VM Instance To Improve Overall Security Of A Web Stacks (LAMP)

by on June 22, 2012 · 8 comments· LAST UPDATED June 22, 2012

in Apache, Linux

A web stack is nothing but collection of many open source software such as an operating system, Web server, database server, server side programming language. The most commonly known web stacks is LAMP. It is an acronym for a solution stack of free, open source software, referring to the first letters of Linux (operating system), Apache Web server, MySQL database software and PHP (or sometimes Perl or Python). All of our security related tutorials recommends running different network services on separate systems or vm instance. Naturally, this limits the number of other services that can be cracked in the event that an attacker is able to successfully exploit a software flaw in one network service. This is also one of the most requested article via email. In this guide, I will explain how to setup a solution that can serve static content, dynamic content, database, and caching by running on separate servers or vm instance.

LAMP: Traditional vs split setup

You end us setting up everything using a single server or vm instance to serve up all content as follows:

                   One Big Server / VM instance
               +-----------------------------------------+
               | Apache+PHP/Perl @ 75.126.153.206:80     |
               | Mysql @ 127.0.0.1:3306 (or unix socket) |
               | Pgsql @ 127.0.0.1:5432 (or unix socket) |
               | netfilter for filtering traffic         |
               +-----------------------------------------+
                 *** Dedicated LAMP server ***
                 OS: RHEL/CentOS/Debian/Ubuntu/*BSD/Unix
                 RAM: 4-8GiB ECC
                 CPU: Single or dual Intel/Amd
                 Storage: RAID-1/5 server class sata/sas

Say if an Apache web server compromised, an attacker will gain access to your database, cache, and other part of network too. So you need to split services per server as follows:

/////////////////////////
/ Internet/ISP router  /
///////////////////////
          \
            |
             \
              ---------| vm00
             75.126.153.206:80 - eth0
             192.168.1.1       - eth1
         +----------------------------+
         | Reverse Proxy              |
         | Firewall                   |       eth0:192.168.1.10/vm01
         +----------------------------+      +----------------------+
         |                                   | Lighttpd static files|
         +-----------------------------------+ /var/www/static      |
         |                                   +----------------------+
         |
         |                                    eth0:192.168.1.11/vm02
         +-----------------------------------+-----------------------+
         |                                   | Apache+php+perl+python|
         |                                   | /var/www/html         |
         |                                   +-----------------------+
         |
         |                                    eth0:192.168.1.12/vm03
         +-----------------------------------+-----------------------+
         |                                   |Database sql caching   |
         |                                   |Redis/Memcached etc    |
         |                                   +-----------------------+
         |
         |                                   eth0:192.168.1.13/vm04 (or a dedicated server for database w/ RAID-10)
         +-----------------------------------+------------------------+
         |                                   | Mysql/pgsql db server  |
         |                                   | @192.168.1.13:3306/5432|
         |                                   +------------------------+
         |
         |                                   eth0:192.168.1.14/vm05 (or a dedicated server for storing all files via NFSv4 w/ RAID-10)
         +-----------------------------------+------------------------+
         |                                   | NFSv4 running on Linux |
         |                                   | /export/{static,html   |
         |                                   +------------------------+

The spit setup offers many benefits such as:

  1. Security
  2. Scalability and growth
  3. Optimization
  4. Ease of use
  5. Ease of monitoring
  6. Add advanced feature such as failover (VIP at vm00 level), load-balancing, cdn and much more easily.

Role for each vm / server:

The following provides more details. A wordpress based blog or drupal based site or custom made app hosted on the following setup can easily serve millions of page views per month.

VM/ServerDescriptionSoftware (part of stack)
vm00 w/ 2 core and 2GiB RAMThis is the only part you are going to expose to the world. This node / vm must have two network interfaces. One interface, called public is used by clients to connect to the your website. The other interface, called private (VLAN) is used for network traffic only between nodes.This vm or server must have two network interfaces.nginx/haproxy as reverse proxy and netfilter as firewall.
vm01 w/ 1 core and 1GiB RAMUse this node to serve static files such as images/js/css. You need to store files on nfs server called vm05 and mount it at /var/www/static/Lighttpd/tux/nginx and Linux nfs client
vm2 w/ 2-4 core and 4GiB RAMThis node is used to serve dynamic pages. For example, you can install wordpress or your custom application written in PHP5/Python/Perl on this node. You need to store application files on nfs server called vm05 and mount it at /var/www/html/. You need store log locally at /var/log/apache directory.Apache 2 with fastcgi+php5+perl+python etc and Linux nfs client.
vm3 w/ 1 core 1-2GiB RAMThis node is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source.Memcached or Redis etc.
vm4 w/ 2-4 core and 2-4GiB RAM+RAID hybrid storageThis node stores your data in RDBMS such as mysql or postgresql.Postgresql or Mysql server.
vm5 w/ 4 core and 4-6GiB RAM+RAID hybrid storageThis node act as Network File System (NFS) server which is a distributed file system. You store images,js,css in /var/www/static directory and php,perl,python files in /var/www/html directory on this server only. You need to export /var/www/html and /var/www/static directories using the /etc/exports configuration file. The client machine vm01 and vm02 requests access to exported data, typically by issuing a mount command. Apache and lighttpd server daemons on the client machine can then view and interact with mounted filesystems on the server. You also need to create ssh / sftp users on this server so that they can upload files.Linux NFSv4 server
ISP router or uplink port at the hosting data center.The vm00 node connects to the Internet using uplink provided by your ISP. You can start with 100Mbps or 1000Mbps uplink port for eth0. The private port (eth1) is used for network traffic only between all vm nodes and it can be on 100/1000Mbps based VLAN setup.Site specific so no recommendations for uplink.

How does it works?

Let's see how our setup and a reverse proxy works. In this example, I will be placing reverse proxy and http servers inside my firewall. See Figure 1. The web site name is www.example.com which resolves to a static public IPv4 address 202.54.1.1 which is assigned to eth0. 192.168.1.1 is our internal IP and assigned to eth1. This site is your reverse proxy. Rest of the all servers are inside the LAN and can not be accessed directly over the Internet.

Figure 1

Figure 1 ~ click to enlarge

The hardware (or OpenBSD / Linux software) firewall rules only allow 202.54.1.1 on ports 80 and 443. All other ports are blocked. Each vm node also runs iptables and only allows access to required ports. Your reverse proxy defines pool of http server as follows:

## our upstream for www.example.com ##
upstream mybackend {
      server 192.168.1.10:80; #server1
      server 192.168.1.11:80; #server2
      ....
      ..
      ..
      server 192.168.1.100:80; # server100
}

The Apache and lighttpd server will access its files using nfs server configured at vm05. The Apache web server is configured with PHP. Our PHP application is configured to connect to the database server hosted on vm04. Our PHP application uses vm03 to cache sql quires using memcached server.

NOTE: You can also place reverse proxy on a DMZ and http and other servers inside firewall to increase security. This will increase the cost of the project.

Enough talk, show me the actual server configuration

Most of the actions listed in this post are written with the assumption that they will be executed by the root user running the bash on CentOS 6.x / Red Hat Enterprise Linux 6.x. However, you can easily replicate the setup on any other *nix like operating systems.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

1 Jalal Hajigholamali June 23, 2012 at 2:32 am

Hi,
Thanks….

2 Matthew June 23, 2012 at 8:39 am

Hi there,
Interesting article. Have you considered using a container technology rather than a ‘full’ virtualisation technology? For example I have used OpenVZ and Linux Vserver to achive similar configurations. There are advantages, for example shared filesystem without NFS and next to zero virtualisation overhead. You can even use Openvz on Amazon instances, but the iptables foo gets a bit hairy, prob best to wait for Amazon to allow multiple ip addresses per instance really. As a disavantage, I suppose that the ‘full’ virtulisation solution has a theoretically supperiour security stance, but I suggest that it is not significantly more secure.

Mat.

3 nixCraft June 23, 2012 at 9:50 am

@Matthew,

Yes, I’ve used FreeBSD jails and lxc Linux containers. However, I’ve not used OpenVZ or any other container technology as we have policy to go full VM way (using XEN/KVM/Vmware) or just use dedicated boxes.

Currently, I’m evaluating and testing Amazon solution for large enterprise app. I will post my finding on Amazon here only.

Appreciate all comments.

4 Topper June 27, 2012 at 5:39 am

Offtopic: Where disappear search field on the site ??

5 nixCraft June 27, 2012 at 11:51 am

@Topper,

Whoa, thanks for the heads up! It was error on my part. Last Sunday I upgraded everything on server including WP and I forgot to add the search box :(

Appreciate you comment.

6 John June 28, 2012 at 11:00 pm

I understand that this is all VM based and i like the idea. But what is the bare iron under all of it look like ?

7 Topper June 29, 2012 at 5:26 am

The same neat setup ;)

8 Netritious June 29, 2012 at 6:20 am

While I’ll agree that separation of network services is a good idea physically, doing so virtually does not equal more security. This is a common misconception, and provides a false sense of security. Virt admins will often not read host log files due to this false sense of security.

Virtualization software is just as vulnerable as some off-the-shelf LAMP application eg WordPress, Drupal, or Joomla!; as vulnerable as Apache, PHP, and/or SQL databases engines, and as vulnerable as the OS’s network stack interfacing the public web. In other words, all software is vulnerable at some point in time. Take the case of OpenBSD – even it has had one or two remotely executable vulnerabilities and it is touted as the most secure OS in the world.

The scope of virtualization is solely economics. Virtualization is just a tool to reduce the number of bare-metal machines and there are several reasons to do so — energy, time, complexity, cost, etc but It has absolutely nothing to do with security.

In fact, I would argue that it’s probably more secure to run multiple instances of the same LAMP stack — one on your net and one on some other net, and sync the two, but only have one live to the public. Use the secondary to check the primary, and notify when unreachable. (I use email-to-SMS.) Once you have your head wrapped around why the primary is dead, fix as needed on the secondary (sec patches, iptables amendments, package updates, etc), update your DNS A records to point to the secondary, and open up your firewall, screen and tail some logs, take a deep breath and read. Most of which should be scripted already if you care about your resources.

Comments on this FAQ are closed. If you'd like to continue the discussion on this topic, you can do so at our forum.

Tagged as:

Previous post: