For a little while now I have been running multiple web servers which I would like to access from the internet. Up until now I would just add a NAT translation on a random port to be able to connect to the correct web server. This works as designed but isn't very sleek. I therefore decided to setup a load balancer / reverse proxy to be able to access all web servers on a single IP and port (80/443).

The load balancing soution which I would use had to offer at least to following:

  • Easy setup and configuration
  • TLS (Come on people, SSL has long been dead ) offloading
  • Load balance decision based on hostname
    • one.example.com and two.example.com should point to different back ends

I played around with a couple of solutions such as:

  • HAProxy
    • Easy setup and use but the hostname differentiation wouldn't stick
  • F5 BIG-IP
    • Enterprise level feature set; too many bells and whistles for now
  • Zen
    • Not easy to get into
  • Apache Mod Proxy
    • My Apache knowledge is lacking

In the end I decided on using a solution which I haven't seen around much: Pound

What Pound is:

  1. a reverse-proxy: it passes requests from client browsers to one or more back end servers.
  2. a load balancer: it will distribute the requests from the client browsers among several back end servers, while keeping session information.
  3. an SSL wrapper: Pound will decrypt HTTPS requests from client browsers and pass them as plain HTTP to the back end servers. (Also support HTTPS connections to the back end servers, red.)
  4. an HTTP/HTTPS sanitizer: Pound will verify requests for correctness and accept only well-formed ones.
  5. a fail over-server: should a back end server fail, Pound will take note of the fact and stop passing requests to it until it recovers.
  6. a request redirector: requests may be distributed among servers according to the requested URL.

What Pound is not:

  1. Pound is not a Web server: by itself, Pound serves no content - it contacts the back end server(s) for that purpose.
  2. Pound is not a Web accelerator: no caching is done - every request is passed "as is" to a back end server.

Architecture

During this tutorial we will be using three different hosts. The web servers are serving different web applications. We will not be setting up load balancing right now, as I do not require this for my home environment.(I do not have redundant web servers) The Pound server will also offload TLS, but connections to the back ends will also use TLS (rather than be being clear text) with a self singed certificate. All hosts will be running Ubuntu 14.04 LTS Server Edition.

  1. Pound
    • The Pound Load Balancer/Reverse Proxy
  2. one.example.com
    • Back end web server/application one
  3. two.example.com
    • Back end web server/application two

Architecture

Installation

Repository and Pound version

The version available in the Ubuntu repository is v2.6. As we will require a couple of options (Such as DisableSSLv3 and SSLHonorCipherOrder) which are not available in that version, we will be building the latest (currently Release 2.7 R81) version from source.

Build and Install Pound

  1. Make sure your OS and Packages/Software are up-to-date.
    sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get dist-upgrade -y && sudo apt-get autoremove -y

  2. Create a working directory for Pound. Next, change directory to the new folder
    sudo mkdir /etc/pound sudo mkdir /etc/pound/certs cd /etc/pound

  3. Install make, GCC, OpenSSL and libssl-dev (OpenSSL Development tools) packages from the Ubuntu repository using apt-get.
    sudo apt-get install make gcc openssl libssl-dev -y

  4. Download the latest version from the Pound website and untar the archive.
    sudo wget http://www.apsis.ch/pound/Pound-2.7.tgz sudo tar -xvf Pound-2.7.tgz

  5. Change directory into the new folder and run configure to get prepare the system to build Pound. Use the --with-dh=2048 parameter to be able to use Diffie-Hellman 2048bit
    cd /etc/pound/Pound-2.7/ sudo ./configure --with-dh=2048

  6. Compile Pound (This might take some time as it will generate the Diffie Hellman parameter and key)
    make

  7. Install Pound
    sudo make install

    If, like me, you have done step 4,5 & 6 a thousand times but was never 100% what you were doing, take a look at George Brucklehurst's article on configure, make and make install here.

  8. Pound is now installed, let's clean up a bit
    cd .. sudo rm -rf /etc/pound/Pound-2.7*

Configuration

The configuration file for Pound is located at '/usr/local/etc/pound.cfg'. I like to have things in one place so I created a symbolic link to the config file in /etc/pound/
sudo ln -s /usr/local/etc/pound.cfg /etc/pound/pound.cfg

The config file is divided into four pieces: General, Listeners, Services and Back ends. We will start of with the General part.

General

In the general section, like the name suggest we will configure some generic settings. Open the pound.cfg file using your favorite editor and make it look like below.
sudo nano -c /etc/pound/pound.cfg (-c just shows you the line #)

##### General
##################################################
## Run Pound using the user and group below
User    "www-data"  
Group   "www-data"

## Logging: (goes to syslog by default)
##    0   no logging
##    1   normal
##    2   extended
##    3   Apache-style (common log format)
LogLevel    2

## check back end every X secs if it is still up:
Alive        30

## Ignore case when used in URLs
IgnoreCase 1  
##################################################

Listener & Service

In the listener section we will define to which client requests Pound will respond. Next Pound will try to match the received request to a service. In the service section we will define to which request they can answer. Multiple conditions can be defined such as specific URLs or specific headers. The Service section also allows us to define a session mechanism is desired. (Binding a client to a back end server).

We will define both an HTTP Listener and HTTPS listener. The HTTP listener will redirect all traffic to an HTTPS URL (Basically redirecting to the HTTPS Listener). We will define this per hostname.

Note: The Listener and service sections need to be closed using 'End'. I therefore use indents to keep them organized

HTTP Listener
  1. Open the pound.cfg file using your favorite editor and make it look like below (Adjusted to your hostnames).
    sudo nano -c /usr/local/etc/pound.cfg
####### HTTP Listener
##################################################
ListenHTTP  
        ## Listen on all IP addresses
        Address 0.0.0.0
        ## Listen on port 80
        Port 80

        ## Remove the "X-Forwarded-For" header if it is there. 
        ## Prevents multiple comma separated ip address to show up in the logs.
        HeadRemove "X-Forwarded-For"

        ## one.example.com
        Service
                ## Require the header to match the pattern
                HeadRequire "Host: .*one.example.com.*"
                ## Redirect to HTTPS
                Redirect "https://one.example.com"
        End

        ## two.example.com
        Service
                ## Require the header to match the pattern
                HeadRequire "Host: .*two.example.com.*"
                ## Redirect to HTTPS
                Redirect "https://two.example.com"
        End
End  
##################################################

language-bash

HTTPS Listener
  1. Open the pound.cfg file using your favorite editor and make it look like below
    sudo nano -c /usr/local/etc/pound.cfg

  2. Edit the path to the certificates to use. I like to store the certificates in /etc/pound/certs/. The Certificate file needs to be unprotected and present the full CA Chain in the following order:
    < site private key > < public cert > < issuers cert > < root ca cert >

  3. Edit the back end addresses to match yours

  4. Leave the HTTPS and Port 443 options if your back end used HTTPS. Note: Pound does not check the back end certificate. (you could use a self singed)

##### HTTPS Listener
##################################################
ListenHTTPS  
        ## Listen on all IP addresses
        Address 0.0.0.0
        ## Listen on port 443
        Port 443

        ## Remove the "X-Forwarded-For" header if it is there. 
        ## Prevents multiple comma separated ip address to show up in the logs.
        HeadRemove "X-Forwarded-For"

        ## Defines which HTTP verbs are accepted. 
        ## Value = 2 to allow default HTTP + standard WebDAV verbs 
        ## Refer to Pound manual for more
        xHTTP 2

        ## certificate for one.exmaple.com 
        Cert "/etc/pound/certs/one.example.com.pem"
        ## certificate for two.eample.com
        Cert "/etc/pound/certs/two.example.com.pem"

        ## Enforce Cipher order
        SSLHonorCipherOrder 1

        ## Do not allow client SSL renegotiation
        SSLAllowClientRenegotiation 0

        ## Define TLS Ciphers. 
        ## This will give you 90/100 score at SSLLabs. Disallow 128bit AES for a 
        ## perfect score. 
        ## (I need it for an application, and I trust Elliptic Curve)
        Ciphers "+CAMELLIA256:+AES256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!RC4!SSLv2:!SSLv3"

        ## Disable SSL (All versions)
        Disable SSLv2
        Disable SSLv3

        ## one.example.com
        Service
               ## Require the header to match the pattern
               HeadRequire "Host: .*one.example.com.*"

               ## Define which back end to use. 
               ## Multiple address if load balance or failover      
               BackEnd
                      Address 192.168.1.10
                      ## Use HTTPS to connect to the back end (default: HTTP)
                      HTTPS
                      Port 443
             End
         End

        ## two.example.com
        Service
               ## Require the header to match the pattern
               HeadRequire "Host: .*one.example.com.*"

               ## Define which back end to use. 
               ## Multiple address if load balance or failover             
               BackEnd
                      Address 192.168.1.20
                      ## Use HTTPS to connect to the back end (default: HTTP)
                      HTTPS
                      Port 443
                End
        End
End  
##################################################

Running

  1. Create /etc/default/pound and add '1' to it
    sudo bash -c "sudo echo 'startup=1' > /etc/default/pound"

  2. Edit /etc/init.d/pound file and copy the script below.
    sudo nano -c /etc/init.d/pound

    This will make sure that the Pound daemon is started at boot and enables us to interact with Pound using: sudo /etc/init.d/pound {start|stop|restart|force-reload|status}

    /etc/init.d/pound script original here

#! /bin/sh
### BEGIN INIT INFO
# Provides:          pound
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Should-Start:      $named
# Should-Stop:       $named
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: reverse proxy and load balancer
# Description:       reverse proxy, load balancer and
#                    HTTPS front-end for Web servers
### END INIT INFO
#
# pound    - reverse proxy, load-balancer and https front-end for web-servers

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin/pound  
#DAEMON=/usr/sbin/pound
DAEMON=/usr/local/sbin/pound  
DESC="reverse proxy and load balancer"  
NAME=pound

# Exit if the daemon does not exist (anymore)
test -f $DAEMON || exit 0

. /lib/lsb/init-functions

# Check if pound is configured or not
if [ -f "/etc/default/pound" ]  
then  
  . /etc/default/pound
  if [ "$startup" != "1" ]
  then
    log_warning_msg "$NAME will not start unconfigured."
    log_warning_msg "Please configure; afterwards, set startup=1 in /etc/default/pound."
    exit 0
  fi
else  
  log_failure_msg "/etc/default/pound not found"
  exit 1
fi

# The real work of an init script
case "$1" in  
  start)
    log_daemon_msg "Starting $DESC" "$NAME"
    if [ ! -d "/var/run/pound" ]
    then
        mkdir -p /var/run/pound
    fi
    start_daemon $DAEMON $POUND_ARGS
    log_end_msg $?
    ;;
  stop)
    log_daemon_msg "Stopping $DESC" "$NAME"
    killproc $DAEMON
    log_end_msg $?
    ;;
  restart|force-reload)
    log_daemon_msg "Restarting $DESC" "$NAME"
    killproc $DAEMON
    start_daemon $DAEMON $POUND_ARGS
    echo "."
    ;;
  status)
        pidofproc $DAEMON >/dev/null
    status=$?
    if [ $status -eq 0 ]; then
            log_success_msg "$NAME is running"
        else
            log_success_msg "$NAME is not running"
        fi
    exit $status
        ;;
  *)
    echo "Usage: $0 {start|stop|restart|force-reload|status}"
    exit 1
    ;;
esac

# Fall through if work done.
exit 0
  1. Set the correct permissions on the script
    sudo chmod 755 /etc/init.d/pound

  2. Verify the configuration file
    sudo pound -c

  3. If the config test results in 'OK' go ahead and start the Pound daemon.
    sudo /etc/init.d/pound start

  4. To make redirection work in practice change your web application's DNS record to Pound's IP.

Dotting the i's

  1. Limiting permissions and changing the owner of the certificates
    sudo chmod 644 /etc/pound/certs/* sudo chown www-data:www-data /etc/pound/certs/*
    sudo chown www-data:www-data /etc/pound/*

  2. Perfect forward secrecy
    Pound is not able to enforce perfect forward secrecy. As I do want to enforce it, I configured it on my Apache back end. All you have to do is add the line below to your Apache config file. (Usually located in /etc/apache2/sites-enabled/*)
    Header always set Strict-Transport-Security "max-age=15552000"

    Example site configuration:

<VirtualHost *:443>  
        ServerName one.example.com
        DocumentRoot /var/www/one.example.com/

        Header always set Strict-Transport-Security "max-age=15552000"

        __<removed>__

       <Directory /var/www/one.example.com>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All
                Require all granted
        </Directory>
</VirtualHost>  

Final Notes

Pound Missing Features

  1. Pound does not verify the certificate presented by the back end server. (Self signed works without issues; CA not trusted)
  2. Incorporate forward secrecy

To Do

  1. Run Pound in chroot
  2. Research client authentication (Pound -> back end)
  3. Tighten IPTables etc.

Disclaimer

I have only touched a couple of Pound's features, I am not a subject matter expert, but you are free to ask me anything.

Like this post? Please consider leaving a comment; it's what keeps me going