Load balancing a web service with HAProxy and an SSL certificate on Ubuntu 18.04 – Clouding.io

Important

This article is obsolete and may contain outdated information. Please consult the latest documentation for up-to-date information.

HAProxy is software that lets us balance the load across multiple servers. It is commonly used to load balance web services, but it can load balance any service that uses the TCP protocol. In this article, we will show you several HAProxy configurations, including how to account for cookies. This may seem trivial, but it can cause our website to fail. Imagine you have 2 web servers, which we will refer to as frontends. When a user first visits your website, they reach frontal-01. When the user registers, a cookie is created. The user continues browsing, and suddenly HAProxy sends the traffic to frontal-2. The cookie does not exist on that server, so the user is logged out :(. Configuring HAProxy correctly prevents this.

Installing HAProxy

The minimum configuration for load balancing requires 3 servers: HAProxy + 2 web frontends. The advantage of using HAProxy is that you can add more frontends at any time.
Another advantage of using HAProxy is that we can restrict access to the frontends through their public IP addresses so that they cannot be accessed directly. We can install HAProxy with the following commands.

# apt update
# apt upgrade
# apt install haproxy
# mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.org

Basic configuration

We can now create our first configuration file by running: vi /etc/haproxy/haproxy.cfg

global
    daemon
    maxconn 256
    user    haproxy
    group   haproxy
    log     127.0.0.1       local0
    log     127.0.0.1       local1  notice
 
defaults
        log     global
        mode    http
        option  tcplog
        option  dontlognull
        maxconn 8000
        timeout connect 5s
        timeout client 300s
        timeout server 300s
        retries 3
        timeout check 10s
        timeout queue 1m
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
 
listen test_kb
    bind *:80
    mode http
    stats enable
    stats auth  cda:cda
    balance roundrobin
    server frontend-01 10.20.10.54:80 #Private IP frontal-01
    server frontend-02 10.20.10.56:80 #Private IP frontal-02

Now that the configuration is complete, we need to restart the HAProxy service. We will do this every time we change the configuration.

# systemctl restart haproxy

Viewing statistics

To view statistics through the web, we will modify the following configuration using vi /etc/haproxy/haproxy.cfg:

listen stats
        bind *:8083
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth haproxy:TestKB

Now we will open our browser and go to http://IP_Publica_HAProxy:8083/stats. When prompted for a username and password, we will enter haproxy and TestKB.

If we want the statistics page to refresh automatically every 10 minutes, we can add the directive

stats refresh 10m

Checking status and managing cookies

We can ensure that a user always accesses the same frontend. In our example, we will use a PHP variable, but we can use any variable.
Another important HAProxy feature is that it can check whether a frontend is down or overloaded and temporarily disconnect it. We will edit the configuration using vi /etc/haproxy/haproxy.cfg

listen test_kb
    bind *:80
    mode http
    stats enable
    stats auth  cda:cda
    balance roundrobin
    option httpchk GET /index.php HTTP/1.0
    http-check expect rstatus (2|3)[0-9][0-9]|503
    cookie PHPSESSID prefix indirect nocache
    server frontend-01 10.20.10.54:80 check maxconn 4000 fall 3 cookie L1
    server frontend-02 10.20.10.56:80 check maxconn 4000 fall 3 cookie L2

Adding a Let's Encrypt SSL certificate for HAProxy

Now that HAProxy is running, this is a good time to make some changes so that our load-balanced website is protected with a certificate. Some points to consider are:

HAProxy and Let's Encrypt cannot use the same ports, 80 and 443. We will modify Let's Encrypt and automatic renewal to use a different port.
HAProxy cannot read the SSL certificate from multiple files as they are generated automatically. We will modify the SSL certificate so that HAProxy can read it.
We will create a filtering rule in HAProxy so that Let's Encrypt renewals work and all other requests are sent to the backend.
We will create a script to enable automatic certificate renewal, adapting it for use with HAProxy.

First, we will install Let's Encrypt with the following commands:

# add-apt-repository -y ppa:certbot/certbot
# apt-get update
# apt-get install -y certbot

Now we will edit the following file:

# vi /etc/haproxy/haproxy.cfg

And set its contents as follows:

global
daemon
maxconn 256
user haproxy
group haproxy
log 127.0.0.1 local0
log 127.0.0.1 local1 notice

defaults
log global
mode http
option tcplog
option dontlognull
maxconn 8000
timeout connect 5s
timeout client 300s
timeout server 300s
retries 3
timeout check 10s
timeout queue 1m
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

listen test_kb
bind *:80
bind *:443 ssl crt /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host.pem

# Force HAProxy to redirect to HTTPS
redirect scheme https code 301 if !{ ssl_fc }


# Send Lets Encrypt requests to the backend letsencrypt block
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt if letsencrypt-acl

#Send all other requests to the servidoresBackend block
default_backend servidoresBackend

#lets encrypt backend
backend letsencrypt
server letsencrypt 127.0.0.1:8888

#default backend
backend servidoresBackend
mode http
stats enable
stats auth cda:cda
balance roundrobin
option httpchk GET /index.php HTTP/1.0
http-check expect rstatus (2|3)[0-9][0-9]|503
cookie PHPSESSID prefix indirect nocache
server frontend-01 10.20.10.8:80 check maxconn 4000 fall 3 cookie L1
server frontend-02 10.20.10.9:80 check maxconn 4000 fall 3 cookie L2

#HaProxy Stats backend
listen stats
bind *:8083
mode http
stats enable
stats uri /stats
stats realm HAProxy\ Statistics
stats auth haproxy:TestKB

In this code, the identifier "edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host" must be replaced with the domain name used for the website, and the server IP addresses must be replaced with those configured on your servers.

The next step is to generate the Let's Encrypt SSL certificate for our domain, which we will then configure in HAProxy. Run the following command:

# certbot certonly --standalone -d edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host --non-interactive --agree-tos --email soporte@clouding.io --http-01-port=8888

As you can see, we use port 8888 to avoid a conflict with HAProxy.

Now we need to create the directory and file where we will store the SSL certificate adapted for HAProxy to read. To do this, we will run:

# mkdir -p /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host
# cat /etc/letsencrypt/live/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/fullchain.pem /etc/letsencrypt/live/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/privkey.pem | tee /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host.pem

The certificate is now adapted and is used in the previous HAProxy configuration on this line:

bind *:443 ssl crt /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host.pem

To renew the Let's Encrypt certificate automatically, we need to create a small script and add it to crontab so that it runs once a month:

# crontab -e

And add the following line:

0 0 1 * * root bash /opt/renovar-certificados.sh

Now we create the script:

# vi /opt/renovar-certificados.sh

and add the following contents:

#!/usr/bin/env bash  
# Renew certificate
certbot renew --force-renewal --tls-sni-01-port=8888
# Adapt the certificate for haproxy
bash -c "cat /etc/letsencrypt/live/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/fullchain.pem /etc/letsencrypt/live/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/privkey.pem | tee /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host.pem"
# Reload haproxy
service haproxy reload

Finally, we should check that Let's Encrypt will be able to renew the certificate by running:

# certbot renew --tls-sni-01-port=8888

Articles in this section