Load balancing a web service with HAProxy and an SSL certificate on Ubuntu 22.04 LTS

HAProxy is software that distributes the load across multiple servers. It is commonly used to load balance web services, but it can load balance any service that uses the TCP protocol. This article covers several HAProxy configurations, including how to account for cookies. This may seem trivial, but it can cause your website to fail. Imagine you have 2 web servers, referred to here as frontends. When a user first visits your website, they reach frontal-01. When the user registers, a cookie is created. The user continues browsing, and suddenly HAProxy sends the traffic to frontal-2. The cookie does not exist on that server, so the user is logged out. Configuring HAProxy correctly prevents this from happening.

The minimum setup for load balancing consists of 3 servers: HAProxy + 2 web frontends. One advantage of using HAProxy is that you can add more frontends at any time.
Another advantage of using HAProxy is that you can restrict access to the frontends by public IP address, so they cannot be accessed directly.

There are also other services for load balancing a server, such as nginx.

Installing HAProxy

Run the following commands to install HAproxy:

# apt update && apt upgrade
# apt install haproxy
# mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.org

Basic configuration

You can now create the first configuration file. Run:

nano /etc/haproxy/haproxy.cfg

Copy the configuration below, replacing the server IP addresses with the ones you are using:

global
    daemon
    maxconn 256
    user    haproxy
    group   haproxy
    log     127.0.0.1       local0
    log     127.0.0.1       local1  notice
 
defaults
        log     global
        mode    http
        option  tcplog
        option  dontlognull
        maxconn 8000
        timeout connect 5s
        timeout client 300s
        timeout server 300s
        retries 3
        timeout check 10s
        timeout queue 1m
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend Haproxy
bind *:80
option forwardfor
default_backend servidores

backend servidores
balance roundrobin
server server1 10.20.10.8:80 check
server server2 10.20.10.9:80 check

Now that the configuration is ready, restart the HAProxy service. You must do this every time you change the configuration.

# systemctl restart haproxy

Viewing statistics

To view statistics through the web interface, add the following configuration using nano /etc/haproxy/haproxy.cfg: 

listen stats
        bind *:8083
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth haproxy:TestKB

Now open http://IP_Publica_HAProxy:8083/stats in your browser. When prompted for a username and password, enter haproxy and TestKB.

If you want the statistics page to refresh automatically every 10 minutes, add this directive:

stats refresh 10s

Checking status and managing cookies

You can ensure that a user always accesses the same frontend. Our example uses a PHP variable, but you can use any variable.

backend servidores
    balance roundrobin
    cookie PHPSESSID prefix indirect nocache
    server server1 10.20.10.8:80 check maxconn 4000 fall 3 cookie L1
    server server2 10.20.10.9:80 check maxconn 4000 fall 3 cookie L2

Adding a Let's Encrypt SSL certificate for HAProxy

Now that HAProxy is running, it is a good time to make some changes to protect the load-balanced website with a certificate. In this example, we will use Let's Encrypt. Here are some points to keep in mind:

• HAProxy and Let's Encrypt cannot use the same ports, 80 and 443. We will modify Let's Encrypt and automatic renewal to use a different port.
• HAProxy cannot read the SSL certificate from the separate files that are generated automatically. You will modify the SSL certificate so that HAProxy can read it.
• You will need to create a filtering rule in HAProxy so that Let's Encrypt renewals work and all other requests are sent to the backend.
• You will need to create a script to enable automatic certificate renewal, adapting it for use with HAProxy.
• Remember that DNS must be configured. You can find out how in the following article: Configuring DNS zones for our server

First, install Let's Encrypt with the following commands:

# apt install -y certbot

Now edit the following file:

# nano /etc/haproxy/haproxy.cfg

Set its contents as follows:

global
daemon
maxconn 256
user haproxy
group haproxy
log 127.0.0.1 local0
log 127.0.0.1 local1 notice

defaults
log global
mode http
option tcplog
option dontlognull
maxconn 8000
timeout connect 5s
timeout client 300s
timeout server 300s
retries 3
timeout check 10s
timeout queue 1m
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend Haproxy
bind *:80
option forwardfor
default_backend servidores
#bind *:443 ssl crt /etc/ssl/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host.pem

# Force HAProxy to redirect to HTTPS
#redirect scheme https code 301 if !{ ssl_fc }


# Send Lets Encrypt requests to the letsencrypt backend block
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt if letsencrypt-acl

#Backend for lets encrypt
backend letsencrypt
server letsencrypt 127.0.0.1:8888

backend servidores
balance roundrobin
cookie PHPSESSID prefix indirect nocache
server server1 10.20.10.8:80 check maxconn 4000 fall 3 cookie L1
server server2 10.20.10.9:80 check maxconn 4000 fall 3 cookie L2

listen stats
bind *:8083
mode http
stats enable
stats uri /stats
stats realm HAProxy\ Statistics
stats auth haproxy:TestKB

In this code, the identifier "edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host" must be replaced with the domain name used for the website, and the server IP addresses must be replaced with those configured on your server.

Restart HAproxy:

# systemctl restart haproxy

The next step is to generate the Let's Encrypt SSL certificate for the domain, which you will then configure in HAProxy. Run the following command:

# certbot certonly --standalone -d de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host --non-interactive --agree-tos --email soporte@clouding.io --http-01-port=8888

As you can see, you must use port 8888 to avoid a conflict with HAProxy.

Now create the directory and file where you will store the SSL certificate adapted for HAProxy to read. To do this, run:

# mkdir -p /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host
# cat /etc/letsencrypt/live/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/fullchain.pem /etc/letsencrypt/live/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/privkey.pem | tee /etc/ssl/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host.pem

The certificate is now adapted for use in the previous HAProxy configuration on this line:

bind *:443 ssl crt /etc/ssl/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host/edf10be0-f2c0-4cef-b38b-eee8e9577d6f.clouding.host.pem

You can now uncomment the earlier lines:

bind *:443 ssl crt /etc/ssl/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host.pem
redirect scheme https code 301 if !{ ssl_fc }

To renew the Let's Encrypt certificate automatically, create a small script and add it to crontab to run once a month:

# crontab -e

Add the following line:

0 0 1 * * root bash /opt/renovar-certificados.sh

Now create the script:

# nano /opt/renovar-certificados.sh

and add the following content:

#!/usr/bin/env bash  
# Renew certificate
certbot renew --force-renewal --tls-sni-01-port=8888
# Adapt the certificate for haproxy
bash -c "cat /etc/letsencrypt/live/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/fullchain.pem /etc/letsencrypt/live/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/privkey.pem | tee /etc/ssl/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host/de9dcd77-65a7-439a-8fa2-29eda02025a3.clouding.host.pem"
# Reload haproxy
service haproxy reload

Finally, check that Let's Encrypt will be able to renew the certificate by running:

# certbot renew --tls-sni-01-port=8888

Remember, if you have any questions about this or any other issue related to your Clouding servers, feel free to email soporte@clouding.io We're here for whatever you need. Get in touch!