Varnish Cache is a web application accelerator installed in front of any HTTP server and configured to store a copy of the requested resource in the server cache. Magento 2 has built-in caching mechanisms, and its full-page cache system has native support for Varnish.
This article is a step-by-step guide to configuring Varnish for the Magento 2 preinstalled image.
Installing Varnish
The Ubuntu repository includes the Varnish package; you do not need to add the official repository or a third-party repository to install it. You can simply run:
# apt install varnish -yYou will currently get Varnish version 6.2.1, which is compatible with Magento on the preinstalled image. You can check the version installed from the repository by running the following:
# varnishd -VThe following output should appear:
varnishd (varnish-6.2.1 revision 9f8588e4ab785244e06c3446fe09bf9db5dd8753)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2019 Varnish Software ASConfiguring Varnish and Magento
Once installed, the default configuration needs to be modified. Edit the systemd Varnish service file:
# nano /lib/systemd/system/varnish.serviceYou only need to change the default port (6081) to the HTTP port (80) and increase the cache size limit. It should look something like this:
[Unit]
Description=Varnish Cache, a high-performance HTTP accelerator
After=network-online.target nss-lookup.target
[Service]
Type=forking
KillMode=process
# Maximum number of open files (for ulimit -n)
LimitNOFILE=131072
# Locked shared memory - should suffice to lock the shared memory log
# (varnishd -l argument)
# Default log size is 80MB vsl + 1M vsm + header -> 82MB
# unit is bytes
LimitMEMLOCK=85983232
# Enable this to avoid "fork failed" on reload.
TasksMax=infinity
# Maximum size of the corefile.
LimitCORE=infinity
ExecStart=/usr/sbin/varnishd \
-a :80 \
-a localhost:8443,PROXY \
-p feature=+http2 \
-f /etc/varnish/default.vcl \
-s malloc,2g
ExecReload=/usr/sbin/varnishreload
[Install]
WantedBy=multi-user.targetOnce you have added the configuration, lock the file so that it cannot be modified by a Varnish package update:
# chattr +i /lib/systemd/system/varnish.serviceOnce modified and locked, you can configure it in Magento under Stores, Configuration, Advanced, System and expand the Full Page Cache:
Clear the checkboxes for using system values to select Varnish Cache, download the VCL for the corresponding version, and save the changes.
Configuring NGINX and deploying Varnish
The preinstalled image uses NGINX as its web server, but you can deploy it on any web server. Therefore, before deploying Varnish, change the default NGINX HTTP listening port to 8080 TCP by editing the site file:
# nano /etc/nginx/sites-enabled/defaultReplace the listen value:
listen 8080;Also enter your site's domain or subdomain in server_name:
server_name ejemplo.com;And add health_check to the php location line:
location ~ (index|get|static|report|404|503|health_check)\.php$ {Once you have saved the changes, restart the service to apply them:
# systemctl restart nginx.serviceCheck that the port has been changed:
# ss -tulpn | grep nginxThe following should appear:
tcp LISTEN 0 511 0.0.0.0:8080 0.0.0.0:* users:(("nginx",pid=50572,fd=6),("nginx",pid=50571,fd=6))With the VCL downloaded, replace the current one with it and add the configured NGINX port:
# rm /etc/varnish/default.vcl && nano /etc/varnish/default.vclYou can directly copy the following preconfigured example:
# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;
import std;
# The minimal Varnish version is 6.0
# for SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
backend default {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 600s;
.probe = {
.url = "health_check.php";
.timeout = 2s;
.interval = 5s;
.window = 10;
.threshold = 5;
}
}
acl purge {
}
sub vcl_recv {
if (req.restarts > 0) {
set req.hash_always_miss = true;
}
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
# To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
# has been added to the response in your backend server config. This is used, for example, by the
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}
if (req.http.X-Magento-Tags-Pattern) {
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}
return (synth(200, "Purged"));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Bypass shopping cart and checkout
if (req.url ~ "/checkout") {
return (pass);
}
# Bypass health check requests
if (req.url ~ "/pub/health_check.php") {
return (pass);
}
# Set initial grace period usage status
set req.http.grace = "none";
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");
# collect all cookies
std.collect(req.http.Cookie);
# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
# No point in compressing these
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unknown algorithm
unset req.http.Accept-Encoding;
}
}
# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
set req.url = regsub(req.url, "[?|&]+$", "");
}
# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
#unset req.http.Https;
#unset req.http.X-Forwarded-Proto;
#unset req.http.Cookie;
}
# Authenticated GraphQL requests should not be cached by default
if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
return (pass);
}
return (hash);
}
sub vcl_hash {
if (req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}
# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
if (req.url ~ "/graphql") {
call process_graphql_headers;
}
}
sub process_graphql_headers {
if (req.http.Store) {
hash_data(req.http.Store);
}
if (req.http.Content-Currency) {
hash_data(req.http.Content-Currency);
}
}
sub vcl_backend_response {
set beresp.grace = 3d;
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}
if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
if (beresp.http.X-Magento-Debug) {
set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}
# cache only successfully responses and 404s
if (beresp.status != 200 && beresp.status != 404) {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
set beresp.uncacheable = true;
set beresp.ttl = 86400s;
return (deliver);
}
# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
unset beresp.http.set-cookie;
}
# If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store") ||
beresp.http.Vary == "*") {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
if (resp.http.x-varnish ~ " ") {
set resp.http.X-Magento-Cache-Debug = "HIT";
set resp.http.Grace = req.http.grace;
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
} else {
unset resp.http.Age;
}
# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "-1";
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# Hit within TTL period
return (deliver);
}
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 300s > 0s) {
# Hit after TTL expiration, but within grace period
set req.http.grace = "normal (healthy server)";
return (deliver);
} else {
# Hit after TTL and grace expiration
return (restart);
}
} else {
# server is not healthy, retrieve from cache
set req.http.grace = "unlimited (unhealthy server)";
return (deliver);
}
}Then reload the daemons to apply the service change:
# systemctl daemon-reloadAnd then restart the service:
# systemctl restart varnishCheck that Varnish is listening on the HTTP port (80 TCP):
# ss -tulpn | grep varnishIt should display the following output:
tcp LISTEN 0 1024 0.0.0.0:80 0.0.0.0:* users:(("cache-main",pid=56913,fd=3),("varnishd",pid=56889,fd=3))
tcp LISTEN 0 10 127.0.0.1:6082 0.0.0.0:* users:(("varnishd",pid=56889,fd=7))
tcp LISTEN 0 1024 [::]:80 [::]:* users:(("cache-main",pid=56913,fd=4),("varnishd",pid=56889,fd=4))
HTTPS configuration for NGINX
If you have previously configured HTTPS for NGINX using Configure SSL on the Magento image, skip obtaining the certificate and only configure the reverse proxy (replacing the 443 configuration with 8080).
To obtain the certificate, run the following (change the example domain):
# apt install python3-certbot-nginx -y && certbot certonly --webroot -w /var/www/html/magento2/ -d ejemplo.comOnce you have obtained and configured the certificate, configure the reverse proxy for Varnish in the NGINX HTTPS configuration:
server {
server_name ejemplo.com;
listen 443 ssl http2;
ssl_certificate /etc/letsencrypt/live/ejemplo.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/ejemplo.com/privkey.pem;
ssl_session_timeout 1d;
ssl_session_tickets off;
# HSTS (ngx_http_headers_module is required) (63072000 seconds)
add_header Strict-Transport-Security "max-age=63072000" always;
# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
location / {
proxy_pass http://127.0.0.1;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
fastcgi_buffer_size 32k;
fastcgi_buffers 4 32k;
}
}
Then you can purge the Magento cache by deleting it directly:
# rm -R /var/www/html/magento2/var/cache/*Finally, restart the services for the new configuration:
# systemctl restart varnish && systemctl restart nginx.serviceCongratulations, Varnish is configured! You can now start preparing your Magento store.
Web performance comparison
The following benchmark uses the Magento 2 preinstalled image for both servers, with the same hardware resources and configuration. The only difference is that one server will be configured with Varnish and the other without it.
The first test will consist of running the httpstat tool to check the TTFB.
The results for the server without Varnish are as follows:
DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer
[ 2ms | 2ms | 11ms | 59ms | 3ms ]
| | | | |
namelookup:2ms | | | |
connect:4ms | | |
pretransfer:15ms | |
starttransfer:74ms |
total:77ms
The server processing time is 59 ms, a reasonable time.
Results for the Magento server with Varnish configured:
DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer
[ 4ms | 2ms | 12ms | 10ms | 2ms ]
| | | | |
namelookup:4ms | | | |
connect:6ms | | |
pretransfer:18ms | |
starttransfer:28ms |
total:30ms
The difference is very noticeable: just 10 ms of processing, approximately six times faster.
The second test will execute more than one request; you can use the following tool for this: h2load.
Results from running ten requests against the server without Varnish:
finished in 477.89ms, 20.93 req/s, 184.71KB/s
requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 88.27KB (90390) total, 26.72KB (27360) headers (space savings 0.00%), 60.58KB (62030) data
min max mean sd +/- sd
time for request: 156.07ms 463.20ms 305.79ms 120.96ms 60.00%
time for connect: 4.26ms 13.67ms 8.31ms 3.21ms 60.00%
time to 1st byte: 160.34ms 476.87ms 314.09ms 124.00ms 60.00%
req/s : 2.10 6.22 3.77 1.73 70.00%
A total of ten requests completed in 477.89 ms, with a maximum request time of 463.20 ms and an average of 20,93 requests per second.
The time has increased considerably; like any PHP application with an SQL database, Magento struggles considerably with many requests. With Varnish, however, it should be different:
finished in 26.26ms, 380.88 req/s, 3.25MB/s
requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 87.29KB (89380) total, 18.15KB (18590) headers (space savings 26.49%), 68.39KB (70030) data
min max mean sd +/- sd
time for request: 6.90ms 11.62ms 9.08ms 1.97ms 50.00%
time for connect: 12.00ms 15.23ms 13.54ms 876us 80.00%
time to 1st byte: 19.20ms 25.57ms 22.62ms 2.60ms 60.00%
req/s : 38.81 51.64 44.43 5.21 60.00%
The ten requests completed in just 26,26 ms, with an average of 380,88 requests per second and a maximum time of 11,62 ms. The difference speaks for itself, but you can push harder with wrk to check the results.
Results for the server without Varnish, with ten connections on a single thread for five seconds using wrk:
Running 5s test @ https://magento.clouding.host/
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 481.68ms 74.20ms 654.80ms 84.69%
Req/Sec 20.11 10.31 40.00 91.11%
98 requests in 5.01s, 865.06KB read
Requests/sec: 19.57
Transfer/sec: 172.79KB
Only 98 requests in five seconds, with slightly higher latency than before. Changing the default configuration should produce better results.
However, with the server running Varnish:
Running 5s test @ https://magentovarnish.clouding.host/
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.17ms 4.54ms 96.11ms 78.07%
Req/Sec 1.69k 132.41 1.82k 90.00%
8408 requests in 5.00s, 77.60MB read
Requests/sec: 1680.60
Transfer/sec: 15.51MB
A total of 8408 requests in five seconds! With a maximum latency of 96,11 ms and an average below 10 ms.
Remember, if you have questions about this or any other issue related to your servers on Clouding, feel free to write to soporte@clouding.io We're here for whatever you need. Get in touch!