Why we went back to single-node performance

Cloud and microservices split applications into ever-smaller pieces, and somewhere along the way almost nobody worried about C10K or C100K anymore. Not enough performance? Add machines. That reflex got worse with cloud and k8s: there is no hardware to buy, you call an API or click a few times and hundreds or thousands of VMs or Pods spin up, and containers run into the tens of thousands. When engineers cost more than machines, that bill doesn’t stand out.

But bills accumulate. After the pandemic, when everyone’s pockets got tighter, a lot of teams looked at their cloud spend and realized how much had quietly leaked out. Cheerleading for cloud or against it is beside the point; the point is the economics. And once the economics matter again, single-node performance deserves a second look.

In web acceleration specifically, Google and nginx each had their moment. The PageSpeed + nginx combination once powered companies like Yottaa, which Google acquired for $500M, promising HTML/CSS minification, asset trimming, and dynamic acceleration for web apps. Then, somewhere around 2018, the whole category went quiet.

The reason isn’t mysterious. Modern frontend frameworks do all that trimming and optimization at build time, and many lean on SSR (server-side rendering) anyway, often because a pure client-side app is so unfriendly to search engines that the pages never get indexed, and no indexing means no traffic, and no traffic means no business. So the rendering work got pushed back to the server. But server compute is finite and you pay for it, while client compute scales with your users and is free. Taking work that could run on the client and forcing it back onto the server is a losing trade.

That’s why server-side page optimizers like PageSpeed aren’t mainstream today, and we won’t pretend otherwise. The case for them is narrow: you run a traditional, server-rendered monolith with no modern frontend build pipeline, you genuinely need to minify HTML, inline critical CSS, and combine assets, and you can’t or won’t rewrite the frontend. If your site is already a modern SPA or SSR app, skip this post.

People will tell you to just pull someone’s prebuilt Docker image or deb package. We build from source instead, so the version, the modules, and the linking are all ours, and when something breaks we know where to look. What follows is the procedure we actually run in production.

Environment and versions

OS:      CentOS 7.9
GCC:     4.8.5
NGINX:   1.25.0
PAGESPEED (NPS): 1.13.35.2

PageSpeed links in a long list of image, audio/video, and math development libraries, so its dependency list is much longer than a plain nginx build. Here is the full set:

Install

Dependencies

yum install epel-release.noarch -y
yum install wget gcc gcc-c++ zlib-devel libcurl-devel openssl-devel bzip2-devel gmp-devel.x86_64 mpfr-devel.x86_64 libmpc-devel.x86_64 libSM-devel.x86_64 libxml2-devel gcc-gfortran.x86_64 libjpeg-turbo-devel.x86_64 openjpeg-devel.x86_64 libpng12-devel.x86_64 libtiff-devel cmake3.x86_64 libtool-devel yum-utils libffi-devel jemalloc-devel.x86_64 jasper-devel.x86_64 tbb-devel libdc1394-devel gstreamer-devel.x86_64 blosc-devel libaec-devel brotli-devel.x86_64 giflib-devel.x86_64 gtk2-devel libaio-devel libsqlite3x-devel.x86_64 make eigen3-devel libpcre3-devel

Download and extract sources

wget -O nps.zip https://github.com/apache/incubator-pagespeed-ngx/archive/71e24c1c47113acb5924d8cb523d572b376e9dd0.zip && unzip nps
wget http://nginx.org/download/nginx-1.25.0.tar.gz
wget -O psol.tar.gz https://dl.google.com/dl/page-speed/psol/1.13.35.2-x64.tar.gz

PageSpeed ships in two parts: the nginx module (NPS), which is source, and PSOL (PageSpeed Optimization Library), a prebuilt binary that Google publishes and that must match your NPS version. The commit 71e24c1c47113acb5924d8cb523d572b376e9dd0 above is the code for NPS 1.13.35.2.

The step that breaks the build: LD_LIBRARY_PATH

PSOL’s libraries extract into the module’s usr/lib directory. nginx needs to find them when it configures and compiles the PageSpeed module, so you have to point LD_LIBRARY_PATH there explicitly:

LD_LIBRARY_PATH=../incubator-pagespeed-ngx-${PAGESPEED_VERSION}/usr/lib:/usr/lib

Here incubator-pagespeed-ngx-71e24c1c47113acb5924d8cb523d572b376e9dd0 is the PageSpeed source directory extracted in the previous step for this exact version. Miss this and the build fails outright.

Configure, make, install

LD_LIBRARY_PATH=../incubator-pagespeed-ngx-${PAGESPEED_VERSION}/usr/lib:/usr/lib ./configure --sbin-path=/usr/sbin --modules-path=/usr/lib/nginx  --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_v2_module --with-http_realip_module --with-http_sub_module --with-http_gunzip_module --with-http_secure_link_module --with-http_stub_status_module --with-threads --with-stream --with-stream_ssl_module  --without-http_autoindex_module --without-http_browser_module --without-http_userid_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_split_clients_module --without-http_uwsgi_module --without-http_scgi_module --without-http_upstream_ip_hash_module --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --add-module=../ngx_devel_kit --add-module=../ngx_cache_purge --add-module=../nginx-module-vts --add-module=../echo-nginx-module --add-module=../srcache-nginx-module --add-module=../set-misc-nginx-module  --add-module=../headers-more-nginx-module --add-module=../ngx_http_substitutions_filter_module --add-module=../incubator-pagespeed-ngx-71e24c1c47113acb5924d8cb523d572b376e9dd0

The configure line makes a few deliberate trades. The modules we turn on are the ones we actually use: http_ssl and http_v2 for HTTPS and HTTP/2, realip to recover the client’s real IP (there’s usually an LB or CDN in front), gzip_static and gunzip for precompressed assets, threads and file-aio so disk I/O doesn’t block the worker, stream and stream_ssl for L4 proxying, and stub_status plus vts (nginx-module-vts) for metrics.

Everything we turn off is something we don’t use and that only adds attack surface and binary size: autoindex (no directory listings), the mail pop3/imap/smtp modules (this box isn’t a mail proxy), uwsgi and scgi (we only use fastcgi and reverse proxy), plus browser, userid, split_clients, and so on. Whatever can be --without, we drop, for a cleaner binary.

The trailing --add-module flags are third-party modules: ngx_devel_kit and set-misc are prerequisites for many of the others, cache_purge for active cache invalidation, echo and srcache for cache logic, headers-more for response headers, substitutions_filter for response-body rewriting, and finally PageSpeed itself.

Once configure passes:

make
make install

After install, nginx’s config lives under /etc/nginx.

Takeaway

There’s really only one trap here: point PSOL’s LD_LIBRARY_PATH at the right directory, or the PageSpeed module won’t compile. Everything else is getting the dependencies in place and choosing configure flags to match your use.

As for whether to do this at all: if your site is a traditional server-rendered monolith with no modern frontend build, PageSpeed still saves you a round of manual optimization. If it’s already a SPA/SSR app, it isn’t worth it, and your money is better spent leaving that compute on the client. We’ll cover the PageSpeed filter configuration itself in the next post.