Docker Nginx HTTP/3 and Brotli
Update (13-11-2024)
Somehow i lost the creating of a new image to copy all the builder items from. It changed nothing about the way the image worked but the image size is much smaller now. It is under 100mb from more then 2gb. Somewhere in my previous debugging we lost this step. So the lastest update can be found below again.
Introduction
In here you will find a working Dockerfile I created with the below defined compatibility needs for myself. This is posted here for my own archive and to share of course. I added the required support for the following.
- HTTP/2 HPACK
- HTTP/3
- Boringssl
- Brotli support
This version of Nginx supports HTTP/3 quic and Brotli compression as a main benefit with some other fixes. Not having Brotli support these days is just a step back in the delorean.
More info about how HPACK accelerates your HTTP/2. And here you can read more details about BoringSSL
Brotli compared to GZIP
As mentioned this is a Nginx version which supports Brotli. Brotli is very useful as it is a better compression algorithm then GZIP. If you are still not convinced watch the video below.
Nginx base Docker file
This is the base of my dockerfile which takes care of all the things mentioned above. You can of course edit this or use different versions of alpine or Nginx regarding this wont break building the other modules. I also updated Nginx, NJS and BoringSSL to the latest versions.
I use this image instead of the official Nginx to load all my web projects. See this as a base image to define all your running webserver projects in. Only you get "free" Brotli compression and HTTP/3 support with it so you can make your sites faster and more up to date.
##################################################
# Nginx with Quiche (HTTP/3) and Brotli
# VERSION 1.05 / 13-11-2024
##################################################
# This build compiles Nginx with Brotli and
# HTTP/3 support. It uses a multi-stage build to
# minimize the final image size by copying only
# the necessary artifacts from the builder image.
# Based on Alpine for a lean and efficient image.
##################################################
# Builder Stage
FROM alpine:latest AS builder
# Set the versions of Nginx, NJS, and BoringSSL
ENV NGINX_VERSION=1.27.2
ENV NJS_VERSION=0.8.7
ENV BORINGSSL_VERSION=c59bf8bf189dcbde868e04efcd53b705ed155231
ENV BORINGSSL="/tmp/boring-nginx"
# Build-time metadata
ARG BUILD_DATE
ARG VCS_REF
# Install dependencies for building Nginx
RUN addgroup -S nginx \
&& adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx \
&& apk update \
&& apk upgrade \
&& apk add --no-cache ca-certificates openssl \
&& update-ca-certificates \
&& apk add --no-cache --virtual .build-deps \
gcc libc-dev make pcre-dev zlib-dev linux-headers gnupg \
libxslt-dev gd-dev geoip-dev perl-dev \
&& apk add --no-cache --virtual .brotli-build-deps \
autoconf libtool automake git g++ cmake go perl rust cargo patch \
libxml2-dev byacc flex libstdc++ libmaxminddb-dev lmdb-dev file openrc
# Install and build BoringSSL
RUN mkdir $BORINGSSL \
&& cd $BORINGSSL \
&& git clone https://boringssl.googlesource.com/boringssl \
&& cd boringssl \
&& git checkout -q ${BORINGSSL_VERSION} \
&& mkdir build \
&& cd build \
&& cmake -DBUILD_SHARED_LIBS=1 .. \
&& make
# Prepare BoringSSL for Nginx compilation
RUN mkdir -p "$BORINGSSL/boringssl/.openssl/lib" \
&& cd "$BORINGSSL/boringssl/.openssl" \
&& ln -s ../include include \
&& cd "$BORINGSSL/boringssl" \
&& cp "build/crypto/libcrypto.so" ".openssl/lib" \
&& cp "build/ssl/libssl.so" ".openssl/lib" \
&& cp ".openssl/lib/libssl.so" /usr/lib/ \
&& cp ".openssl/lib/libcrypto.so" /usr/lib/
# Clone Nginx modules
RUN mkdir /usr/src \
&& cd /usr/src \
&& git clone --depth=1 --recursive --shallow-submodules https://github.com/google/ngx_brotli \
&& git clone --branch $NJS_VERSION --depth=1 --recursive --shallow-submodules https://github.com/nginx/njs
# Build Nginx with modules
RUN cd /usr/src \
&& wget -qO nginx.tar.gz https://nginx.org/download/nginx-$NGINX_VERSION.tar.gz \
&& tar -zxC /usr/src -f nginx.tar.gz \
&& rm nginx.tar.gz \
&& cd /usr/src/nginx-$NGINX_VERSION \
&& mkdir /root/.cargo \
&& echo $'[net]\ngit-fetch-with-cli = true' > /root/.cargo/config.toml \
&& ./configure --prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-pcre-jit \
--with-http_ssl_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_random_index_module \
--with-http_secure_link_module \
--with-http_stub_status_module \
--with-http_auth_request_module \
--with-http_xslt_module=dynamic \
--with-http_image_filter_module=dynamic \
--with-http_geoip_module=dynamic \
--with-http_perl_module=dynamic \
--with-threads \
--with-stream \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-stream_realip_module \
--with-stream_geoip_module=dynamic \
--with-http_slice_module \
--with-mail \
--with-mail_ssl_module \
--with-compat \
--with-file-aio \
--with-http_v2_module \
--with-http_v3_module \
--add-module=/usr/src/ngx_brotli \
--add-module=/usr/src/njs/nginx \
--with-cc-opt="-I$BORINGSSL/boringssl/include -Wno-error" \
--with-ld-opt="-L$BORINGSSL/boringssl/build/ssl -L$BORINGSSL/boringssl/build/crypto" \
--with-select_module \
--with-poll_module \
--build="docker-nginx-http3-$VCS_REF-$BUILD_DATE ngx_brotli-$(git --git-dir=/usr/src/ngx_brotli/.git rev-parse --short HEAD) njs-$(git --git-dir=/usr/src/njs/.git rev-parse --short HEAD)" \
&& touch "$BORINGSSL/boringssl/.openssl/include/openssl/ssl.h" \
&& make -j$(getconf _NPROCESSORS_ONLN) \
&& make install \
&& rm -rf /etc/nginx/html/ \
&& mkdir /etc/nginx/conf.d/ \
&& mkdir -p /usr/share/nginx/html/ \
&& install -m644 html/index.html /usr/share/nginx/html/ \
&& install -m644 html/50x.html /usr/share/nginx/html/ \
&& ln -s /usr/lib/nginx/modules /etc/nginx/modules \
&& strip /usr/sbin/nginx* \
&& strip /usr/lib/nginx/modules/*.so \
&& rm -rf /etc/nginx/*.default /etc/nginx/*.so \
&& rm -rf /usr/src \
&& apk add --no-cache --virtual .gettext gettext \
&& mv /usr/bin/envsubst /tmp/ \
\
# Determine runtime dependencies
&& runDeps="$( \
scanelf --needed --nobanner /usr/sbin/nginx /usr/lib/nginx/modules/*.so /tmp/envsubst \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u \
)" \
&& echo "$runDeps" > /nginx-run-deps.txt \
\
# Cleanup
&& apk del .brotli-build-deps .build-deps .gettext \
&& rm -rf /root/.cargo \
&& rm -rf /var/cache/apk/* \
&& mv /tmp/envsubst /usr/local/bin/ \
&& mkdir -p /etc/ssl/private \
&& openssl req -x509 -newkey rsa:4096 -nodes -keyout /etc/ssl/private/localhost.key -out /etc/ssl/localhost.pem -days 365 -sha256 -subj '/CN=localhost' \
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
# Final Stage
FROM alpine:latest
# Copy Nginx and related files from the builder stage
COPY --from=builder /usr/sbin/nginx /usr/sbin/nginx
COPY --from=builder /etc/nginx /etc/nginx
COPY --from=builder /usr/lib/nginx /usr/lib/nginx
COPY --from=builder /usr/share/nginx /usr/share/nginx
COPY --from=builder /usr/local/bin/envsubst /usr/local/bin/envsubst
COPY --from=builder /var/log/nginx /var/log/nginx
COPY --from=builder /var/cache/nginx /var/cache/nginx
COPY --from=builder /var/run /var/run
COPY --from=builder /etc/ssl /etc/ssl
COPY --from=builder /usr/lib/libssl.so* /usr/lib/
COPY --from=builder /usr/lib/libcrypto.so* /usr/lib/
COPY --from=builder /nginx-run-deps.txt /nginx-run-deps.txt
# Install runtime dependencies
RUN apk add --no-cache ca-certificates openssl libstdc++ libgcc \
&& apk add --no-cache $(cat /nginx-run-deps.txt) \
&& rm /nginx-run-deps.txt \
&& update-ca-certificates \
\
# Forward logs to Docker
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
# Create nginx user and group
RUN addgroup -S nginx \
&& adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx
# Expose ports
EXPOSE 80 443
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]Nginx.conf sample
Here you will find a sample nginx.conf which give you an idea of how i run my nginx environment.
The most important difference in this file compared to the normal nginx.conf should be the brotli compression settings.
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 1024;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
fastcgi_send_timeout 300s;
fastcgi_read_timeout 300s;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
##
# Gzip Settings
##
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Brotli Settings (modify the brotli_comp_level to match your optimal cpu load vs compression)
##
brotli on;
brotli_static on;
brotli_comp_level 11;
brotli_types
text/plain
text/css
text/xml
text/javascript
text/x-component
application/xml
application/xml+rss
application/javascript
application/json
application/atom+xml
application/vnd.ms-fontobject
application/x-font-ttf
application/x-font-opentype
application/x-font-truetype
application/x-web-app-manifest+json
application/xhtml+xml
application/octet-stream
font/opentype
font/truetype
font/eot
font/otf
font/woff
font/woff2
image/svg+xml
image/x-icon
image/vnd.microsoft.icon
image/bmp;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
}
Site.conf sample
You need to watch out that the higher you set the brotli compression the more cpu intensive it becomes. TTFB (time to first byte) can become an issue when the compression ratio is set to high with to slow kubernetes/ high loads or everything in between. so you need to test on real world scenario's and adjust accordingly. This is not only a Brotli issue but Gzip also has the same disadvantages when setting compression levels to high while the decrease in file size isn't worth it.
You can use this file to take all the HTTP/3 settings defined here below. This will optimize and enable HTTP/3 support for your site.
server {
# HTTP/3 and Quic Listen
listen 443 quic reuseport;
http3 on;
# HTTP/3 Add Alt-Svc header negotiation.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
# HTTP/2 Listen
listen 443 ssl http2;
http2 on;
# Certificate locations for HTTPS.
ssl_certificate /etc/ssl/certs/wildcard-cert-23.crt;
ssl_certificate_key /etc/ssl/private/wildcard-cert-23.key;
# Enable SSL settings for performance.
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;
# Enable SSL settings for security.
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Base settings (root and hostname).
root /var/www/html;
index index.php index.html index.htm;
server_name kubernetes.servername.nl;
# HTTP/2 settings
http2_push_preload on;
# /End of HTTP/2.
# HTTP/3 settings
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
# @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
quic_retry on;
quic_gso on;
quic_host_key /etc/ssl/private/wildcard-cert-23.key;
# /End of HTTP/3.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
try_files $uri =404;
# Bunch of PHP and FPM settings
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass your-php-fpm-instance:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info
# Add a bunch of other headers based on your desired settings.
add_header X-protocol $server_protocol always;
add_header X-Cache $upstream_cache_status always;
add_header X-Frame-Options SAMEORIGIN always;
add_header X-Content-Type-Options nosniff always;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload" always;
add_header Referrer-Policy strict-origin-when-cross-origin;
add_header Pragma public;
add_header Cache-Control "public";
}
}
Multiple H3 Connections from a single Nginx
Here i will explain how you can make multiple server blocks within Nginx work with HTTP/3. This has some caveats since http3 uses UDP with the setting reuseport which makes it unable to assign port 443 multiple times when using HTTP/3
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;This part of the header informs that our HTTP/3 connection accepts port 443. When you have a single nginx container setup and you have more server blocks configured you will get issues regarding the reuse for port 443.
With Http3 you can only define a single port once within a server block. So if you have more server blocks you have the option to let your website communicate over different ports then 443 using this alt-svc header which tells the browser it accepts HTTP/3 and on which port.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":444"; ma=86400, h3=":444"; ma=86400' always;Check out the differences in the h3 ports here. This way you can make multiple server blocks use h3 by specifying unique ports.
All you need to do is make your instance listen to the right port specified in your alt-svc header. If there is a successful connection possible the browser will try to switch to h3. Below a sample how you can listen to port 443 and 444 with HTTP3
server {
server_name http3-port-443
# HTTP/3 and Quic Listen
listen 443 quic reuseport;
http3 on;
# HTTP/3 Add Alt-Svc header negotiation.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400' always;
# HTTP/2 Listen
listen 443 ssl http2;
http2 on;
# HTTP/3 settings
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
# @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
quic_retry on;
quic_gso on;
quic_host_key /etc/ssl/private/wildcard-cert-23.key;
# /End of HTTP/3.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
}server {
server_name http3-port-444
# HTTP/3 and Quic Listen
listen 444 quic reuseport;
http3 on;
# HTTP/3 Add Alt-Svc header negotiation.
add_header alt-svc 'h2=":443"; ma=86400, h3-29=":444"; ma=86400, h3=":444"; ma=86400' always;
# HTTP/2 Listen
listen 443 ssl http2;
http2 on;
# HTTP/3 settings
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
# @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
quic_retry on;
quic_gso on;
quic_host_key /etc/ssl/private/wildcard-cert-23.key;
# /End of HTTP/3.
location / {
try_files $uri $uri/ /index.php?$query_string;
}
}It is that easy to make your Reverse proxy or multiple server blocks establish a H3 connection. You do have to make the right port forwards within your firewall/network also.
When using docker you configure this as shown below. When specifying UDP you have to specify every port seperately in docker. Below is my sample which includes UDP ports 443 up to 449.
services:
############################################################
# proxy webservice to connect to multiple nginx instances
############################################################
reverse-proxy:
image: custom-build-nginx:latest
restart: unless-stopped
ports:
- '80:80'
- '443:443'
- '443:443/udp'
- '444:444/udp'
- '445:445/udp'
- '446:446/udp'
- '447:447/udp'
- '448:448/udp'
- '449:449/udp'
volumes:Update (03-04-2024)
Added some versioning to my dockerfile to better compare versions. I also cleaned up the build from yesterday and finally decided to add a ENV for setting the boringssl version also. Since their build broke my dockerfile we better update it manually. I also updated nginx to 1.25.4 and NJS to 0.8.3 and verified the build
Update (02-04-2024)
I only recently found out that an update to BoringSSL done around 28 february broke my build of BoringSSL. below the updated dockerfile that compiles again. I had to build libssl as a shared library for it to function again. I will cleanup this image a bit more but it is at least working again.
Update (18-03-2024)
Added a section to explain how you can use multiple H3 connections when you have a Nginx that has more than 1 server block. This also holds the key when you use Nginx as a Reverse Proxy and want to make use of H3.
Update (04-09-2023)
I found out that this build didnt compile anymore due to a change in the brotli source code? It might had something to do with updates from github i didnt version in this Dockerfile. I completely reworked this file to work with the updated Nginx 1.25.2 which has "built in" HTTP/3 support. I also made sure we have native Brotli support again. This should keep functioning from now on.
Update (30-08-2023)
I removed the modsec support there was before. This was giving random Errors and since i wont use it i removed it. It might return later.