Nov 17, 2017 9 min read

Kubernetes WAF

Motivation

I'm not going to attempt to properly define what a Web Application Firewall (WAF) is. I will just say this: unlike a traditional firewall that filters/routes ports & TCP packets, a WAF inspects the actual request/response data being communicated. This could be the Headers, HTTP query parameters, HTTP body etc.

As you might expect, this can affect performance significantly. No matter what the internals of the WAF are, the request/response data has to be parsed and interpreted somehow. Most commonly this is done via Regular Expressions and compared against a set of known rules. Regular expressions are slow, especially if you have a lot of rules to validate the traffic against. To improve that, most engines use the PCRE library with JIT to compile and cache the regex rules.

A project called lua-resty-waf takes this even further and implements the WAF engine using LUA running on top of the OpenResty framework. And this is blazing fast.

My goal for the rest of this article is to document how I integrated OpenResty and lua-resty-waf with a Kubernetes Ingress Controller.

Kubernetes

When it comes to running the WAF in Kube we have two options:

Run the WAF as a reverse proxy with each Replication Controller or Deployment.
Install the WAF at the Ingress Controller level.

The second option has the advantage of protecting all the apps that run behind that Ingress Controller. In order to make this feasible when running multiple applications, we will have to pick the base Ingress Controller very carefully. Let me list how I envision the requirements for running this:

The WAF engine gets initialized on the Ingress Controller.
Rules are enabled for each app via the Ingress resource.

This flow has the benefit of allowing applications to specify their own rules. Those rules can live in the source control along with all the Kube resource definitions.

Currently there are two main implementations of the Nginx Ingress Controller:

the one created by the Nginx team
the Kubernetes official Nginx Ingress Controller

Now, the Kubernetes implementation would normally be the go-to project, but unfortunately they don't support customizing the Nginx configs via Ingress annotations or configmaps (unlike the Nginx implementation).

I should mention that recently Kubernete's Nginx Ingress Controller added support for the ModSecurity WAF. If you want to go that route you can follow their instructions.

I will describe a way to customize the Nginx's Ingress Controller to use OpenResty instead of Nginx and package the lua-resty-waf as part of the Docker image.

Docker image

My starting point for the OpenResty Waf image is the official OpenResty image.

Below you can see my updated Dockerfile. I could have just pulled from the base image but since I want to update the configuration flags I need to modify the original. We want OpenResty to match the Nginx binary and folders so we can seamlessly swap it in the Ingress Controller.

# Dockerfile - Ubuntu Xenial
# https://github.com/openresty/docker-openresty

FROM ubuntu:xenial

MAINTAINER Evan Wies <evan@neomantra.net>

# Docker Build Arguments
ARG RESTY_VERSION="1.11.2.3"
ARG RESTY_LUAROCKS_VERSION="2.3.0"
ARG RESTY_OPENSSL_VERSION="1.0.2k"
ARG RESTY_PCRE_VERSION="8.39"
ARG RESTY_J="1"
ARG RESTY_CONFIG_OPTIONS="\
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --modules-path=/usr/lib/nginx/modules \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --user=nginx \
    --group=nginx \
    --with-file-aio \
    --with-http_addition_module \
    --with-http_auth_request_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_geoip_module \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_image_filter_module=dynamic \
    --with-http_mp4_module \
    --with-http_random_index_module \
    --with-http_realip_module \
    --with-http_secure_link_module \
    --with-http_slice_module \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_sub_module \
    --with-http_v2_module \
    --with-http_xslt_module=dynamic \
    --with-ipv6 \
    --with-mail \
    --with-mail_ssl_module \
    --with-md5-asm \
    --with-pcre-jit \
    --with-sha1-asm \
    --with-stream \
    --with-stream_ssl_module \
    --with-threads \
    "

# These are not intended to be user-specified
ARG _RESTY_CONFIG_DEPS="--with-openssl=/tmp/openssl-${RESTY_OPENSSL_VERSION} --with-pcre=/tmp/pcre-${RESTY_PCRE_VERSION}"


# 1) Install apt dependencies
# 2) Download and untar OpenSSL, PCRE, and OpenResty
# 3) Build OpenResty
# 4) Cleanup

RUN useradd -ms /bin/false nginx

RUN \
    DEBIAN_FRONTEND=noninteractive apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        build-essential \
        ca-certificates \
        curl \
        libgd-dev \
        libgeoip-dev \
        libncurses5-dev \
        libperl-dev \
        libreadline-dev \
        libxslt1-dev \
        make \
        perl \
        unzip \
        zlib1g-dev \
        git \
        python \
        liblua5.1-0-dev \
    && cd /usr/local \
    && git clone https://github.com/SpiderLabs/owasp-modsecurity-crs \
    && cd /tmp \
    && curl -fSL https://www.openssl.org/source/openssl-${RESTY_OPENSSL_VERSION}.tar.gz -o openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
    && tar xzf openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
    && curl -fSL https://ftp.pcre.org/pub/pcre/pcre-${RESTY_PCRE_VERSION}.tar.gz -o pcre-${RESTY_PCRE_VERSION}.tar.gz \
    && tar xzf pcre-${RESTY_PCRE_VERSION}.tar.gz \
    && curl -fSL https://openresty.org/download/openresty-${RESTY_VERSION}.tar.gz -o openresty-${RESTY_VERSION}.tar.gz \
    && tar xzf openresty-${RESTY_VERSION}.tar.gz \
    && git clone --recursive https://github.com/p0pr0ck5/lua-resty-waf \
    && cd /tmp/openresty-${RESTY_VERSION} \
    && ./configure -j${RESTY_J} ${_RESTY_CONFIG_DEPS} ${RESTY_CONFIG_OPTIONS} \
    && make -j${RESTY_J} \
    && make -j${RESTY_J} install \
    && cd /tmp/pcre-${RESTY_PCRE_VERSION} \
    && ./configure && make && make install \
    && cd /tmp \
    && rm -rf \
        openssl-${RESTY_OPENSSL_VERSION} \
        openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
        openresty-${RESTY_VERSION}.tar.gz openresty-${RESTY_VERSION} \
        pcre-${RESTY_PCRE_VERSION}.tar.gz pcre-${RESTY_PCRE_VERSION} \
    && curl -fSL http://luarocks.org/releases/luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz -o luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz \
    && tar xzf luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz \
    && cd luarocks-${RESTY_LUAROCKS_VERSION} \
    && ./configure \
        --prefix=/etc/nginx/luajit \
        --with-lua=/etc/nginx/luajit \
        --lua-suffix=jit-2.1.0-beta2 \
        --with-lua-include=/etc/nginx/luajit/include/luajit-2.1 \
    && make build \
    && make install \
    && cd /tmp/lua-resty-waf \
    && export PATH=$PATH:/etc/nginx/luajit/bin:/etc/nginx/bin \
    && make OPENRESTY_PREFIX=/etc/nginx && make install OPENRESTY_PREFIX=/etc/nginx \
    && cd /tmp \
    && rm -rf luarocks-${RESTY_LUAROCKS_VERSION} luarocks-${RESTY_LUAROCKS_VERSION}.tar.gz \
    && DEBIAN_FRONTEND=noninteractive apt-get autoremove -y \
    && ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log

# Download of GeoIP databases
RUN curl -sSL -o /etc/nginx/GeoIP.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz \
  && curl -sSL -o /etc/nginx/GeoLiteCity.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz \
  && gunzip /etc/nginx/GeoIP.dat.gz \
  && gunzip /etc/nginx/GeoLiteCity.dat.gz

# Add additional binaries into PATH for convenience
ENV PATH=$PATH:/etc/nginx/luajit/bin/:/etc/nginx/bin/

# TODO: remove any other apt packages?
ENTRYPOINT ["/usr/sbin/nginx", "-g", "daemon off;"]

Next I built my own image from this modified Dockerfile:

$ docker build -t tzumby/openresty-waf .
$ docker push tzumby/openresty-waf

Nginx Ingress Controller

Now that we have the docker image built and pushed we can modify the Nginx Ingress Controller and replace the base image they are using with our own.

I forked the ingress repo for my updates. You can clone that and follow along, I placed the modifications in a separate folder: openresty-controller. We'll need to compile the nginx-ingress binary as it is not included in the repository:

~ $ cd kubernets-ingress/openresty-controller
~ $ make nginx-ingress BUILD_IN_CONTAINER=1

The command above will compile the nginx-ingress binary in a Go lang docker image. I'm not a Go developer and I must be missing a configuration / environment setting but as it stands the compiled binary gets exported in the kubernetes-ingress/nginx-controller. I'm still looking into this, but for now my temporary hack job fix is to move the binary into my openresty-controller folder (mv ../nginx-controller/nginx-ingress).

Now we can build the Docker image for the ingress controller:

FROM tzumby/openresty-waf

# forward nginx access and error logs to stdout and stderr of the ingress
# controller process
RUN ln -sf /proc/1/fd/1 /var/log/nginx/access.log \
        && ln -sf /proc/1/fd/2 /var/log/nginx/error.log

RUN mkdir -p /etc/nginx/conf.d

COPY nginx-ingress nginx/ingress.tmpl nginx/nginx.conf.tmpl /

ENTRYPOINT ["/nginx-ingress"]

And run the docker build:

docker build -t tzumby/openresty-ingress-waf .

Kubernetes Ingress Controller

We now create the Ingress Controller. We'll use one of the example Ingress Controller resources available in the same repo.

Note that we are creating the Ingress Controller in the kube-system namespace.

First let's create a ConfigMap and initialize the lua-resty-waf. Create a file called nginx-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: kube-system
data:
  http-snippets: |
    include /etc/nginx/custom-snippets/waf.conf;

This will include the waf.conf snippet in the http block of our nginx.conf. We will mount the /etc/nginx/custom-snippets/waf.conf file when we create the controller resource.

kubectl create -f nginx-config.yaml

Here's the sample replication controller resource for the ingress controller nginx-ingress-rc.yaml:

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-ingress-rc
  namespace: kube-system
  labels:
    app: nginx-ingress
spec:
  replicas: 1
  selector:
    app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
    spec:
      containers:
      - image: tzumby/openresty-ingress-waf
        imagePullPolicy: Always
        name: nginx-ingress
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        args:
          - -nginx-configmaps=default/nginx-config
        volumeMounts:
        - name: waf-config-volume
          mountPath: /etc/nginx/custom-snippets
      volumes:
        - name: waf-config-volume
          configMap:
            name: waf-config

Few things to note:

we're using the Docker image we just created: tzumby/openresty-ingress-waf
we're passing the -nginx-configmaps=default/nginx-config parameter when starting the container - this is the ConfigMap we created earlier.
we're mounting the snippet as a volume at /etc/nginx/custom-snippets

The last resource we need is the ConfigMap of the lua-resty-waf init block. Create a file called waf.conf:

lua_shared_dict lua_waf_storage 64m;

init_by_lua_block {
    require "resty.core"
    local lua_resty_waf = require "resty.waf"
    lua_resty_waf.init()
}

Now create the ConfigMap from this config file:

kubectl -n kube-system create configmap waf-config --from-file=./waf.conf

Lastly, create the Ingress Controller:

kubectl create -f nginx-ingress-rc.yaml

We can check that everything ran successfully:

$ kubectl get pods
nginx-ingress-rc-bnrvx           1/1       Running   0          1m

Now connect to the running pod to make sure everything ran successfully:

$ kubectl -n kube-system exec -it nginx-ingress-rc-bnrvx /bin/bash

We can check the /etc/nginx/nginx.conf file:

$ cat /etc/nginx/nginx.conf
...
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    include /etc/nginx/custom-snippets/waf.conf;
...

Great, the include directive is there. You can run nginx -t as well for an extra check.

Ingress Resource

Ok, so we now have an Ingress Controller that will listen to the Kubernetes API for Ingress events. When we create our Ingress resource in a few moments, the controller will pick that up and generate a virtual host configuration for that resource.

Here's what we have right now. You can check the lua-resty-waf repo for more configuration options. Note that we're using the annotations to customize the configuration this time.

You can use the Rails app I configured to work with Kubernetes in a previous article. All you have to do is replace the ingress.yaml resource. This is an example of how easy it would be to stand this Ingress Controller in front of any application.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.org/server-snippets: |
      access_by_lua_block {
        local lua_resty_waf = require "resty.waf"

        local waf = lua_resty_waf:new()

        waf:set_option("debug", true)
        waf:set_option("mode", "ACTIVE")

        waf:set_option("event_log_target", "error")

        waf:set_option("event_log_level", ngx.WARN)
        waf:set_option("event_log_request_arguments", true)
        waf:set_option("event_log_request_body", true)
        waf:set_option("event_log_request_headers", true)

        waf:exec()
      }

      header_filter_by_lua_block {
          local lua_resty_waf = require "resty.waf"
          local waf = lua_resty_waf:new()
          waf:exec()
      }

      body_filter_by_lua_block {
          local lua_resty_waf = require "resty.waf"
          local waf = lua_resty_waf:new()
          waf:exec()
      }

      log_by_lua_block {
          local lua_resty_waf = require "resty.waf"
          local waf = lua_resty_waf:new()
          waf:exec()
      }
spec:
  rules:
    - host: rails.local
      http:
        paths:
          - backend:
              serviceName: rails
              servicePort: 3000
            path: /

After we run the kubectl create -f ingress.yaml we can check to see if the controller created our virtual host file:

$ ls /etc/nginx/conf.d
default-myapp.conf

Quick Test

Let's try to run a request that contains a <script> tag. First, stream the logs from the ingress controller:

$ kubectl -n kube-system logs -f nginx-ingress-rc-bnrvx

And from a different terminal run:

$ curl -I 'http://rails.local?query=<script></script>'
HTTP/1.1 403 Forbidden
Server: openresty/1.11.2.3
Date: Mon, 26 Jun 2017 16:59:36 GMT
Content-Type: text/html
Content-Length: 175
Connection: keep-alive

This is what you should see in your controller logs:

2017/06/30 05:13:12 [warn] 256#256: *118 [lua] log.lua:52: {"timestamp":1498799592,"request_headers":{"host":"rails.local","accept":"*\/*","user-agent":"curl\/7.51.0"},"id":"7f147af8eafd029da1be","method":"HEAD","uri":"\/","client":"192.168.64.1","uri_args":{"query":"<script><\/script>"},"alerts":[{"msg":"XSS (Cross-Site Scripting)","id":42059,"match":1},{"msg":"XSS (Cross-Site Scripting) - HTML Tag Handler","id":42069,"match":1},{"msg":"XSS (Cross-Site Scripting) - IE Filter","id":42083,"match":1},{"logdata":12,"msg":"Request score greater than score threshold","id":99001,"match":12}]} while logging request, client: 192.168.64.1, server: rails.local, request: "HEAD /?query=<script></script> HTTP/1.1", host: "rails.local"
192.168.64.1 - - [30/Jun/2017:05:13:12 +0000] "HEAD /?query=<script></script> HTTP/1.1" 403 0 "-" "curl/7.51.0" "-"

Conclusion

Since running this requires a lot of manual compilation steps, having a complete Docker image with the WAF makes it very easy to deploy it quickly.

I haven't covered logging for the WAF but since the project supports pushing log messages to a Socket it would be very easy to parse and visualize them. Alternatively you can go with the default option and have the WAF logs alongside your nginx logs.

Since lua resty waf supports loading ModSecurity rules you can stay up to date with the latest rule sets (such as CRS 3) and also get the speed benefits of Lua.