Rails on Kubernetes - Part 2
Overview
This is Part 2 of the Rails on Kubernetes series.
- Part 1: Rails and Docker Compose
- Part 2: Kubernetes
- Part 3: Deployments, Rolling updates and Scaling
Updates
- 2017-11-13: Updated all docker images to pull from Alpine.
Code
You can skip straight to the completed deployment here Rails app on Github.
Rails Docker Image
If you followed my previous post you should have a Rails application working with Docker Compose.
In order to prep for the Kubernetes deploy we need to package the Rails Docker image and push to a Docker registry. I'll use a public Docker Hub repo for simplicity, but you should always store your images in a private registry if your source code lives in the image.
bin/rails g task docker push_image
We create a rake task for this. It's very simple, we're grabbing the latest git revision hash and using that as an image tag. We're also attaching a latest. Note that it's recommended you use the actual commit hash in your Kube deploys to increase visibility into what's running on each pod - I'm just tagging latest for simplicity.
namespace :docker do
desc "Push docker images to DockerHub"
task :push_image do
TAG = `git rev-parse --short HEAD`.strip
puts "Building Docker image"
sh "docker build -t tzumby/rails-app:#{TAG} ."
IMAGE_ID = `docker images | grep tzumby\/rails-app | head -n1 | awk '{print $3}'`.strip
puts "Tagging latest image"
sh "docker tag #{IMAGE_ID} tzumby/rails-app:latest"
puts "Pushing Docker image"
sh "docker push tzumby/rails-app:#{TAG}"
sh "docker push tzumby/rails-app:latest"
puts "Done"
end
end
Now we can run rake docker:push_image every time we want to push a new image version.
Kubernetes
We are ready to translate the docker-compose.yml config into Kubernetes resources. For our local development we accessed the app under the classic port 3000 but with Kubernetes we will setup an Ingress resource running Nginx and proxy requests to a Rails service.
Postgres
We'll start with our Postgres server. Here are the Kube resources we will use to create the DB:
- Service
- The service maps traffic from the Ingress to our pods. There are a several ways to do this via the
typeoption: NodePort, ClusterIP, LoadBalancer or ExternalName. If we don't specify anything, Kube will create the service as ClusterIP - meaning it will only be accessible from within the cluster. This is what we want for Postgres as we will only be connecting to it from the Rails pods.
- The service maps traffic from the Ingress to our pods. There are a several ways to do this via the
- Secret
- This is an object used for storing sensitive information such as passwords or TLS certificates. We will create one for our Postgres username and password.
- Persistent Volume (PV).
- Pod storage is ephemeral just like the container file system. Using Kube's API we can allocate space using a number of different file systems (local, EBS, CephFS, iSCSI etc.)
- Persistent Volume Claim (PVC)
- If the PV allocates the space, the PVC binds that resource to our Pods.
- Replication Controller (RC)
- The RC controls the life cycle of our Pods. We specify what image to pull, how to mount the persistent volumes, what commands to run and define ENV variables to be used by our app.
Let's create the secret to store our username and password first:
$ kubectl create secret generic db-user-pass --from-literal=password=mysecretpass
$ kubectl create secret generic db-user --from-literal=username=postgres
Here is the yaml file containing the Service, PV, PVC and RC objects:
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
app: rails-kube-app
spec:
ports:
- port: 5432
selector:
app: rails-kube-app
tier: postgres
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-pv
labels:
type: local
spec:
capacity:
storage: 4Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
---
apiVersion: v1
kind: ReplicationController
metadata:
name: postgres
labels:
app: rails-kube-app
spec:
replicas: 1
selector:
app: rails-kube-app
tier: postgres
template:
metadata:
name: postgres
labels:
app: rails-kube-app
tier: postgres
spec:
volumes:
- name: postgres-pv
persistentVolumeClaim:
claimName: postgres-pvc
containers:
- name: postgres
image: postgres:9.6-alpine
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: db-user
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: db-user-pass
key: password
- name: POSTGRES_DB
value: rails-kube-demo_production
- name: PGDATA
value: /var/lib/postgresql/data
ports:
- containerPort: 5432
volumeMounts:
- mountPath: "/var/lib/postgresql/data"
name: postgres-pv
Names and labels
name - uniquely identifies the object we are creating. A Service or a Replaication Controller will use the value directly because Kube will create one one resource for each. The Pods created by the RC will append a random string to the name because the number of pods is dynamic and depends on the number of replicas we specify in a Replication Controller. If name the Postgres RC postgres this is what we'll also use in our database.yml for example - Kube's DNS will resolve that to the proper resource.
labels - are key-value pairs used to organize and group resources. For example: environment (dev, staging, production) or tier (frontend, backend, database, cache). Given those examples we could run queries against our system such as:
kubectl get pods -l environment=production,tier=frontend
Back to our Postgres deploy, let's run this to create the Kube resources:
kubectl create -f postgres.yaml
We can verify that the pod ran successfully:
kubectl get pods -w
NAME READY STATUS RESTARTS AGE
postgres-k3mqv 1/1 Running 0 1m
Redis Deployment
Next is the Redis deployment. We're not using volumes here for simplicity's sake. This could be a problem in a production environment: we will loose the memory stored Redis data if we re-deploy. If there are unprocessed jobs store in there that could be a big problem. I would recommend looking a setting up a Redis cluster with Kube for a production environment.
apiVersion: v1
kind: Service
metadata:
name: redis
labels:
app: rails-kube-app
spec:
ports:
- port: 6379
selector:
app: rails-kube-app
tier: redis
---
apiVersion: v1
kind: ReplicationController
metadata:
name: redis
spec:
replicas: 1
selector:
app: rails-kube-app
tier: redis
template:
metadata:
name: redis
labels:
app: rails-kube-app
tier: redis
spec:
containers:
- name: redis
image: redis:3.2-alpine
ports:
- containerPort: 6379
We'll run this yaml file as well:
kubectl create -f redis.yaml
Now we should have both Postgres and Redis running:
NAME READY STATUS RESTARTS AGE
postgres-k3mqv 1/1 Running 0 5m
redis-dz20q 1/1 Running 0 1m
Rails Migrations
In my previous post I ran the migrations using a separate docker-compose service and passing it a bash script that waited for the Postgres server to go up and then ran the migrations.
Kubernetes provides a special Job resource for this.
This is pretty neat, Kube will bring this Pod up, run the command and then shut it down. Note that we're running the db:create and db:migrate with a single Job. Normally you would create a separate job for the db creation and another one for ongoing jobs like running migrations.
We already have the kube secrets for the Postgres DB. Let's create another one for the secret key base that Rails will use in production as well:
$ kubectl create secret generic secret-key-base --from-literal=secret-key-base=50dae16d7d1403e175ceb2461605b527cf87a5b18479740508395cb3f1947b12b63bad049d7d1545af4dcafa17a329be4d29c18bd63b421515e37b43ea43df64
And this is our Kube Job:
apiVersion: batch/v1
kind: Job
metadata:
name: setup
spec:
template:
metadata:
name: setup
spec:
containers:
- name: setup
image: tzumby/rails-app:latest
args: ["rake db:create && rake db:migrate"]
env:
- name: DATABASE_NAME
value: "rails-kube-demo_production"
- name: DATABASE_URL
value: "postgres"
- name: DATABASE_PORT
value: 5432
- name: DATABASE_USER
valueFrom:
secretKeyRef:
name: "db-user"
key: "username"
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: "db-user-pass"
key: "password"
- name: RAILS_APP
value: "production"
- name: REDIS_URL
value: "redis"
- name: REDIS_PORT
value: "6379"
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: "secret-key-base"
key: "secret-key-base"
restartPolicy: Never
Running the job is predictable:
kubectl create -f setup.yaml
Let's check if everything ran successfully:
kubectl get jobs
NAME DESIRED SUCCESSFUL AGE
setup 1 1 1m
Rails app
We are now ready to deploy the Rails app. It's a pretty standard config with a Service and a Replication Controller. We're using the handy RAILS_LOG_TO_STDOUT environment flag to trigger Rails logging to stdout. Since the Nginx server runs on a separate server we'll have to serve the static assets from Rails. There are a few ways to go around this if we wanted Nginx to serve the assets without hitting the Rails server:
- We could run an Nginx instance on the Rails pods and configure it to serve the assets.
- We could customize the Nginx Ingress Controller and copy the assets there on each deploy (this one doesn't seem feasable).
For now let's configure Rails to server its own assets via RAILS_SERVE_STATIC_FILES. I think the best compromise for a production setup would be to configure the Pagespeed module in the Nginx Intress Controller - but that's a topic for another post.
apiVersion: v1
kind: Service
metadata:
name: rails
labels:
app: rails-kube-app
spec:
ports:
- port: 3000
selector:
app: rails-kube-app
tier: rails
---
apiVersion: v1
kind: ReplicationController
metadata:
name: rails
spec:
replicas: 1
selector:
app: rails-kube-app
tier: rails
template:
metadata:
name: rails
labels:
app: rails-kube-app
tier: rails
spec:
containers:
- name: rails
image: tzumby/rails-app:latest
args: ["rails s -p 3000 -b 0.0.0.0"]
env:
- name: DATABASE_URL
value: "postgres"
- name: DATABASE_NAME
value: "rails-kube-demo_production"
- name: DATABASE_PORT
value: "5432"
- name: DATABASE_USER
valueFrom:
secretKeyRef:
name: "db-user"
key: "username"
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: "db-user-pass"
key: "password"
- name: RAILS_APP
value: "production"
- name: RAILS_LOG_TO_STDOUT
value: "true"
- name: RAILS_SERVE_STATIC_ASSETS
value: "true"
- name: REDIS_URL
value: "redis"
- name: REDIS_PORT
value: "6379"
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: "secret-key-base"
key: "secret-key-base"
ports:
- containerPort: 3000
The args parameter is equivalent to a Dockerfile's CMD and it will add that command as an argument to our ENTRYPOINT bash script we defined earlier.
Let's run this one as well:
kubectl create -f rails.yaml
Sidekiq
The Sidekiq deploy is very similar to Rails. The only exception is the args we pass to the ENTRYPOINT:
apiVersion: v1
kind: ReplicationController
metadata:
name: sidekiq
spec:
replicas: 1
selector:
app: rails-kube-app
tier: sidekiq
template:
metadata:
name: sidekiq
labels:
app: rails-kube-app
tier: sidekiq
spec:
containers:
- name: sidekiq
image: tzumby/rails-app:latest
args: ["sidekiq -C config/sidekiq.yml"]
env:
- name: DATABASE_NAME
value: "rails-kube-demo_production"
- name: DATABASE_PORT
value: "5432"
- name: DATABASE_URL
value: "postgres"
- name: DATABASE_USER
valueFrom:
secretKeyRef:
name: "db-user"
key: "username"
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: "db-user-pass"
key: "password"
- name: RAILS_APP
value: "production"
- name: REDIS_URL
value: "redis"
- name: REDIS_PORT
value: "6379"
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: "secret-key-base"
key: "secret-key-base"
Note that I'm also not specifying a port number. We will not connect to this Pod directly: the sidekiq worker connects to the Redis service and listens for jobs.
Ingress Controller
We're almost ready to access our newly deployed Rails application. I created a simple Ingress resource that listens for rails.local HOST header and targets a service called rails on port 3000. This matches what I defined in my Rails kube deployment:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: rails-demo-ing
spec:
rules:
- host: rails.local
http:
paths:
- backend:
serviceName: rails
servicePort: 3000
path: /
We use the handy create command to spin this one:
kubectl create -f ingress.yaml
Now we can access our app at http://rails.local. I added an entry in my /etc/hosts file that points the minikube IP to the rails.local domain:
~ minikube ip
192.168.64.2
And my hosts file:
192.168.64.2 rails.local
If you deploy this on AWS or GCE you can have the option to spin a Load Balancer when you create an Ingress. You would then take the LB's Address and create the proper DNS records in your domain.
If you are running minikube locally, make sure you enable the nginx ingress addon by running:
minikube addons enable ingress
Then check if it's running in your kube-system namespace:
kubectl -n kube-system get po -w
You should see the ingress controller and the default backend running:
NAME READY STATUS RESTARTS AGE
default-http-backend-cgf5r 1/1 Running 0 2m
nginx-ingress-controller-gprm2 1/1 Running 0 2m
Now you can point your browser to http://rails.local and access your newly deployed app.
What's next
In the next post we will look at a few deployments and rolling updates scenarios. We'll also use apache bench to load test our setup and see how quickly we can respond to increase in load.