Rails on Kubernetes - Part 3

Overview

This is Part 3 of the Rails on Kubernetes series.

• Part 1: Rails and Docker Compose
• Part 2: Kubernetes
• Part 3: Deployments, Rolling updates and Scaling

Code

If you don't want to read the previous posts and just follow along you can clone my repo and get going right away.

Deployments

So far we created our Pods using ReplicationControllers. While this is fine, the recommended way is to use Deployments instead. This is a new feature in Kubernetes that allows you to control Rolling Upgrades, Rollbacks and more.

This is how the basic template looks:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  minReadySeconds: 1 # number of seconds after pod is up that it's considered ready (default 0)
  strategy:
    type: RollingUpdate # the other option is Recreate where all pods killed before update
    maxUnavailable: 1 # max number of pods that can become unavailable during update
    maxSurge: 1 # max number of extra pods created during deploy (3 + 1 in our case)
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

I went ahead and re-organized the /kube folder in our application. You can see the changes in this commit log.

This is our folder structure now:

├── deployments
│   ├── postgres_deploy.yaml
│   ├── rails_deploy.yaml
│   ├── redis_deploy.yaml
│   └── sidekiq_deploy.yaml
├── ingresses
│   └── ingress.yaml
├── jobs
│   └── setup.yaml
├── services
│   ├── postgres_svc.yaml
│   ├── rails_svc.yaml
│   └── redis_svc.yaml
└── volumes
    └── postgres_volumes.yaml

Initial Deploy

Since we changed things around we should re-deploy. You can tear down all the resources you created on your own. I will start on a fresh namespace:

~ $ kubectl create namespace rails-staging

Let's create the secrets:

~ $ kubectl -n rails-staging create secret generic db-user-pass --from-literal=password=mysecretpass
~ $ kubectl -n rails-staging create secret generic db-user --from-literal=username=postgres
~ $ kubectl -n rails-staging create secret generic secret-key-base --from-literal=secret-key-base=50dae16d7d1403e175ceb2461605b527cf87a5b18479740508395cb3f1947b12b63bad049d7d1545af4dcafa17a329be4d29c18bd63b421515e37b43ea43df64

Now we can run Redis and Postgres:

~ $ kubectl -n rails-staging create -f kube/volumes/postgres_volumes.yaml
~ $ kubectl -n rails-staging create -f kube/services/postgres_svc.yaml
~ $ kubectl -n rails-staging create -f kube/services/redis_svc.yaml
~ $ kubectl -n rails-staging create -f kube/deployments/postgres_deploy.yaml --record
~ $ kubectl -n rails-staging create -f kube/deployments/redis_deploy.yaml --record

Finally, let's run the Setup job for migrations and the Rails app:

~ $ kubectl -n rails-staging create -f kube/jobs/setup.yaml
~ $ kubectl -n rails-staging create -f kube/services/rails_svc.yaml
~ $ kubectl -n rails-staging create -f kube/deployments/rails_deploy.yaml --record
~ $ kubectl -n rails-staging create -f kube/deployments/sidekiq_deploy.yaml --record
~ $ kubectl -n rails-staging create -f kube/ingresses/ingress.yaml

Rollout History

Because we ran our deployment resources with the --record flag we can now access a Revision History for each deploy.

~ $ kubectl -n rails-staging rollout history deploy/rails-deployment
deployments "rails-deployment"
REVISION  CHANGE-CAUSE
1         kubectl create --namespace=rails-staging --filename=kube/deployments/rails_deploy.yaml --record=true

Applying an update

Let's see what happens if we push an update. First let's add some changes to our app. In my case I added Bootstrap. You can see the changes here.

To test this locally I can either run the app locally or use docker-compose to build and serve it.

~ $ docker-compose build && docker-compose up

Now I can navigate to http://localhost:3000 and check that everything looks good.

Let's push this new image to DockerHub:

~ $ bundle exec rake docker:push_image
...
Done pushing image a7f0b5d

Now we can set a new image for our rails and sidekiq deployments:

~ $ kubectl -n rails-staging set image deploy/rails-deployment rails=tzumby/rails-app-alpine:57b3e12

Since we only have 1 Replica, our rollout settings won't mean anything - Kube will simply create a new Pod and kill the old one. We can check the rollout history:

~ $ kubectl -n rails-staging rollout history deploy/rails-deployment
deployments "rails-deployment"
REVISION  CHANGE-CAUSE
1         kubectl create --namespace=rails-staging --filename=kube/deployments/rails_deploy.yaml --record=true
2         kubectl set image deploy/rails-deployment rails=tzumby/rails-app-alpine:85d97f1 --namespace=rails-stagin

Undo a deploy

Let's say whatever change we pushed is crashing and we want to revert back. This is pretty simple:

~ $ kubectl -n rails-staging rollout undo deployment/rails-deployment --to-revision=1

Scaling

If you read Part 1 of this series you'll see I mentioned scaling as one of the benefits of running your Rails app in Kubernetes. It's time to deliver on that promise and perform some load testing & scaling.

Let's take a quick look at the resources we created so far.

~ $ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
postgres-695fcd89f9-frz59             1/1       Running   0          5h
rails-deployment-59cd86c755-m66tq     1/1       Running   0          5h
redis-deployment-746c545869-tjbx6     1/1       Running   0          5h
sidekiq-deployment-7f5bcf6ccf-ncgxs   1/1       Running   0          5h

In my case I have one pod running for each service. These also come with their own Replica Sets:

~ $ kubectl get rs
AME                            DESIRED   CURRENT   READY     AGE
postgres-695fcd89f9             1         1         1         14m
rails-deployment-59cd86c755     1         1         1         6m
redis-deployment-746c545869     1         1         1         14m
sidekiq-deployment-7f5bcf6ccf   1         1         1         12m

Notice how the Desired, Current and Ready are all set to 1. This is because we used one replica when we defined those resources:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: rails
spec:
  replicas: 1 # this is the number of replicas

Apache Benchmark

We'll use ab to test our initial req/s throughput. This will be very basic - I just want to see an increase in the amount of req/s the app is handling (this is not by any means a performance test).

~ $ ab -n 500 -c 500 http://rails.local/
Requests per second:    172.24 [#/sec] (mean)

Now let's scale our Rails app to 4 replicas:

~ $ kubectl -n rails-staging scale deployment rails-deployment --replicas=4

We can check the Replica Sets and make sure everything worked:

~ $ kubectl -n rails-staging get rs
NAME                            DESIRED   CURRENT   READY     AGE
rails-deployment-59cd86c755     4         4         4         5h

We have 4 of the Pods ready it seems. Let's try to hammer the app again.

~ $ ab -n 500 -c 500 http://rails.local/
Requests per second:    256.35 [#/sec] (mean)

Looks like we're doing a lot better now at 256 req/s. Again, this doesn't really tell us anything about the performance of our app. I just wanted to use those numbers to verify that our deployment scaling worked.

We ran those scale commands without --record. If we did that we would even get those changes listed as Revisions:

~ $ kubectl -n rails-staging scale deployment rails-deployment --replicas=1 --record

~ $ kubectl -n rails-staging rollout history deployment/rails-deployment
REVISION  CHANGE-CAUSE
1         kubectl create --namespace=rails-staging --filename=kube/deployments/rails_deploy.yaml --record=true
2         kubectl set image deploy/rails-deployment rails=tzumby/rails-app-alpine:85d97f1 --namespace=rails-staging
3         kubectl scale deployment rails-deployment --namespace=rails-staging --replicas=1 --record=true

Conclusion

We looked at the new Deployments resource in Kubernetes and how we can use that to deploy new code or revert code changes. Deployments also allow us to scale our pods up and down. I didn't cover auto-scaling here but you can check that out on Kubernete's site.

What I described is a very manual process. This is great for learning how the ecosystem works but in a real production application you would most likely use a CI/CD tool that builds and pushes your containers on demand (via git-hooks or some manual trigger).