Contents

Applying Canary Release concepts in K8s natively

Introduction

Previously we talked about blue-gree deployment basic concept with kubernetes native, without any additional components. In order to continue deployments strategies in kubernetes, we are going to show an example of canary release deployment. Again, for learning purposes, for now we are not going to use any additional resources. To accomplish it, we built a Docker image from nodeJS app developed for this purpose. The app is very simple, it only returns version1.0 or version2.0 according to some criterios. Both versions will be packaged up in Docker images with distinct versions as Docker image tags. First of all, we are going to deploy two apps, the prod deploy with three replicas and canary deploy with one replica with the respectives labels. In addition, we are going to deploy a service to access these deploys. After that, I am going to show you how we can access these resources separately for test purposes or any other needs. If you would like to install a kubernetes cluster with kubeadm you can view my prior post here or for local testing you can install via very simple way with kink or minikube or any other of you choice. For a tool running local Kubernetes clusters using Docker container as “nodes” you can consider using kind. It is very useful and simple to run and setup. All resources used in this post you can download via my github.

What is canary release and Why should I use?

The main idea of the canary release is to rollout a new release to small subset of users for reducing all risks involved with releasing new software versions. Once you validate the new version you can rollout the changes to the rest of the infrastructure. According to what we saw in the previous post, it differs from blue green deployments because the blue gree deploys all the new release alongside the old version and then there will be a turn of keys at the entire instead of canary release that will incrementally release the new version and decrease the older version. Both of the strategies has Pros and Cons, so use them according your needs.

Canary release provides risk mitigation. If there is an issue with your code, would you want 100% of your users to encounter the issue or 1%? Using a canary release gives you that extra layer of protection when releasing your features. Canaries are very handy for avoiding problematic deployments, if one canary deployment fails, the rest of your users aren’t affected and you can simply ditch it and fix the root cause. Canary release usually frees the users to implement techniques such as Features Toggles, that are sometimes hard to maintain. However you can use both of them to improve your release techniques allowing you to test your features in production , convert your monolith to microservices, perform A/B testing, and so much more. You should use a percentage rollout to release the feature to a small percentage of users, measure its impact against your baseline metrics, and then decide whether or not to kill the canary, which will route the users who were seeing the new feature back to the old feature, or continue to slowly increment the percentage allocation until you reach 100%.

The Docker Image

First of all, I created a very simple app in NodeJS. The app only prints a json response with version 1.0 or version 2.0 according to the calling resource. As mentioned before, you can download the source in my github account and the respective Dockerfile. Notice that version is set to 1.0 we should cal “prod” and the version 2.0 should be the new version to be deployed and we should call canary.

We are going to use express to develop our app. Here is the server.ts file which is configured to listen on port 3333 and setup to use the router.

import express from 'express';
import router from './routes';

const app = express();

app.use(express.json());
app.use(router);

app.listen(3333, () => {
    console.log('🚀 Server started on port 3333');
  });

Another file is our router file that returns the json response which version we are using.

import { Router } from 'express';

const router = Router();


router.get('/', (request, response) => {
    return response.json("Version 1.0");
  });


export default router;

That’s all, this is our app. We are going to use yarn to build, run and install our dependencies. Please see the package.json file. Another important file is a Dockerfile where we are going to use it to build our image. Here is the file:

FROM node:14-slim

WORKDIR /usr/src/app

COPY package*.json ./

RUN yarn

COPY . .

EXPOSE 3333

CMD [ "yarn", "server" ]

Now, let’s build our first version image with the follow docker command:

docker build . -t filipemotta/node_app:1.0

Now you shoud change the router.ts version to build our version 2.0 and build our image again:

docker build . -t filipemotta/node_app:2.0

Now, when you list the images you should view the following, each of them representing the respective version.

docker images

REPOSITORY             TAG         IMAGE ID       CREATED          SIZE

filipemotta/node_app   2.0         1d178775caec   2 minutes ago    313MB

filipemotta/node_app   1.0         5f173b0cba0c   4 minutes ago   313MB

Before deploy our apps in kubernetes, we are going to test our images, then run the follow docker run commands:

docker run -d -p 8888:3333 filipemotta/node_app:1.0

docker run -d -p 9999:3333 filipemotta/node_app:2.0

As our app run in 3333 port and we’ve exposed it via docker, we have now to bind another port to test, so the fisrt one will run in 8888 port and the second will run in 9999 port. So, let’s test it:

curl -i localhost:8888

HTTP/1.1 200 OK

X-Powered-By: Express

Content-Type: application/json; charset=utf-8

...

"Version 1.0"
            


❯ curl -i localhost:9999

HTTP/1.1 200 OK

X-Powered-By: Express

Content-Type: application/json; charset=utf-8

...

"Version 2.0"

That’s ok, our images are working well.

Deploying in Kubernetes

We are going to create our first version in kubernetes. The first version we are going to call production and it has 3 replicas that use the filipemotta/node_app:1.0 image. To achieve it, I’m going to generate the first deploy file and change some options.

❯ kubectl create deploy nodeapp_production --replicas=3 --image=filipemotta/node_app:1.0  --dry-run=client -o yaml > deploy_nodeapp_prod.yaml

❯ cat deploy_nodeapp_prod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nodeapp    #change
    env: prod       #add
  name: nodeapp_production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nodeapp   #change
      env: prod      #add
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nodeapp   #change
        env: prod      #add
    spec:
      containers:
      - image: filipemotta/node_app:1.0
        name: node-app-1.0    #change
        readinessProbe:     #add
          httpGet:          #add
            path: /         #add
            port: 3333      #add
        resources: {}
status: {}


❯ kubectl apply -f deploy_nodeapp_prod.yaml
❯ kubectl rollout status  deployment nodeapp-production

deployment "nodeapp-production" successfully rolled out

Please review the #change and #add comments and change as needed. Note that the labels, all in kubernetes to select service, deploys, replicaSets and pods are working with labels. Labels are key/value pairs that are attached to objects, such as pods. Labels can be used to organize and to select subsets of objects. For now we’ve setup two main labels in this deployment, app and env labels which values nodeapp and prod respectively. We’ve configured a redinessProbe to know when a container is ready to start accepting traffic. A Pod is only considered ready when all of its containers are ready, so the readinessProb ensures it.

At this time, after applying the deploy, it’s so important to test our app step by step, then let’s test our app accessing some of the pods. You should have a pod IP. To do this, I runned a simple nginx pod.

kubectl exec nginx-test -- curl 10.32.0.7:3333

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

Dload  Upload   Total   Spent    Left  Speed

100    13  100    13    0     0   3250      0 --:--:-- --:--:-- --:--:--  3250

"Version 1.0"%

That’s ok, at least one of these pods is accessible via 3333 port so it works. After that, let’s expose our deployment using service. The selectors in the service will work with labels to identify and group kubernetes objects. In this case, our service will select first our prod deploy. After, will select too our canary deploy that will be deployed yet.

For now, we are going to generate our service file and delete and change some options.

❯ kubectl expose deployment nodeapp-production --port=80 --target-port=3333 --type=NodePort --dry-run=client -o yaml > svc.yaml

❯ cat svc.yaml          
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: nodeapp
    env: prod          #delete
  name: nodeapp-svc    #change
spec:
	type: NodePort 
  ports:
  - port: 80
    protocol: TCP
    targetPort: 3333
  selector:
    app: nodeapp
    env: prod        #delete
status:
  loadBalancer: {}

❯ kubectl apply -f svc.yaml

At this point, there are some options to comment. Note that I deleted the env label because our service will select two kinds of deployment ( prod and canary ) that have the label app: nodeapp and not app: nodeapp AND env: prod. As mentioned, the canary deploy will have too the app: nodeapp label. The NodePort is to be accessible outside our cluster, but you can use ClusterIP with no problems. Our service is accessible via port 80 and will redirect to 3333 target port. (pods)

Let’s describing our service. Note that the endpoints selected matches with prod deployment labels.

kubectl describe svc nodeapp-svc       
Name:              nodeapp-svc
Namespace:         default
Labels:            app=nodeapp
Annotations:       <none>
Selector:          app=nodeapp
Type:              NodePort
IP:                10.107.208.162
Port:              <unset>  80/TCP
TargetPort:        3333/TCP
Endpoints:         10.32.0.7:3333,10.46.0.4:3333,10.46.0.5:3333
Session Affinity:  None
Events:            <none>kubectl get ep                  
NAME          ENDPOINTS                                      AGE
kubernetes    192.168.5.11:6443                              24d
nodeapp-svc   10.32.0.7:3333,10.46.0.4:3333,10.46.0.5:3333   15s

As we are testing our app/deploy step by step, now let’s try to access our service outside our cluster via nodePort.

kubectl get svc

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE

nodeapp-svc   NodePort    10.101.73.202   <none>        80:32089/TCP   6m4scurl -i 192.168.5.11:32089        

HTTP/1.1 200 OK

X-Powered-By: Express

Content-Type: application/json; charset=utf-8
 ...

"Version 1.0"

The IP 192.168.11.5 is that the some of one host in the cluster accessing via NodePort (32089). Now we are accessing our app via kubernetes service, not only pod directly but loadbalancing between pods that belong our deploy selected via app label.

Applying the Canary Release

At this time we are going to start our canary deployment to redirect some users to use some pods running… The service deployed previously will now select the deployment that has the app label. That is, both prod and canary deploy will be selected by a service. Now, to make the file build process, let’s copy the prod deployment file created previously and make some changes and use it to deploy our canary deploy.

❯ cp -a deploy_nodeapp_prod.yaml deploy_nodeapp_canary.yaml

❯ cat deploy_nodeapp_canary.yaml                           
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nodeapp
    env: canary              #change
  name: nodeapp-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nodeapp
      env: canary           #change
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nodeapp
        env: canary        #change
    spec:
      containers:
      - image: filipemotta/node_app:2.0      #change
        name: nodepp-prod-c2
        readinessProbe:
          httpGet:
            path: /
            port: 3333
        resources: {}
      imagePullSecrets:
      - name: regcred
status: {}

❯ kubectl apply -f deploy_nodeapp_canary.yaml 

Notice that the label value env is different but both prod and canary share the same value for the app label. After apply the canary deploy let’s see our pods:

kubectl get pods --show-labels
NAME                                 READY   STATUS    RESTARTS   AGE   LABELS
nodeapp-canary-76b467c8-q6s5t        1/1     Running   1          22h   **app=nodeapp**,env=canary,pod-template-hash=76b467c8
nodeapp-production-795986f77-mphcn   1/1     Running   1          22h   **app=nodeapp**,env=prod,pod-template-hash=795986f77
nodeapp-production-795986f77-rv682   1/1     Running   1          22h   **app=nodeapp**,env=prod,pod-template-hash=795986f77
nodeapp-production-795986f77-tdbxc   1/1     Running   1          22h   **app=nodeapp**,env=prod,pod-template-hash=795986f77

Now let’s test our canary pod with the same way we tested our production pod…

kubectl exec nginx-test -- curl 10.40.0.1:3333

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

Dload  Upload   Total   Spent    Left  Speed

100    13  100    13    0     0    928      0 --:--:-- --:--:-- --:--:--  1000

"Version 2.0"

It’s ok, that’s all we would want: version 2.0 now is returned when we call the pod.

We don’t need to do any changes to the service, it already has a selector applied to the label app: nodeapp which spans across both deployments.

At this moment, for canary we have only 1 replica which means that controls the ratio/percentage of how much of our users are going to hit the canary release. In this case, we have a 3:1 ratio for prod and canary, so 75% of requests hitting the service will get the prod and 25% wil get the canary release. Then, let’s simulate:

for i in $(seq 15); do curl -i 192.168.5.11:32089; sleep 1;  done
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 1.0"

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 2.0"

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 2.0"

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 1.0"

HTTP/1.1 200 OK
X-Powered-By: Express
...
"Version 1.0"
...

Great !! Now once the version 2.0 is tested and accepted we can proceed by increasing the number of canary deployment replicas and decreasing the prod deployment replicas launch new pods while terminating the old ones. Another way is to simply change the prod image to use version 2.0 and delete the canary deploy after. Choose according to your needs.

kubectl get deployments.apps

NAME                 READY   UP-TO-DATE   AVAILABLE   AGE

nodeapp-canary       1/1     1            1           15h

nodeapp-production   3/3     3            3           16hkubectl scale deployment nodeapp-canary --replicas=2

deployment.apps/nodeapp-canary scaledkubectl get deployments.apps

NAME                 READY   UP-TO-DATE   AVAILABLE   AGE

nodeapp-canary       2/2     2            2           15h

nodeapp-production   3/3     3            3           16hkubectl scale deployment nodeapp-production --replicas=2

deployment.apps/nodeapp-production scaledkubectl get deployments.apps

NAME                 READY   UP-TO-DATE   AVAILABLE   AGE

nodeapp-canary       2/2     2            2           15h

nodeapp-production   2/2     2            2           16hfor i in $(seq 15); do curl 192.168.5.11:32089; sleep 1;  done

HTTP/1.1 200 OK

"Version 2.0"HTTP/1.1 200 OK

"Version 1.0"HTTP/1.1 200 OK

"Version 2.0"HTTP/1.1 200 OK

"Version 1.0"HTTP/1.1 200 OK

Using Ingress

You already have asked yourself if you would test the canary version separately… What can we do? There are some tools that can help us to accomplish it like Istio, but we can do it using ingress resources. There are some types of ingress resources in kubernetes. We are going to use nginx-ingress-controller. You can deploy an ingress resource in kubernetes using pods. Our goal with Ingress resources would be to split the traffic of the canary deployment to be accessible via some URL or some subfolder making it way easier to test the canary release separately from the prod deployment.

Our deploys already have both labels app and env each. The only thing we’ve to do is to delete our service because now we’ll have two services, one for each deployment and our ingress will select the respective deploy according to our needs. The new services we are going to deploy are described in the follow:

❯ cat svc-prod.yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: nodeapp
    env: prod
  name: nodeapp-svc-prod
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 3333
  selector:
    app: nodeapp
    env: prod

and

❯ cat svc-canary.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: nodeapp
    env: canary
  name: nodeapp-svc-canary
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 3333
  selector:
    app: nodeapp
    env: canary
status:
  loadBalancer: {}

Note that now both services select two labels, not only one as a previous service.

❯ kubectl describe service nodeapp-svc-prod    
Name:              nodeapp-svc-prod
Namespace:         default
Labels:            app=nodeapp
                   env=prod
Annotations:       <none>
Selector:          app=nodeapp,env=prod
Type:              ClusterIP
IP:                10.96.14.241
Port:              <unset>  80/TCP
TargetPort:        3333/TCP
Endpoints:         10.244.0.3:3333,10.244.0.4:3333,10.244.0.5:3333
Session Affinity:  None
Events:            <none>


❯ kubectl describe service nodeapp-svc-canary 
Name:              nodeapp-svc-canary
Namespace:         default
Labels:            app=nodeapp
                   env=canary
Annotations:       <none>
Selector:          app=nodeapp,env=canary
Type:              ClusterIP
IP:                10.96.174.239
Port:              <unset>  80/TCP
TargetPort:        3333/TCP
Endpoints:         10.244.0.2:3333
Session Affinity:  None
Events:            <none>

The prod service has three endpoints and the canary service has one. Similar to the other service but now accessible separately. Once the service is published, we can start creating our Ingress resources. Now let’s create our ingress to route, test and access our canary app separately from the prod.

❯ cat ingress.yaml   
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-prod-canary
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  defaultBackend:
    service:
      name: nodeapp-svc-prod
      port:
        number: 80
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nodeapp-svc-prod
            port:
              number: 80
      - path: /canary
        pathType: Prefix
        backend:
          service:
            name: nodeapp-svc-canary
            port:
              number: 80

Explaining some points in the ingress file. We’ll access it through foo.bar.com URL. If we pass any URL that does not match with / or /canary it’ll be redirected to the default backend, if we pass through /canay it’ll access the canary service so the user can test canary release with a different URL from the prod service.

Ingress

❯ kubectl describe ingress ingress-prod-canary 
Name:             ingress-prod-canary
Namespace:        default
Address:          localhost
Default backend:  nodeapp-svc:80 (10.244.0.10:3333,10.244.0.8:3333,10.244.0.9:3333)
Rules:
  Host         Path  Backends
  ----         ----  --------
  foo.bar.com  
               /         nodeapp-svc:80 (10.244.0.10:3333,10.244.0.8:3333,10.244.0.9:3333)
               /canary   nodeapp-svc-canary:80 (10.244.0.11:3333)
Annotations:   nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    22m (x2 over 23m)  nginx-ingress-controller  Scheduled for sync

Finally, let’s simulate to access your canary deployment in isolation.

❯ curl -H "Host:foo.bar.com" http://localhost        
"Version 1.0"%
                                                                                                                                            
❯ curl -H "Host:foo.bar.com" http://localhost/canary 
"Version 2.0"%

In this case, we’ve two separated different URL to access resources/app for test purposes. For us to be able to access the canary release on a separate path, we need to add some the ingress routes that point to the canary service. In the same way with prod, we have set the default backend and root /. The ingress resource will check the host header and judge which service it should send the traffic to.

As soon the canary app be tested we can promoting our canary version to production using the same way discussed earlier: increasing the number of the deploy replicas of the canary and decreasing prod or change the image of the prod deploy and delete the resources of canary. And then, you can change your ingress to access to the new app.

Conclusion

This post had a learning proposal, so, if you intend to deploy a canary release deployment in production you should take account in using tools like Istio, analyzing other situations and possible side effects.

I hope this post was useful.