Introduction
Previously we talked about blue-gree deployment basic concept with kubernetes native, without any additional components. In order to continue deployments strategies in kubernetes, we are going to show an example of canary release deployment. Again, for learning purposes, for now we are not going to use any additional resources. To accomplish it, we built a Docker image from nodeJS app developed for this purpose. The app is very simple, it only returns version1.0 or version2.0 according to some criterios. Both versions will be packaged up in Docker images with distinct versions as Docker image tags. First of all, we are going to deploy two apps, the prod deploy with three replicas and canary deploy with one replica with the respectives labels. In addition, we are going to deploy a service to access these deploys. After that, I am going to show you how we can access these resources separately for test purposes or any other needs. If you would like to install a kubernetes cluster with kubeadm you can view my prior post here or for local testing you can install via very simple way with kink or minikube or any other of you choice. For a tool running local Kubernetes clusters using Docker container as “nodes” you can consider using kind. It is very useful and simple to run and setup. All resources used in this post you can download via my github.
What is canary release and Why should I use?
The main idea of the canary release is to rollout a new release to small subset of users for reducing all risks involved with releasing new software versions. Once you validate the new version you can rollout the changes to the rest of the infrastructure. According to what we saw in the previous post, it differs from blue green deployments because the blue gree deploys all the new release alongside the old version and then there will be a turn of keys at the entire instead of canary release that will incrementally release the new version and decrease the older version. Both of the strategies has Pros and Cons, so use them according your needs.
Canary release provides risk mitigation. If there is an issue with your code, would you want 100% of your users to encounter the issue or 1%? Using a canary release gives you that extra layer of protection when releasing your features. Canaries are very handy for avoiding problematic deployments, if one canary deployment fails, the rest of your users aren’t affected and you can simply ditch it and fix the root cause. Canary release usually frees the users to implement techniques such as Features Toggles, that are sometimes hard to maintain. However you can use both of them to improve your release techniques allowing you to test your features in production , convert your monolith to microservices, perform A/B testing, and so much more. You should use a percentage rollout to release the feature to a small percentage of users, measure its impact against your baseline metrics, and then decide whether or not to kill the canary, which will route the users who were seeing the new feature back to the old feature, or continue to slowly increment the percentage allocation until you reach 100%.
The Docker Image
First of all, I created a very simple app in NodeJS. The app only prints a json response with version 1.0 or version 2.0 according to the calling resource. As mentioned before, you can download the source in my github account and the respective Dockerfile. Notice that version is set to 1.0 we should cal “prod” and the version 2.0 should be the new version to be deployed and we should call canary.
We are going to use express to develop our app. Here is the server.ts file which is configured to listen on port 3333 and setup to use the router.
import express from 'express';
import router from './routes';
const app = express();
app.use(express.json());
app.use(router);
app.listen(3333, () => {
console.log('🚀 Server started on port 3333');
});
Another file is our router file that returns the json response which version we are using.
import { Router } from 'express';
const router = Router();
router.get('/', (request, response) => {
return response.json("Version 1.0");
});
export default router;
That’s all, this is our app. We are going to use yarn to build, run and install our dependencies. Please see the package.json file. Another important file is a Dockerfile where we are going to use it to build our image. Here is the file:
FROM node:14-slim
WORKDIR /usr/src/app
COPY package*.json ./
RUN yarn
COPY . .
EXPOSE 3333
CMD [ "yarn", "server" ]
Now, let’s build our first version image with the follow docker command:
❯ docker build . -t filipemotta/node_app:1.0
Now you shoud change the router.ts version to build our version 2.0 and build our image again:
❯ docker build . -t filipemotta/node_app:2.0
Now, when you list the images you should view the following, each of them representing the respective version.
❯ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
filipemotta/node_app 2.0 1d178775caec 2 minutes ago 313MB
filipemotta/node_app 1.0 5f173b0cba0c 4 minutes ago 313MB
Before deploy our apps in kubernetes, we are going to test our images, then run the follow docker run commands:
docker run -d -p 8888:3333 filipemotta/node_app:1.0
docker run -d -p 9999:3333 filipemotta/node_app:2.0
As our app run in 3333 port and we’ve exposed it via docker, we have now to bind another port to test, so the fisrt one will run in 8888 port and the second will run in 9999 port. So, let’s test it:
❯ curl -i localhost:8888
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 1.0"
❯ curl -i localhost:9999
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 2.0"
That’s ok, our images are working well.
Deploying in Kubernetes
We are going to create our first version in kubernetes. The first version we are going to call production and it has 3 replicas that use the filipemotta/node_app:1.0 image. To achieve it, I’m going to generate the first deploy file and change some options.
❯ kubectl create deploy nodeapp_production --replicas=3 --image=filipemotta/node_app:1.0 --dry-run=client -o yaml > deploy_nodeapp_prod.yaml
❯ cat deploy_nodeapp_prod.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nodeapp #change
env: prod #add
name: nodeapp_production
spec:
replicas: 3
selector:
matchLabels:
app: nodeapp #change
env: prod #add
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: nodeapp #change
env: prod #add
spec:
containers:
- image: filipemotta/node_app:1.0
name: node-app-1.0 #change
readinessProbe: #add
httpGet: #add
path: / #add
port: 3333 #add
resources: {}
status: {}
❯ kubectl apply -f deploy_nodeapp_prod.yaml
❯ kubectl rollout status deployment nodeapp-production
deployment "nodeapp-production" successfully rolled out
Please review the #change and #add comments and change as needed. Note that the labels, all in kubernetes to select service, deploys, replicaSets and pods are working with labels. Labels are key/value pairs that are attached to objects, such as pods. Labels can be used to organize and to select subsets of objects. For now we’ve setup two main labels in this deployment, app and env labels which values nodeapp and prod respectively. We’ve configured a redinessProbe to know when a container is ready to start accepting traffic. A Pod is only considered ready when all of its containers are ready, so the readinessProb ensures it.
At this time, after applying the deploy, it’s so important to test our app step by step, then let’s test our app accessing some of the pods. You should have a pod IP. To do this, I runned a simple nginx pod.
❯ kubectl exec nginx-test -- curl 10.32.0.7:3333
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13 100 13 0 0 3250 0 --:--:-- --:--:-- --:--:-- 3250
"Version 1.0"%
That’s ok, at least one of these pods is accessible via 3333 port so it works. After that, let’s expose our deployment using service. The selectors in the service will work with labels to identify and group kubernetes objects. In this case, our service will select first our prod deploy. After, will select too our canary deploy that will be deployed yet.
For now, we are going to generate our service file and delete and change some options.
❯ kubectl expose deployment nodeapp-production --port=80 --target-port=3333 --type=NodePort --dry-run=client -o yaml > svc.yaml
❯ cat svc.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: nodeapp
env: prod #delete
name: nodeapp-svc #change
spec:
type: NodePort
ports:
- port: 80
protocol: TCP
targetPort: 3333
selector:
app: nodeapp
env: prod #delete
status:
loadBalancer: {}
❯ kubectl apply -f svc.yaml
At this point, there are some options to comment. Note that I deleted the env label because our service will select two kinds of deployment ( prod and canary ) that have the label app: nodeapp and not app: nodeapp AND env: prod. As mentioned, the canary deploy will have too the app: nodeapp label. The NodePort is to be accessible outside our cluster, but you can use ClusterIP with no problems. Our service is accessible via port 80 and will redirect to 3333 target port. (pods)
Let’s describing our service. Note that the endpoints selected matches with prod deployment labels.
❯ kubectl describe svc nodeapp-svc
Name: nodeapp-svc
Namespace: default
Labels: app=nodeapp
Annotations: <none>
Selector: app=nodeapp
Type: NodePort
IP: 10.107.208.162
Port: <unset> 80/TCP
TargetPort: 3333/TCP
Endpoints: 10.32.0.7:3333,10.46.0.4:3333,10.46.0.5:3333
Session Affinity: None
Events: <none>
❯ kubectl get ep
NAME ENDPOINTS AGE
kubernetes 192.168.5.11:6443 24d
nodeapp-svc 10.32.0.7:3333,10.46.0.4:3333,10.46.0.5:3333 15s
As we are testing our app/deploy step by step, now let’s try to access our service outside our cluster via nodePort.
❯ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nodeapp-svc NodePort 10.101.73.202 <none> 80:32089/TCP 6m4s
❯ curl -i 192.168.5.11:32089
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 1.0"
The IP 192.168.11.5 is that the some of one host in the cluster accessing via NodePort (32089). Now we are accessing our app via kubernetes service, not only pod directly but loadbalancing between pods that belong our deploy selected via app label.
Applying the Canary Release
At this time we are going to start our canary deployment to redirect some users to use some pods running… The service deployed previously will now select the deployment that has the app label. That is, both prod and canary deploy will be selected by a service. Now, to make the file build process, let’s copy the prod deployment file created previously and make some changes and use it to deploy our canary deploy.
❯ cp -a deploy_nodeapp_prod.yaml deploy_nodeapp_canary.yaml
❯ cat deploy_nodeapp_canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nodeapp
env: canary #change
name: nodeapp-canary
spec:
replicas: 1
selector:
matchLabels:
app: nodeapp
env: canary #change
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: nodeapp
env: canary #change
spec:
containers:
- image: filipemotta/node_app:2.0 #change
name: nodepp-prod-c2
readinessProbe:
httpGet:
path: /
port: 3333
resources: {}
imagePullSecrets:
- name: regcred
status: {}
❯ kubectl apply -f deploy_nodeapp_canary.yaml
Notice that the label value env is different but both prod and canary share the same value for the app label. After apply the canary deploy let’s see our pods:
❯ kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nodeapp-canary-76b467c8-q6s5t 1/1 Running 1 22h **app=nodeapp**,env=canary,pod-template-hash=76b467c8
nodeapp-production-795986f77-mphcn 1/1 Running 1 22h **app=nodeapp**,env=prod,pod-template-hash=795986f77
nodeapp-production-795986f77-rv682 1/1 Running 1 22h **app=nodeapp**,env=prod,pod-template-hash=795986f77
nodeapp-production-795986f77-tdbxc 1/1 Running 1 22h **app=nodeapp**,env=prod,pod-template-hash=795986f77
Now let’s test our canary pod with the same way we tested our production pod…
❯ kubectl exec nginx-test -- curl 10.40.0.1:3333
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13 100 13 0 0 928 0 --:--:-- --:--:-- --:--:-- 1000
"Version 2.0"
It’s ok, that’s all we would want: version 2.0 now is returned when we call the pod.
We don’t need to do any changes to the service, it already has a selector applied to the label app: nodeapp which spans across both deployments.
At this moment, for canary we have only 1 replica which means that controls the ratio/percentage of how much of our users are going to hit the canary release. In this case, we have a 3:1 ratio for prod and canary, so 75% of requests hitting the service will get the prod and 25% wil get the canary release. Then, let’s simulate:
for i in $(seq 15); do curl -i 192.168.5.11:32089; sleep 1; done
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 1.0"
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 2.0"
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 2.0"
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
...
"Version 1.0"
HTTP/1.1 200 OK
X-Powered-By: Express
...
"Version 1.0"
...
Great !! Now once the version 2.0 is tested and accepted we can proceed by increasing the number of canary deployment replicas and decreasing the prod deployment replicas launch new pods while terminating the old ones. Another way is to simply change the prod image to use version 2.0 and delete the canary deploy after. Choose according to your needs.
❯ kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
nodeapp-canary 1/1 1 1 15h
nodeapp-production 3/3 3 3 16h
❯ kubectl scale deployment nodeapp-canary --replicas=2
deployment.apps/nodeapp-canary scaled
❯ kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
nodeapp-canary 2/2 2 2 15h
nodeapp-production 3/3 3 3 16h
❯ kubectl scale deployment nodeapp-production --replicas=2
deployment.apps/nodeapp-production scaled
❯ kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
nodeapp-canary 2/2 2 2 15h
nodeapp-production 2/2 2 2 16h
❯ for i in $(seq 15); do curl 192.168.5.11:32089; sleep 1; done
HTTP/1.1 200 OK
"Version 2.0"HTTP/1.1 200 OK
"Version 1.0"HTTP/1.1 200 OK
"Version 2.0"HTTP/1.1 200 OK
"Version 1.0"HTTP/1.1 200 OK
Using Ingress
You already have asked yourself if you would test the canary version separately… What can we do? There are some tools that can help us to accomplish it like Istio, but we can do it using ingress resources. There are some types of ingress resources in kubernetes. We are going to use nginx-ingress-controller. You can deploy an ingress resource in kubernetes using pods. Our goal with Ingress resources would be to split the traffic of the canary deployment to be accessible via some URL or some subfolder making it way easier to test the canary release separately from the prod deployment.
Our deploys already have both labels app and env each. The only thing we’ve to do is to delete our service because now we’ll have two services, one for each deployment and our ingress will select the respective deploy according to our needs. The new services we are going to deploy are described in the follow:
❯ cat svc-prod.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: nodeapp
env: prod
name: nodeapp-svc-prod
spec:
ports:
- port: 80
protocol: TCP
targetPort: 3333
selector:
app: nodeapp
env: prod
and
❯ cat svc-canary.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: nodeapp
env: canary
name: nodeapp-svc-canary
spec:
ports:
- port: 80
protocol: TCP
targetPort: 3333
selector:
app: nodeapp
env: canary
status:
loadBalancer: {}
Note that now both services select two labels, not only one as a previous service.
❯ kubectl describe service nodeapp-svc-prod
Name: nodeapp-svc-prod
Namespace: default
Labels: app=nodeapp
env=prod
Annotations: <none>
Selector: app=nodeapp,env=prod
Type: ClusterIP
IP: 10.96.14.241
Port: <unset> 80/TCP
TargetPort: 3333/TCP
Endpoints: 10.244.0.3:3333,10.244.0.4:3333,10.244.0.5:3333
Session Affinity: None
Events: <none>
❯ kubectl describe service nodeapp-svc-canary
Name: nodeapp-svc-canary
Namespace: default
Labels: app=nodeapp
env=canary
Annotations: <none>
Selector: app=nodeapp,env=canary
Type: ClusterIP
IP: 10.96.174.239
Port: <unset> 80/TCP
TargetPort: 3333/TCP
Endpoints: 10.244.0.2:3333
Session Affinity: None
Events: <none>
The prod service has three endpoints and the canary service has one. Similar to the other service but now accessible separately. Once the service is published, we can start creating our Ingress resources. Now let’s create our ingress to route, test and access our canary app separately from the prod.
❯ cat ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-prod-canary
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
defaultBackend:
service:
name: nodeapp-svc-prod
port:
number: 80
rules:
- host: foo.bar.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nodeapp-svc-prod
port:
number: 80
- path: /canary
pathType: Prefix
backend:
service:
name: nodeapp-svc-canary
port:
number: 80
Explaining some points in the ingress file. We’ll access it through foo.bar.com URL. If we pass any URL that does not match with / or /canary it’ll be redirected to the default backend, if we pass through /canay it’ll access the canary service so the user can test canary release with a different URL from the prod service.
Ingress
❯ kubectl describe ingress ingress-prod-canary
Name: ingress-prod-canary
Namespace: default
Address: localhost
Default backend: nodeapp-svc:80 (10.244.0.10:3333,10.244.0.8:3333,10.244.0.9:3333)
Rules:
Host Path Backends
---- ---- --------
foo.bar.com
/ nodeapp-svc:80 (10.244.0.10:3333,10.244.0.8:3333,10.244.0.9:3333)
/canary nodeapp-svc-canary:80 (10.244.0.11:3333)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 22m (x2 over 23m) nginx-ingress-controller Scheduled for sync
Finally, let’s simulate to access your canary deployment in isolation.
❯ curl -H "Host:foo.bar.com" http://localhost
"Version 1.0"%
❯ curl -H "Host:foo.bar.com" http://localhost/canary
"Version 2.0"%
In this case, we’ve two separated different URL to access resources/app for test purposes. For us to be able to access the canary release on a separate path, we need to add some the ingress routes that point to the canary service. In the same way with prod, we have set the default backend and root /. The ingress resource will check the host header and judge which service it should send the traffic to.
As soon the canary app be tested we can promoting our canary version to production using the same way discussed earlier: increasing the number of the deploy replicas of the canary and decreasing prod or change the image of the prod deploy and delete the resources of canary. And then, you can change your ingress to access to the new app.
Conclusion
This post had a learning proposal, so, if you intend to deploy a canary release deployment in production you should take account in using tools like Istio, analyzing other situations and possible side effects.
I hope this post was useful.