CICD for Kubernetes

, , , ,

Designing CICD pipeline is not easy, you have to find the balance between
automating things and making it maintainable. Maintainable in a sense that the other DevOps engineer can understand what is happening in the pipeline, so that they fix it if something goes wrong or improve it. In this entry, I will discuss my take on CICD for Kubernetes.

PROs of using this Setup

Later we are going to dive deeper on how the following feats were achieved.

  • Using GitOps, to learn more about GitOps visit opengitops.dev.
  • All resources used in the CICD are K8s native.
  • Apps are easy to deploy and troubleshoot.
  • Changes to configmaps are picked up and applied to the apps automatically, no need to restart pods.

Tools used in this CICD pipeline

Argo Workflows

Argo workflow is what I’ve used for the workflow engine/orchestrator. This will handle things like running of static analysis, creation of docker images, and replacing and committing the new image to the QA app config repo, etc. The main reasons why I’ve used Argo Workflow is because it’s k8s native, it’s opensource, and it works seemlessly with ArgoCD.

Kaniko

Kaniko is the tool that lets us build the container images inside kubernetes.

Argo CD

Argo CD is our GitOps controller. It will watch over the app config repo for changes. And if there are, it will automatically reconcile the desired state to the cluster state.

Kubevela

We are using kubevela as OAM to make app deployment easier. To learn more about kubevela, you can read here.

Reloader

Reloader app ensures that our workloads use the most recent configmap. If there are changes on the configmap of the apps it’s managing, it will restart all instances of that app to update the config in the pods.

CICD in Action

In this scenario, we are assuming that the development and unit tests are happening on the developers local machine.

Once the Devs pushed their changes into the app code repo, they will have to run a pipeline that will execute static analysis, deploy the changes, integration and regression tests on the QA environment.

Note: We can use an events tool like Argo Events to detect if there are commits to the app code repo and run the CI pipeline automatically.

Pre-requisites

Before we can run the CI pipeline and CD, we need to setup the following.
So right now, we will be wearing our DevOps/Operator hat while setting up the cluster.

Setting Up the Cluster

1
2
3
4
5
6
7
8
9
10
11
12
13
14
### Clone the dev-k8s repo
git clone git@github.com:lordpangan/dev-k8s.git

### Setup the cluster
kind create cluster --name cluster1 --config simple-cluster.yaml

### Install ingress and argocd. Wait for argocd to finish deployment.
kubectl kustomize cluster-init/overlay/setup-ingress-argocd | kubectl apply -f -
kubectl wait --namespace argocd --for=condition=ready pod --selector=app.kubernetes.io/name=argocd-server --timeout=90s

### Install argo-workflows kubevela, and reloader apps
kubectl apply -f cluster-init/cluster-apps/argo-workflow.yaml
kubectl apply -f cluster-init/cluster-apps/kubevela.yaml
kubectl apply -f cluster-init/cluster-apps/reloader.yaml

Access https://localhost on your browser to make sure all cluster app are successfully deployed.

Note: Initially an unsecure warning will pop-up, select proceed and accept risk.

To get the initial password for the admin user, run this:

1
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Login and verify that all cluster apps (i.e. argo-workflow, kubevela and reloader) are deployed successfully.

Fork it!

Now we will switch to our Developer Hat.
We will be using these sample repos for our pipeline, so we will need to fork these.
sample-springboot - App code repo of our sample springboot app.
sample-kubevela-argocd - App config repo for our QA and PROD env.

Clone the k8s-cicd repo.

Edit the qa-sample-app.yaml and prod-sample-app.yaml, and change the repoURL to your sample-kubevela-argocd forked repo.

1
2
3
4
5
6
# On k8s-cicd/apps/qa-sample-app.yaml and k8s-cicd/apps/prod-sample-app.yaml
...
  source:
    path: prod
    repoURL: https://github.com/lordpangan/sample-kubevela-argocd.git # <to-your-sample-kubevela-argocd-forked-repo>
    targetRevision: HEAD

Apply the Application manifests.

1
2
3
4
cd k8s-cicd
kubectl apply -f apps/cluster-config.yaml
kubectl apply -f apps/qa-sample-app.yaml
kubectl apply -f apps/prod-sample-app.yaml

Edit the ci.yaml. Edit the app_repo value on get-git-hash and build-container-image step with your sample-springboot forked repo. Then edit the app_config_repo on deploy-to-qa step with your sample-kubevela-argocd forked repo.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# On k8s-cicd/workflows/ci.yaml
...
      - name: get-git-hash
        template: git-hash
        arguments:
          parameters:
          - name: app_repo
            value: lordpangan/sample-springboot.git # <to-your-sample-springboot-forked-repo>
          - name: app_name
            value: sample-springboot
        dependencies:
          - static-tests
      - name: build-container-image
        template: container-image
        arguments:
          parameters:
          - name: app_repo
            value: git://github.com/lordpangan/sample-springboot.git # <to-your-sample-springboot-forked-repo>
          - name: container_image
            value: tykola/hello-spring
          - name: container_tag
            value: "{{tasks.get-git-hash.outputs.parameters.hash-param}}"
        dependencies:
        - get-git-hash
      - name: deploy-to-qa
        template: git-commit
        arguments:
          parameters:
          - name: app_config_repo
            value: lordpangan/sample-kubevela-argocd.git # <to-your-sample-kubevela-argocd-forked-repo>
          - name: app_name
            value: sample-kubevela-argocd
          - name: env
            value: qa
          - name: container_name
            value: hello-spring
          - name: dockerhub_user
            value: tykola
          - name: new_container_tag
            value: "{{tasks.get-git-hash.outputs.parameters.hash-param}}"
        dependencies:
        - build-container-image
...

Add required Secrets

Add the regcred(docker secret) and git-token to argo namespace.

1
2
3
4
5
6
7
8
9
10
11
# Create 'git-token', secret for commiting to app config repo
kubectl -n argo create secret generic git-token --from-literal=git-token=$GIT_TOKEN

# Create 'regcred', secret for pushing to docker repo
kubectl --namespace argo \
    create secret \
    docker-registry regcred \
    --docker-username=$DOCKER_USER \
    --docker-password=$DOCKER_USER \
    --docker-email=$DOCKER_EMAIL

Run the CI Workflow

First, open http://localhost/qa on your browser and inspect the web page.

Now, for example we want to edit the app, in this case we will modify the background. Edit the background-color of the sample-app on your sample-springboot forked repo.

1
2
3
4
5
6
7
# On sample-springboot/src/main/java/hello/HelloWorldCtrl.java
...
    public String index() {
        String prefix = System.getenv().getOrDefault("ENV_PREFIX", "Blank");
        return "<h1 style='background-color:<edit-to-color-of-your-choice>;width: 100%;height: 100%;text-align: center;'> 'Greetings from "+prefix+"' </h1>";
    }
...

Commit the changes and push to git.

Now run the CI workflow.

1
argo -n argo submit workflows/ci.yaml

Let’s visit the Argo Workflows UI to see what’s happening in the workflow.
Open your browser to http://localhost/argo and click on the running build workflow. This CI workflow will do the following:

  1. Check out app code
  2. Perform a mock static analysis
  3. Building of new image and pushing to dockerhub
  4. Replacing and committing it to QA app config repo for deployment
  5. Run a mock integration test on QA
  6. Run a mock regression/acceptance test on QA
  7. Mock sending of results to Dev

Note: Once the old is replaced by the new image and commited to git, ArgoCD will make sure that it will be deployed to the K8s cluster.

Wait a couple of minutes and check http://localhost/qa. Verify that your modifications were applied.

Test the Reloader

Edit the value of env.prefix on sample-kubevela-argocd/qa/sample-app.properties to any value of your choice.

1
2
3
# On sample-kubevela-argocd/qa/sample-app.properties

env.prefix=value-of-your-choice

Commit and push it to your sample-kubevela-argocd repo. Wait a couple of minutes for your modifications to be picked up and applied by ArgoCD to the configmaps. The Reloader app then will detect that the sample-app pod is not using the most recent sample-app-properties and terminate the pod. A new pod with the correct configmap will be spawned. Visit http://localhost/qa and verify that your changes were applied.

An Imgur Image

Run the CD(Continuos Delivery) workflow

If all tests passed, the Devs can decide to deploy the new release to Prod.
To do that, they will have to run the CD workflow.

1
argo -n argo submit workflows/cd.yaml

Visit http://localhost/argo to check on the running deploy workflow. This will fetch the image deployed on QA, replace it, and commit it to the PROD app config repo, and ArgoCD will do the rest. And that’s it, our changes have reached Prod. WOOT!

There are many things that can be improved on this Kubernetes CICD pipeline, but I think it’s a good starting point for those who want to setup their CICD pipeline with GitOps and all K8s native component. I really enjoyed working with ArgoCD + Kubevela + Reloader, the auto reconcilation is very convinient. And if you add that to how simplified Kubevela made application deployment is, if I put myself on the shoes of the Developer I think, Yes, I’ll be able to focus more on coding my app!

And that’s all I have for you guys, thank you and ciao. :)