Skip to content

SLO Validation with GitOps

Scenario: Validate SLOs and promote a new version of a K8s app

Problem: You have a new version of a K8s app. You want to verify that it satisfies latency and error rate SLOs, and promote it to production as the stable version of your app in a GitOps-y manner.

Solution: In this tutorial, you will dark launch the new version of your K8s app along with an Iter8 experiment. Iter8 will validate that the new satisfies latency and error-based objectives (SLOs) using built-in metrics and promote the new version by raising a pull-request in a GitHub repo.

SLO Validation GitOps

Setup K8s cluster and local environment
  1. If you completed the Iter8 getting-started tutorial (highly recommended), you may skip the remaining steps of setup.
  2. Setup K8s cluster
  3. Install Iter8 in K8s cluster
  4. Get Helm 3.4+.
  5. Get iter8ctl
  6. Fork the Iter8 GitHub repo. Clone your fork, and set the ITER8 environment variable as follows.
    export USERNAME=<your GitHub username>
    
    git clone git@github.com:$USERNAME/iter8.git
    cd iter8
    export ITER8=$(pwd)
    

1. Create stable version

Create version 1.0 of the hello world app as follows.

# USERNAME is exported as part of setup steps.
kubectl apply -f https://raw.githubusercontent.com/$USERNAME/iter8/master/samples/deployments/app/deploy.yaml
kubectl apply -f https://raw.githubusercontent.com/$USERNAME/iter8/master/samples/deployments/app/service.yaml

2. Create candidate version

Create version 2.0 of the hello world app in the staging environment as follows. For the purpose of this tutorial, the production environment is the default namespace, and the staging environment is the staging namespace.

kubectl create ns staging
# create version 2.0 of hello world app in the staging namespace
kubectl set image --local -f https://raw.githubusercontent.com/$USERNAME/iter8/master/samples/deployments/app/deploy.yaml hello='gcr.io/google-samples/hello-app:2.0' -o yaml | kubectl apply -n staging -f -
kubectl apply -f https://raw.githubusercontent.com/$USERNAME/iter8/master/samples/deployments/app/service.yaml -n staging

Adapt these instructions to verify that stable and candidate versions of your app are running.

3. Enable Iter8 GitOps

3.1) Create a personal access token on GitHub. In Step 8 of this process, grant repo and read:org permissions to this token. This will ensure that the token can be used by Iter8 to update your app manifest in GitHub.

3.2) Create K8s secret

# replace $GHTOKEN with GitHub token created above
kubectl create secret generic -n staging ghtoken --from-literal=token=$GHTOKEN

3.3) Provide RBAC permission

kubectl create role -n staging ghtoken-reader \
  --verb=get \
  --resource=secrets \
  --resource-name=ghtoken
kubectl create rolebinding -n staging ghtoken-reader-binding \
  --role=ghtoken-reader \
  --serviceaccount=iter8-system:iter8-handlers

4. Create Iter8 experiment

Deploy an Iter8 experiment for SLO validation and GitOps-y promotion of the app as follows.

helm upgrade -n staging my-exp $ITER8/samples/slo-gitops \
  --set URL='http://hello.staging.svc.cluster.local:8080' \
  --set limitMeanLatency=50.0 \
  --set limitErrorRate=0.0 \
  --set limit95thPercentileLatency=100.0 \
  --set username=$USERNAME \
  --set newImage='gcr.io/google-samples/hello-app:2.0' \
  --install

The above command creates an Iter8 experiment that generates requests, collects latency and error rate metrics for the candidate version of the app, and verifies that the candidate satisfies mean latency (50 msec), error rate (0.0), 95th percentile tail latency SLO (100 msec) SLOs.

In the above command, the USERNAME environment variable was defined during setup. After the Iter8 experiment validates SLOs for the candidate, it uses the GitHub token (also provided during setup) to promote the candidate to production using a GitHub pull-request.

5. View and observe experiment

View the Iter8 experiment as described here. Observe the experiment by following these steps. Ensure correct namespace (staging) is used.

6. Review Iter8's PR

Once the experiment completes, you can visit your fork at https://github.com/$USERNAME/iter8/pulls to review the pull-request created by Iter8.

7. Cleanup

# remove Iter8 experiment and candidate version of the app
kubectl delete ns staging
# remove stable version of the app
kubectl delete deploy/hello
kubectl delete svc/hello

Next Steps

Use with your git repo/app
  1. The Helm chart used in this tutorial is located at $ITER8/samples/slo-gitops. Within this folder, the file templates/experiment.yaml contains the Iter8 experiment template. The following lines in this template are responsible for cloning the git repo, modifying it locally, and pushing the changes.

    # clone repo using token
    git clone https://$USERNAME:$TOKEN@github.com/$USERNAME/iter8.git
    
    # commit changes locally and push to a branch
    cd iter8
    git checkout -b iter8-gitops
    yq e -i '.spec.template.spec.containers[0].image = "{{ required ".Values.newImage is required!" .Values.newImage }}"' samples/deployments/app/deploy.yaml 
    git commit --allow-empty -a -m "promote candidate version to production"
    git push -f origin iter8-gitops
    
    Change these lines in the template so that the correct repo is cloned and your app is modified correctly.

  2. Ensure that correct GitHub access token is supplied to Iter8.

  3. You may need to supplement or replace the Helm values username and newImage used in this tutorial with other Helm values as needed by your repo/app.

Back to top