A/B Testing with the Iter8 SDK¶

This tutorial describes how to do A/B testing of a backend component using the Iter8 SDK.

Install the Iter8 controller¶

helm install --repo https://iter8-tools.github.io/iter8 --version 0.1.12 iter8 controller

helm install --repo https://iter8-tools.github.io/iter8 --version 0.1.12 iter8 controller \
--set clusterScoped=true

kubectl apply -k 'https://github.com/iter8-tools/iter8.git/kustomize/controller/namespaceScoped?ref=v0.17.1'

kubectl apply -k 'https://github.com/iter8-tools/iter8.git/kustomize/controller/clusterScoped?ref=v0.17.1'

Deploy the sample application¶

A sample application using the Iter8 SDK is provided. Deploy both the frontend and backend components of this application as described in each tab:

kubectl create deployment frontend --image=iter8/abn-sample-frontend-node:0.15.0
kubectl expose deployment frontend --name=frontend --port=8090

kubectl create deployment frontend --image=iter8/abn-sample-frontend-go:0.15.0
kubectl expose deployment frontend --name=frontend --port=8090

kubectl create deployment backend --image=iter8/abn-sample-backend:0.13-v1
kubectl label deployment backend iter8.tools/watch="true"

kubectl expose deployment backend --name=backend --port=8091

Describe the application¶

In order to support Lookup(), Iter8 needs to know what the application component versions look like. A ConfigMap is used to describe the make up of possible versions:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: backend
  labels:
    app.kubernetes.io/managed-by: iter8
    iter8.tools/kind: routemap
    iter8.tools/version: "v0.17"
immutable: true
data:
  strSpec: |
    versions:
    - resources:
      - gvrShort: svc
        name: backend
        namespace: default
      - gvrShort: deploy
        name: backend
        namespace: default
    - resources:
      - gvrShort: svc
        name: backend-candidate-1
        namespace: default
      - gvrShort: deploy
        name: backend-candidate-1
        namespace: default
EOF

In this definition, each version of the application is composed of a Service and a Deployment. In the primary version, both are named backend. In any candidate version they are named backend-candidate-1. Iter8 uses this definition to identify when any of the versions of the application are available. It can then respond appropriately to Lookup() requests.

Generate load¶

In separate shells, port-forward requests to the frontend component and generate load for multiple users. A script is provided to do this. To use it:

kubectl port-forward service/frontend 8090:8090

curl -s https://raw.githubusercontent.com/iter8-tools/docs/v0.15.0/samples/abn-sample/generate_load.sh | sh -s --

Deploy candidate¶

Deploy the candidate version of the backend component, naming it backend-candidate-1.

kubectl create deployment backend-candidate-1 --image=iter8/abn-sample-backend:0.13-v2
kubectl label deployment backend-candidate-1 iter8.tools/watch="true"

kubectl expose deployment backend-candidate-1 --name=backend-candidate-1 --port=8091

Until the candidate version is ready; that is, until all expected resources are deployed and available, calls to Lookup() will return only the version number 0; the existing version. Once the candidate version is ready, Lookup() will return both version numbers (0 and 1) so that requests can be distributed across versions.

Compare versions using Grafana¶

Inspect the metrics using Grafana. If Grafana is deployed to your cluster, port-forward requests as follows:

kubectl port-forward service/grafana 3000:3000

Open Grafana in a browser by going to http://localhost:3000

Add a JSON API data source default/backend with the following parameters:

• URL: http://iter8.default:8080/abnDashboard
• Query string: namespace=default&application=backend

Create a new dashboard by import. Copy and paste the contents of the abn Grafana dashboard into the text box and load it. Associate it with the JSON API data source above.

The Iter8 dashboard allows you to compare the behavior of the two versions of the backend component against each other and select a winner. Since user requests are being sent by the load generation script, the values in the report may change over time. The Iter8 dashboard will look like the following:

Once you identify a winner, it can be promoted, and the candidate version deleted.

Promote candidate¶

To promote the candidate version (backend-candidate-1), first update the primary version, backend, using the new image. You can also overwrite any metadata describing the version.

kubectl set image deployment/backend abn-sample-backend=iter8/abn-sample-backend:0.13-v2

Finally, delete the candidate version:

kubectl delete svc/backend-candidate-1 deploy/backend-candidate-1

Calls to Lookup() will now recommend that all traffic be sent to the primary version backend (currently serving the promoted version of the code).

Cleanup¶

Delete the sample application:

kubectl delete \
svc/frontend deploy/frontend \
svc/backend deploy/backend \
svc/backend-candidate-1 deploy/backend-candidate-1

Delete the application description:

kubectl delete cm/backend

Uninstall the Iter8 controller:

helm delete iter8

kubectl delete -k 'https://github.com/iter8-tools/iter8.git/kustomize/controller/namespaceScoped?ref=v0.17.1'

kubectl delete -k 'https://github.com/iter8-tools/iter8.git/kustomize/controller/clusterScoped?ref=v0.17.1'