Skip to content

Conformance Tutorial

Perform an Iter8-Knative experiment with Conformance testing.

Canary

You will create the following resources in this tutorial.

  1. A Knative app (service) with a single version (revision).
  2. A fortio-based traffic generator that simulates user requests.
  3. An Iter8 experiment that verifies that baseline satisfies mean latency, 95th percentile tail latency, and error rate objectives.
Before you begin, you will need ...

Kubernetes cluster: Ensure that you have Kubernetes cluster with Iter8 and Knative installed. You can do this by following Steps 1, 2, and 3 of the quick start tutorial for Knative.

Cleanup: If you ran an Iter8 tutorial earlier, run the associated cleanup step.

ITER8: Ensure that ITER8 environment variable is set to the root directory of your cloned Iter8 repo. See Step 2 of the quick start tutorial for Knative for example.

iter8ctl: This tutorial uses iter8ctl.

1. Create app

kubectl apply -f $ITER8/samples/knative/conformance/baseline.yaml
Look inside baseline.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: sample-app 
namespace: default 
spec:
template:
    metadata:
    name: sample-app-v1
    spec:
    containers:
    - image: gcr.io/knative-samples/knative-route-demo:blue 
        env:
        - name: T_VERSION
        value: "blue"

2. Generate requests

kubectl wait --for=condition=Ready ksvc/sample-app
URL_VALUE=$(kubectl get ksvc sample-app -o json | jq .status.address.url)
sed "s+URL_VALUE+${URL_VALUE}+g" $ITER8/samples/knative/conformance/fortio.yaml | kubectl apply -f -
Look inside experiment.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: fortio
spec:
  template:
    spec:
      volumes:
      - name: shared
        emptyDir: {}    
      containers:
      - name: fortio
        image: fortio/fortio
        command: ["fortio", "load", "-t", "120s", "-json", "/shared/fortiooutput.json", $(URL)]
        env:
        - name: URL
          value: URL_VALUE
        volumeMounts:
        - name: shared
          mountPath: /shared         
      - name: busybox
        image: busybox:1.28
        command: ['sh', '-c', 'echo busybox is running! && sleep 600']          
        volumeMounts:
        - name: shared
          mountPath: /shared       
      restartPolicy: Never

3. Create Iter8 experiment

kubectl apply -f $ITER8/samples/knative/conformance/experiment.yaml
Look inside experiment.yaml
apiVersion: iter8.tools/v2alpha1
kind: Experiment
metadata:
  name: conformance-sample
spec:
  # target identifies the knative service under experimentation using its fully qualified name
  target: default/sample-app
  strategy:
    # this experiment will perform a conformance test
    testingPattern: Conformance
    actions:
      start: # run the following sequence of tasks at the start of the experiment
      - library: knative
        task: init-experiment
  criteria:
    # mean latency of version should be under 50 milliseconds
    # 95th percentile latency should be under 100 milliseconds
    # error rate should be under 1%
    objectives: 
    - metric: mean-latency
      upperLimit: 50
    - metric: 95th-percentile-tail-latency
      upperLimit: 100
    - metric: error-rate
      upperLimit: "0.01"
  duration:
    intervalSeconds: 10
    iterationsPerLoop: 10
  versionInfo:
    # information about versions used in this experiment
    baseline:
      name: current
      variables:
      - name: revision
        value: sample-app-v1

4. Observe experiment

Observe the experiment in realtime. Paste commands from the tabs below in separate terminals.

Periodically describe the experiment. 

while clear; do
kubectl get experiment conformance-sample -o yaml | iter8ctl describe -f -
sleep 4
done

iter8ctl output

iter8ctl output will be similar to the following. 

****** Overview ******
Experiment name: conformance-sample
Experiment namespace: default
Target: default/sample-app
Testing pattern: Conformance
Deployment pattern: Progressive

****** Progress Summary ******
Experiment stage: Running
Number of completed iterations: 3

****** Winner Assessment ******
Winning version: not found
Recommended baseline: current

****** Objective Assessment ******
+--------------------------------+---------+
|           OBJECTIVE            | CURRENT |
+--------------------------------+---------+
| mean-latency <= 50.000         | true    |
+--------------------------------+---------+
| 95th-percentile-tail-latency   | true    |
| <= 100.000                     |         |
+--------------------------------+---------+
| error-rate <= 0.010            | true    |
+--------------------------------+---------+

****** Metrics Assessment ******
+--------------------------------+---------+
|             METRIC             | CURRENT |
+--------------------------------+---------+
| request-count                  | 448.000 |
+--------------------------------+---------+
| mean-latency (milliseconds)    |   1.338 |
+--------------------------------+---------+
| 95th-percentile-tail-latency   |   4.770 |
| (milliseconds)                 |         |
+--------------------------------+---------+
| error-rate                     |   0.000 |
+--------------------------------+---------+
When the experiment completes (in ~ 2 mins), you will see the experiment stage change from Running to Completed.

kubectl get experiment conformance-sample --watch
kubectl get experiment output

kubectl output will be similar to the following. 

NAME                 TYPE          TARGET               STAGE          COMPLETED ITERATIONS   MESSAGE    
conformance-sample   Conformance   default/sample-app   Running        0                      StartHandlerLaunched: Start handler 'start' launched
conformance-sample   Conformance   default/sample-app   Running        1                      IterationUpdate: Completed Iteration 1
conformance-sample   Conformance   default/sample-app   Running        2                      IterationUpdate: Completed Iteration 2
conformance-sample   Conformance   default/sample-app   Running        3                      IterationUpdate: Completed Iteration 3
conformance-sample   Conformance   default/sample-app   Running        4                      IterationUpdate: Completed Iteration 4
conformance-sample   Conformance   default/sample-app   Running        5                      IterationUpdate: Completed Iteration 5
conformance-sample   Conformance   default/sample-app   Running        6                      IterationUpdate: Completed Iteration 6
conformance-sample   Conformance   default/sample-app   Running        7                      IterationUpdate: Completed Iteration 7
When the experiment completes (in ~ 2 mins), you will see the experiment stage change from Running to Completed.

5. Cleanup

kubectl delete -f $ITER8/samples/knative/conformance/fortio.yaml
kubectl delete -f $ITER8/samples/knative/conformance/experiment.yaml
kubectl delete -f $ITER8/samples/knative/conformance/baseline.yaml
Understanding what happened
  1. You created a Knative service with a single revision, sample-app-v1.
  2. You generated requests for the Knative service using a fortio-job.
  3. You created an Iter8 Conformance experiment. In each iteration, Iter8 observed the mean latency, 95th percentile tail-latency, and error-rate metrics collected by Prometheus, and verified that baseline satisfied all the objectives specified in the experiment.