Skip to content

Benchmark and Validate HTTP services

The slo-validation-istio collects latency and error-related metrics and validates service-level objectives (SLOs) from an application using Istio.

Before you begin

  1. Install Istio.

  2. Install Prometheus plugin.

  3. Enable automatic Istio sidecar injection for the default namespace. This ensures that the pods created in steps 4 and 5 will have the Istio sidecar. 

    kubectl label namespace default istio-injection=enabled --overwrite

  4. Deploy the sample HTTP service in the Kubernetes cluster. 

    kubectl create deploy httpbin --image=kennethreitz/httpbin --port=80
kubectl expose deploy httpbin --port=80

  5. Generate load. 

    kubectl run fortio --image=fortio/fortio --command -- fortio load -t 6000s http://httpbin.default/get

Validate SLOs for a service

The slo-validation-istio experiment will regularly check for metrics and validate SLOs.

The following metrics are collected by default by this experiment:

  • istio/request-count: total number of requests sent
  • istio/error-count: number of error responses
  • istio/error-rate: fraction of error responses
  • istio/latency-mean: mean of observed latency values
  • istio/le500ms-latency-percentile: percentile of requests with less than or equal latency of 500ms

All latency metrics have msec units.

In order to run the slo-validation-istio experiment, the following need to be set.

  • cronjobSchedule, the cron schedule that determines when the metrics collection and SLO validation is run. It is currently configured to run every minute.
  • providerURL, the URL of the metrics provider, in this case Prometheus database endpoint
  • destination_workload, the name of the service
  • destination_workload_namespace, the namespace of the service

In addition, the SLOs to be validated also need to be set. In this case, istio/error-rate and istio/latency-mean will be validated.

iter8 k launch -c slo-validation-istio \
--set cronjobSchedule="*/1 * * * *" \
--set providerURL=http://prometheus.istio-system:9090 \
--set versionInfo.destination_workload=httpbin \
--set versionInfo.destination_workload_namespace=default \
--set SLOs.istio/error-rate=0 \
--set SLOs.istio/latency-mean=100

In the experiment above, the following SLOs are validated.

  • error rate is 0
  • mean latency is under 100 msec

View experiment report

iter8 k report
The text report looks like this 
Experiment summary:
*******************

  Experiment completed: false
  No task failures: true
  Total number of tasks: 2
  Number of completed tasks: 6

Whether or not service level objectives (SLOs) are satisfied:
*************************************************************

  SLO Conditions        |Satisfied
  --------------        |---------
  istio/error-rate <= 0 |true
  istio/latency-mean <= 100 |true


Latest observed values for metrics:
***********************************

  Metric                           |value
  -------                          |-----
  istio/error-count                |0.00
  istio/error-rate                 |0.00
  istio/latency-mean               |6.31
  istio/le500ms-latency-percentile |1.00
  istio/request-count              |2110.00

iter8 k report -o html > report.html # view in a browser
The HTML report looks like this

HTML report

Because the cronjobSchedule has been set to run every minute, the report will change periodically.

Assert experiment outcomes

Assert that the experiment encountered no failures, and all SLOs are satisfied. 

iter8 k assert -c nofailure -c slos
Sample output from assert 
INFO[2021-11-10 09:33:12] experiment has no failure                    
INFO[2021-11-10 09:33:12] SLOs are satisfied                           
INFO[2021-11-10 09:33:12] all conditions were satisfied
Back to top