Skip to content

Benchmark and Validate HTTP with SLOs

The load-test-http experiment generates call requests for HTTP services, collects latency and error-related metrics, and validates service-level objectives (SLOs).

This experiment is designed for the following use-cases.

  • Load test
  • Benchmark
  • Validate service level objectives (SLOs)
  • Safe rollout
  • Continuous integration and delivery (CI/CD)
Before you begin

Run the httpbin sample service from a separate terminal. 

docker run -p 80:80 kennethreitz/httpbin
You can also use Podman or other alternatives to Docker in the above command.

Basic example

Benchmark an HTTP service with a GET endpoint by specifying the url.

iter8 launch -c load-test-http --set url=http://127.0.0.1/get

Metrics and SLOs

The following metrics are collected by default by this experiment:

  • http/request-count: total number of requests sent
  • http/error-count: number of error responses
  • http/error-rate: fraction of error responses
  • http/latency-mean: mean of observed latency values
  • http/latency-stddev: standard deviation of observed latency values
  • http/latency-min: min of observed latency values
  • http/latency-max: max of observed latency values
  • http/latency-pX: Xth percentile latency, for X in [50.0, 75.0, 90.0, 95.0, 99.0, 99.9]

Latency metrics have msec units. Any latency percentile that is specified as part of SLOs is also collected.

For example, set the following parameter values in the iter8 launch command above.

--set SLOs.http/error-rate=0 \
--set SLOs.http/latency-mean=50 \
--set SLOs.http/latency-p90=100 \
--set SLOs.http/latency-p'97\.5'=200

In the above setting, the following SLOs are validated.

  • error rate is 0
  • mean latency is under 50 msec
  • 90th percentile latency is under 100 msec
  • 97.5th percentile latency is under 200 msec

View experiment report

iter8 report
The text report looks like this 
Experiment summary:
*******************

  Experiment completed: true
  No task failures: true
  Total number of tasks: 2
  Number of completed tasks: 2

Whether or not service level objectives (SLOs) are satisfied:
*************************************************************

  SLO Conditions                   |Satisfied
  --------------                   |---------
  http/error-rate <= 0             |true
  http/latency-mean (msec) <= 50   |true
  http/latency-p90 (msec) <= 100   |true
  http/latency-p97.5 (msec) <= 200 |true


Latest observed values for metrics:
***********************************

  Metric                     |value
  -------                    |-----
  http/error-count           |0.00
  http/error-rate            |0.00
  http/latency-max (msec)    |8.93
  http/latency-mean (msec)   |5.49
  http/latency-min (msec)    |2.71
  http/latency-p50 (msec)    |5.38
  http/latency-p75 (msec)    |6.71
  http/latency-p90 (msec)    |7.77
  http/latency-p95 (msec)    |8.27
  http/latency-p97.5 (msec)  |8.60
  http/latency-p99 (msec)    |8.80
  http/latency-p99.9 (msec)  |8.92
  http/latency-stddev (msec) |1.57
  http/request-count         |100.00

iter8 report -o html > report.html # view in a browser
The HTML report looks like this

HTML report

Assertions

Assert that the experiment completed without failures, and all SLOs are satisfied. 

iter8 assert -c completed -c nofailure -c slos

The iter8 assert subcommand asserts if the experiment result satisfies conditions that are specified. If assert conditions are satisfied, it exits with code 0; else, it exits with code 1. Assertions are especially useful inside CI/CD/GitOps pipelines.

Sample output from assert 
INFO[2021-11-10 09:33:12] experiment completed
INFO[2021-11-10 09:33:12] experiment has no failure                    
INFO[2021-11-10 09:33:12] SLOs are satisfied                           
INFO[2021-11-10 09:33:12] all conditions were satisfied

Load profile

Control the characteristics of the load generated by the load-test-http experiment by setting the number of queries (numQueries), the number of queries sent per second (qps), and the number of parallel connections used to send queries (connections).

--set numQueries=200 \
--set qps=10 \
--set connections=5

You can also set the duration of the load generation period. Refer to the chart's values.yaml file for details.

Payload

Send any type of content as payload and specify the HTTP Content Type header. These options switch the HTTP method to POST.

Specify payload as a string. By default, its content type is set to application/octet-stream. 

--set payloadStr="abc123"


--set payloadStr="abc123" \
--set contentType="text/plain"

Fetch JSON content from a URL. Use this JSON as payload. Set content type to application/json. 

--set payloadURL=https://data.police.uk/api/crimes-street-dates \
--set contentType="application/json"

Fetch jpeg image from a URL. Use this image as payload. Set content type to image/jpeg. 

--set payloadURL=https://cdn.pixabay.com/photo/2021/09/08/17/58/poppy-6607526_1280.jpg \
--set contentType="image/jpeg"

Back to top