Benchmark and Validate gRPC with SLOs¶

The load-test-grpc experiment generates call requests for gRPC services, collects latency and error-related metrics, and validates service-level objectives (SLOs).

This experiment is designed for the following use-cases.

• Load test
• Benchmark
• Validate service level objectives (SLOs)
• Safe rollout
• Continuous integration and delivery (CI/CD)

docker run -p 50051:50051 docker.io/grpc/java-example-hostname:latest

Basic example¶

Benchmark a gRPC service by specifying its host, its fully-qualified call (method) name, and the URL of Protocol Buffer file (protoURL) that defines the service.

iter8 launch -c load-test-grpc \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto"

Metrics and SLOs¶

The following metrics are collected by default by this experiment:

• grpc/request-count: total number of requests sent
• grpc/error-count: number of error responses
• grpc/error-rate: fraction of error responses

The following latency metrics are also supported.

• grpc/latency/mean: Mean latency
• grpc/latency/stddev: Standard deviation of latency
• grpc/latency/min: Min latency
• grpc/latency/max: Max latency
• grpc/latency/pX: X-th percentile latency, for any X in the range 0.0 to 100.0

Latency metrics have msec units. Any latency metric that is specified as part of SLOs is also collected.

For example, set the following parameter values in the iter8 launch command above.

--set SLOs.grpc/error-rate=0 \
--set SLOs.grpc/latency/mean=50 \
--set SLOs.grpc/latency/p90=100 \
--set SLOs.grpc/latency/p'97\.5'=200

In the above setting, the following SLOs are validated.

• error rate is 0
• mean latency is under 50 msec
• 90^th percentile latency is under 100 msec
• 97.5^th percentile latency is under 200 msec

View experiment report¶

iter8 report

Experiment summary:
*******************

  Experiment completed: true
  No task failures: true
  Total number of tasks: 2
  Number of completed tasks: 2

Whether or not service level objectives (SLOs) are satisfied:
*************************************************************

  SLO Conditions                   |Satisfied
  --------------                   |---------
  grpc/error-rate <= 0             |true
  grpc/latency/mean (msec) <= 50   |true
  grpc/latency/p90 (msec) <= 100   |true
  grpc/latency/p97.5 (msec) <= 200 |true


Latest observed values for metrics:
***********************************

  Metric                    |value
  -------                   |-----
  grpc/error-count          |0.00
  grpc/error-rate           |0.00
  grpc/latency/mean (msec)  |21.48
  grpc/latency/p90 (msec)   |34.00
  grpc/latency/p97.5 (msec) |37.00
  grpc/request-count        |200.00

iter8 report -o html > report.html # view in a browser

Assertions¶

Assert that the experiment completed without failures, and all SLOs are satisfied.

iter8 assert -c completed -c nofailure -c slos

The iter8 assert subcommand asserts if the experiment result satisfies conditions that are specified. If assert conditions are satisfied, it exits with code 0; else, it exits with code 1. Assertions are especially useful inside CI/CD/GitOps pipelines.

INFO[2021-11-10 09:33:12] experiment completed
INFO[2021-11-10 09:33:12] experiment has no failure                    
INFO[2021-11-10 09:33:12] SLOs are satisfied                           
INFO[2021-11-10 09:33:12] all conditions were satisfied

Load profile¶

Control the characteristics of the generated load generated by setting the number of requests (total), the number of requests per second (rps), number of connections to use (connections), and the number of concurrent request workers to use which will be distributed across the connections (concurrency).

--set total=500 \
--set rps=25 \
--set concurrency=50 \
--set connections=10

Refer to the chart's values.yaml file for additional parameters related to the load profile such as duration, maxDuration, connectTimeout, and keepalive.

Call data¶

gRPC calls may include data serialized as Protocol Buffer messages.

--set data.name="frodo"

--set data.name="frodo" \
--set data.realm.planet="earth" \
--set data.realm.location="middle" 

--set dataFile="/the/path/to/data.json" # "./data.json" also works

--set dataURL="https://location.of/data.json"

--set binaryDataFile="/the/path/to/data.bin" # "./data.bin" also works

--set binaryDataURL="https://location.of/data.bin"

For client streaming or bi-directional calls, this experiment accepts an array of messages, each element representing a single message within the stream call. If a single object is given for data, then it is automatically converted to an array with single element.

--set data[0].name="Joe" \
--set data[1].name="Kate" \
--set data[2].name="Sara"

In case of client streaming, this experiment sends all the data in the input array, and then closes and receives.

Call metadata¶

gRPC calls may include metadata which is information about a particular call.

--set metadata.darth="vader" \
--set metadata.lord="sauron" \
--set metadata.volde="mort"

--set metadataFile="/the/path/to/metadata.json" # "./metadata.json" also works

--set metadataURL="https://location.of/metadata.json"

Proto and reflection¶

The gRPC server method signatures and message formats are defined in a .proto source file, which may also be compiled to a .protoset file.

--set protoFile="/path/to/helloworld.proto" # "./helloworld.proto" also works

--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto"

--set protosetFile="./myservice.protoset"

--set protosetURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.protoset"

--set reflectMetadata.clientId="5hL64dd0" \
--set reflectMetadata.clientMood="delightful"

Streaming gRPC¶

Refer to the values.yaml file which documents additional parameters related to streaming gRPC.