Benchmark and Validate gRPC services¶

The load-test-grpc experiment generates call requests for gRPC services, collects latency and error-related metrics, and validates service-level objectives (SLOs).

This experiment is designed for the following use-cases.

Load test
Benchmark
Validate service level objectives (SLOs)
Safe rollout
Continuous integration and delivery (CI/CD)

Before you begin

Run the gRPC sample service from a separate terminal.

docker run -p 50051:50051 docker.io/grpc/java-example-hostname:latest

You can also use Podman or other alternatives to Docker in the above command.

Basic example¶

Benchmark a gRPC service by specifying its host, its fully-qualified call (method) name, and the URL of Protocol Buffer file (protoURL) that defines the service.

Launch load-test-grpc experiment

iter8 launch -c load-test-grpc \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto"

Specify metrics and SLOs¶

The following metrics are collected by default by this experiment:

grpc/request-count: total number of requests sent
grpc/error-count: number of error responses
grpc/error-rate: fraction of error responses

The following latency metrics are also supported.

grpc/latency/mean: Mean latency
grpc/latency/stddev: Standard deviation of latency
grpc/latency/min: Min latency
grpc/latency/max: Max latency
grpc/latency/pX: X-th percentile latency, for any X in the range 0.0 to 100.0

Latency metrics have msec units.

For example, invoke iter8 launch as follows. The --noDownload flag reuses the Iter8 experiment charts folder downloaded during the previous iter8 launch invocation.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set SLOs.grpc/error-rate=0 \
--set SLOs.grpc/latency/mean=50 \
--set SLOs.grpc/latency/p90=100 \
--set SLOs.grpc/latency/p'97\.5'=200

In the above setting, the following SLOs are validated.

error rate is 0
mean latency is under 50 msec
90^th percentile latency is under 100 msec
97.5^th percentile latency is under 200 msec

View experiment report¶

TextHTML

iter8 report

The text report looks like this

Experiment summary:
*******************

  Experiment completed: true
  No task failures: true
  Total number of tasks: 2
  Number of completed tasks: 2

Whether or not service level objectives (SLOs) are satisfied:
*************************************************************

  SLO Conditions                   |Satisfied
  --------------                   |---------
  grpc/error-rate <= 0             |true
  grpc/latency/mean (msec) <= 50   |true
  grpc/latency/p90 (msec) <= 100   |true
  grpc/latency/p97.5 (msec) <= 200 |true


Latest observed values for metrics:
***********************************

  Metric                    |value
  -------                   |-----
  grpc/error-count          |0.00
  grpc/error-rate           |0.00
  grpc/latency/mean (msec)  |21.48
  grpc/latency/p90 (msec)   |34.00
  grpc/latency/p97.5 (msec) |37.00
  grpc/request-count        |200.00

iter8 report -o html > report.html # view in a browser

The HTML report looks like this

HTML report

Assert experiment outcomes¶

Assert that the experiment completed without failures, and all SLOs are satisfied.

iter8 assert -c completed -c nofailure -c slos

The iter8 assert command asserts if the experiment result satisfies conditions that are specified. If assert conditions are satisfied, it exits with code 0; else, it exits with code 1. Assertions are especially useful inside CI/CD/GitOps pipelines.

Sample output from assert

INFO[2021-11-10 09:33:12] experiment completed
INFO[2021-11-10 09:33:12] experiment has no failure                    
INFO[2021-11-10 09:33:12] SLOs are satisfied                           
INFO[2021-11-10 09:33:12] all conditions were satisfied

Specify load profile¶

Control the characteristics of the generated load generated by setting the number of requests (total), the number of requests per second (rps), number of connections to use (connections), and the number of concurrent request workers to use which will be distributed across the connections (concurrency).

Sample load profile

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set total=500 \
--set rps=25 \
--set concurrency=50 \
--set connections=10

Specify call data¶

gRPC calls may include data serialized as Protocol Buffer messages.

DataData fileData URLBinary data fileBinary data URL

Specify call data as values.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set data.name="frodo"

Specifying nested data

Nested data can also be specified. For example, assuming data.name, data.realm.planet, and data.realm.location are all valid fields, they can be specified as follows.

Nested

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set data.name="frodo" \
--set data.realm.planet="earth" \
--set data.realm.location="middle"

Create a local JSON file.

cat << EOF > data.json
{
  "name": "Iter8 user"
}
EOF

Specify call data JSON file as value.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set data-file="data.json"

Specify a URL that responds with JSON data. Iter8 will download the data and use it in the requests.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set dataURL="https://gist.githubusercontent.com/sriumcp/3f3178f4b698af6696c925832e51b0ba/raw/d02aa698d34aa2067f7a2f6afb4ceb616b0db822/name.json"

Use binary data from a local file serialized as a single binary message or multiple count-prefixed messages.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set binary-file="/the/path/to/data.bin" # "./data.bin" also works

Supply a URL that hosts binary data serialized as a single binary message or multiple count-prefixed messages. Iter8 will download the data from this URL and use it in the requests.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set binaryDataURL="https://location.of/data.bin"

For client streaming or bi-directional calls, this experiment accepts an array of messages, each element representing a single message within the stream call. If a single object is given for data, then it is automatically converted to an array with single element.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set data[0].name="Joe" \
--set data[1].name="Kate" \
--set data[2].name="Sara"

In case of client streaming, this experiment sends all the data in the input array, and then closes and receives.

Specify call metadata¶

gRPC calls may include metadata which is information about a particular call.

MetadataMetadata fileMetadata URL

Supply metadata as values.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set metadata.darth="vader" \
--set metadata.lord="sauron" \
--set metadata.volde="mort"

Use JSON metadata from a local file.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set metadata-file="/the/path/to/metadata.json" # "./metadata.json" also works

Supply a URL that hosts JSON metadata. Iter8 will download the metadata from this URL and use it in the requests.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto" \
--set metadataURL="https://location.of/metadata.json"

Specify proto and reflection¶

The gRPC server method signatures and message formats are defined in a .proto source file, which may also be compiled to a .protoset file.

Proto fileProto URLProtoset fileProtoset URLReflection

Use a local .proto source file.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set proto="/path/to/helloworld.proto" # "./helloworld.proto" also works

Use a URL that hosts a .proto source file. Iter8 will download the Protocol Buffer file and use it in the experiment.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.proto"

Supply the name of a .protoset file that is compiled from .proto source files.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protoset="./myservice.protoset"

Supply a URL that hosts a .protoset file.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set protosetURL="https://raw.githubusercontent.com/grpc/grpc-go/master/examples/helloworld/helloworld/helloworld.protoset"

In the absence of .proto and .protoset information, the experiment will attempt to use server reflection. You can supply reflect metadata.

iter8 launch -c load-test-grpc --noDownload \
--set host="127.0.0.1:50051" \
--set call="helloworld.Greeter.SayHello" \
--set reflect-metadata.clientId="5hL64dd0" \
--set reflect-metadata.clientMood="delightful"

To learn more about all the parameters of the load-test-grpc chart and their default values, please refer to the chart's values.yaml file.