Skip to content

SLO validation using custom metrics (multiple versions)

Validate SLOs for multiple versions of an app by fetching metrics for each app version from a database (like Prometheus). This is a multi-loop Kubernetes experiment.


Before you begin
  1. Try your first experiment. Understand the main concepts behind Iter8 experiments. Try an SLO validation experiment using custom metrics for a single version of an app.
  2. Complete the Istio traffic mirroring tutorial, specifically, the setup steps, the step for creating the default routing policy, and the step for mirroring traffic. Omit the step for cleaning up (you can clean up once you are done with this tutorial).
  3. Install Istio's Prometheus add-on.
  4. Generate load.
    kubectl run fortio --image=fortio/fortio --command -- fortio load -t 6000s http://httpbin.default:8000/get
    

Launch experiment

iter8 k launch \
--set "tasks={custommetrics,assess}" \
--set custommetrics.templates.istio-prom="https://raw.githubusercontent.com/iter8-tools/iter8/master/custommetrics/istio-prom.tpl" \
--set custommetrics.values.destinationWorkloadNamespace=default \
--set custommetrics.values.reporter=destination \
--set custommetrics.versionValues[0].destinationWorkload=httpbin-v1 \
--set custommetrics.versionValues[1].destinationWorkload=httpbin-v2 \
--set assess.SLOs.upper.istio-prom/error-rate=0 \
--set assess.SLOs.upper.istio-prom/latency-mean=100 \
--set runner=cronjob \
--set cronjobSchedule="*/1 * * * *"
About this experiment

This experiment extends the SLO validation experiment using custom metrics for a single app version. There are two versions of the app in this experiment. Variable values that are specific to the first version are specified under custommetrics.versionValues[0], while those that are specific to the second version are specified under custommetrics.versionValues[1]. For the first version, Iter8 merges custommetrics.values with custommetrics.versionValues[0] (the latter takes precedence), and uses the result for template variable substitution. Similarly, for the second version, Iter8 merges custommetrics.values with custommetrics.versionValues[1] (the latter takes precedence), and uses the result for template variable substitution.

Some variations and extensions of this experiment
  1. Define and use your own provider templates. This enables you to use any app-specific metrics from any database as part of Iter8 experiments. Read the documentation for the custommetrics task to learn more.
  2. Alter the cronjobSchedule expression so that experiment loops are repeated at a frequency of your choice. Use use https://crontab.guru to learn more about cronjobSchedule expressions.

Assert experiment outcomes, view experiment report, view experiment logs, and cleanup as described in this experiment.

Back to top