SLO validation using custom metrics (multiple versions)¶
Validate SLOs for multiple versions of an app by fetching metrics for each app version from a database (like Prometheus). This is a multi-loop Kubernetes experiment.
Before you begin
- Try your first experiment. Understand the main concepts behind Iter8 experiments. Try an SLO validation experiment using custom metrics for a single version of an app.
- Complete the Istio traffic mirroring tutorial, specifically, the Before you begin section, the Creating a default routing policy section, and the Mirroring traffic to v2 section. Omit the the
Cleaning up
step (you can clean up once you are done with this tutorial). - Install Istio's Prometheus add-on.
- Generate load.
kubectl run fortio --image=fortio/fortio --command -- fortio load -t 6000s http://httpbin.default:8000/get
Launch experiment¶
iter8 k launch \
--set "tasks={custommetrics,assess}" \
--set custommetrics.templates.istio-prom="https://raw.githubusercontent.com/iter8-tools/iter8/v0.12.2-pre/testdata/custommetrics/istio-prom.tpl" \
--set custommetrics.values.labels.namespace=default \
--set custommetrics.values.labels.destination_app=httpbin \
--set custommetrics.values.labels.reporter=destination \
--set 'custommetrics.versionValues[0].labels.destination_version=v1' \
--set 'custommetrics.versionValues[1].labels.destination_version=v2' \
--set assess.SLOs.upper.istio-prom/error-rate=0 \
--set assess.SLOs.upper.istio-prom/latency-mean=100 \
--set runner=cronjob \
--set cronjobSchedule="*/1 * * * *"
About this experiment
This experiment extends the SLO validation experiment using custom metrics for a single app version. There are two versions of the app in this experiment. Variable values that are specific to the first version are specified under custommetrics.versionValues[0]
, while those that are specific to the second version are specified under custommetrics.versionValues[1]
. For the first version, Iter8 merges custommetrics.values
with custommetrics.versionValues[0]
(the latter takes precedence), and uses the result for template variable substitution. Similarly, for the second version, Iter8 merges custommetrics.values
with custommetrics.versionValues[1]
(the latter takes precedence), and uses the result for template variable substitution.
Assert experiment outcomes, view experiment report, view experiment logs, and cleanup as described in this experiment.
Some variations and extensions of this experiment
- Define and use your own provider templates. This enables you to use any app-specific metrics from any database as part of Iter8 experiments. Read the documentation for the
custommetrics
task to learn more. - Alter the
cronjobSchedule
expression so that experiment loops are repeated at a frequency of your choice. Use use https://crontab.guru to learn more aboutcronjobSchedule
expressions.