custommetrics¶

Fetch metrics from databases (like Prometheus) and other REST APIs.

Usage Example¶

In this example, the custommetrics task fetches metrics from the Prometheus database that is created by Istio's Prometheus add-on.

iter8 k launch \
--set "tasks={custommetrics,assess}" \
--set custommetrics.templates.istio-prom="https://raw.githubusercontent.com/iter8-tools/iter8/v0.14.5/templates/custommetrics/istio-prom.tpl" \
--set custommetrics.values.labels.destination_app=httpbin \
--set custommetrics.values.labels.namespace=default \
--set assess.SLOs.upper.istio-prom/error-rate=0 \
--set assess.SLOs.upper.istio-prom/latency-mean=100 \
--set runner=cronjob \
--set cronjobSchedule="*/1 * * * *"

Parameters¶

How it works¶

The logic of this task is illustrated by the following flowchart.

graph TD
  A([Start]) --> B([Get provider template]);
  B --> C([Compute variable values]);
  C --> D([Create provider spec by combining template with values]);
  D --> E([Query database]);
  E --> F([Process response]);
  F --> G([Update metric value in experiment]);
  G --> H{Done with all metrics?};
  H ---->|No| E;
  H ---->|Yes| I{Done with all versions?};
  I ---->|No| C;
  I ---->|Yes| J([End]);

We describe the concepts or provider spec and provider template next.

Provider spec¶

Iter8 needs the information following in order to fetch metrics from a database.

1. The HTTP URL where the database can be queried.
2. The HTTP headers and method (GET/POST) to be used while querying the database.
3. For each metric to be fetched from the database:
   • The specific HTTP query to be used, in particular, the HTTP query parameters and body (if any).
   • The logic for parsing the query response and retrieving the metric value.

The above information is encapsulated by ProviderSpec, a data structure which Iter8 associates with each provider, and Metric, a data structure which Iter8 associates with each metric provided by a provider.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

type ProviderSpec struct {
  // URL is the database endpoint
  URL string `json:"url" yaml:"url"`
  // Method is the HTTP method that needs to be used
  Method string `json:"method" yaml:"method"`
  // Headers is the set of HTTP headers that need to be sent
  Headers map[string]string `json:"headers" yaml:"headers"`
  // Metrics is the set of metrics that can be obtained
  Metrics []Metric `json:"metrics" yaml:"metrics"`
}

type Metric struct {
  // Name is the name of the metric
  Name string `json:"name" yaml:"name"`
  // Description is the description of the metric
  Description *string `json:"description,omitempty" yaml:"description,omitempty"`
  // Type is the type of the metric, either gauge or counter
  Type string `json:"type" yaml:"type"`
  // Units is the unit of the metric, which can be omitted for unitless metrics
  Units *string `json:"units,omitempty" yaml:"units,omitempty"`
  // Params is the set of HTTP parameters that need to be sent
  Params *[]HTTPParam `json:"params,omitempty" yaml:"params,omitempty"`
  // Body is the HTTP request body that needs to be sent
  Body *string `json:"body,omitempty" yaml:"body,omitempty"`
  // JqExpression is the jq expression that can extract the value from the HTTP
  // response
  JqExpression string `json:"jqExpression" yaml:"jqExpression"`
}

type HTTPParam struct {
  // Name is the name of the HTTP parameter
  Name string `json:"name" yaml:"name"`
  // Value is the value of the HTTP parameter
  Value string `json:"value" yaml:"value"`
}

The ProviderSpec and Metric data structures together supply Iter8 with all the information needed to query databases, process the response to extract metric values, store the metric values in experiments, and display them in experiment reports with auxiliary information (such as description and units). Metric types are defined here.

Provider template¶

Rather than supplying provider specs directly, Iter8 enables users to supply one or more Golang templates for provider specs. Iter8 combines the provider templates with values, in order to generate provider specs in YAML format, and uses them to query for the metrics.

Example providers specs: * istio-prom for Istio's Prometheus plugin

In order to create provider templates and use them in experiments, it is necessary to have a clear understanding of how variable values are computed, and how the response from the database is processed by Iter8. We describe these steps next.

Computing variable values¶

Variable values are configured explicitly by the user during experiment launch. The sole exception to this rule is the elapsedTimeSeconds variable which is computed by Iter8. Please see the tabs below to learn more about how to configure values and how Iter8 computes elapsedTimeSeconds.

iter8 k launch \
--set "tasks={custommetrics,assess}" \
--set custommetrics.templates.istio-prom="https://raw.githubusercontent.com/iter8-tools/iter8/v0.14.5/templates/custommetrics/istio-prom.tpl" \
--set custommetrics.values.labels.namespace=default \
--set custommetrics.values.labels.destination_app=httpbin \
--set custommetrics.values.labels.reporter=destination \
--set 'custommetrics.versionValues[0].labels.destination_version=v1' \
--set 'custommetrics.versionValues[1].labels.destination_version=v2' \
--set assess.SLOs.upper.istio-prom/error-rate=0 \
--set assess.SLOs.upper.istio-prom/latency-mean=100 \
--set runner=cronjob \
--set cronjobSchedule="*/1 * * * *"

sum(last_over_time(istio_requests_total{
  destination_app="httpbin",
  namespace="default"      
}[3600s]))

sum(last_over_time(istio_requests_total{
  destination_app="httpbin",
  namespace="default"      
}[{{ .elapsedTimeSeconds }}s]))

sequenceDiagram
    startingTime-)loop1: elapsedTimeSeconds=60;
    startingTime-)loop2: elapsedTimeSeconds=120;
    startingTime-)loop3: elapsedTimeSeconds=180;

graph TD
  A([Start]) --> B{startingTime parameter supplied?};
  B ---->|Yes| C([elapsedTimeSeconds = currentTime - startingTime]);
  B ---->|No| D([startingTime = time when experiment was launched]);
  D --> C;
  C --> E([End]);

--set custommetrics.values.startingTime="2020-02-01T09:44:40Z"

--set custommetrics.versionValues[0].startingTime="2020-02-01T09:44:40Z" \
--set custommetrics.versionValues[1].startingTime="2020-02-05T14:22:15Z"

Processing response¶

The metrics provider is expected to respond to Iter8's HTTP request for a metric with a JSON object. The format of this JSON object is provider-specific. Iter8 uses jq to extract the metric value from the JSON response of the provider. The jqExpression used by Iter8 is supplied as part of the metric definition. When the jqExpression is applied to the JSON response, it is expected to yield a number.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "value": [1556823494.744, "21.7639"]
      }
    ]
  }
}    

echo '{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "value": [1556823494.744, "21.7639"]
      }
    ]
  }
}' | jq ".data.result[0].value[1] | tonumber"

Defining and using providers¶

1. Understand how the custommetrics task works; this is described in this section.
2. Create your provider template and serve it from a URL. A sample provider template is in this section.
3. Configure the custommetrics task with one or more provider templates. An example of custommetrics configuration is in this section.
4. The metrics fetched by this task can be used to assess app versions in Iter8 experiments. An example that illustrates the use of both custommetrics and assess tasks together is in this section.