apiVersion:iter8.tools/v2alpha2kind:Experimentmetadata:name:quickstart-expspec:# target identifies the knative service under experimentation using its fully qualified nametarget:default/sample-appstrategy:testingPattern:A/BdeploymentPattern:Progressiveactions:finish:# run the following sequence of tasks at the end of the experiment-if:CandidateWon()run:kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/candidate.yaml-if:not CandidateWon()run:kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/baseline.yamlcriteria:requestCount:iter8-knative/request-countrewards:# Business rewards-metric:iter8-knative/user-engagementpreferredDirection:High# maximize user engagementobjectives:-metric:iter8-knative/mean-latencyupperLimit:50-metric:iter8-knative/95th-percentile-tail-latencyupperLimit:100-metric:iter8-knative/error-rateupperLimit:"0.01"duration:intervalSeconds:10iterationsPerLoop:5versionInfo:# information about app versions used in this experimentbaseline:name:sample-app-v1weightObjRef:apiVersion:serving.knative.dev/v1kind:Servicename:sample-appnamespace:defaultfieldPath:.spec.traffic.percentcandidates:-name:sample-app-v2weightObjRef:apiVersion:serving.knative.dev/v1kind:Servicename:sample-appnamespace:defaultfieldPath:.spec.traffic.percent
This document describes version v2alpha2 of Iter8's experiment API.
Identifies the app under experimentation and determines which experiments can run concurrently. Experiments that have the same target value will not be scheduled concurrently but will be run sequentially in the order of their creation timestamps. Experiments whose target values differ from each other can be scheduled by Iter8 concurrently.
The time when the status was most recently updated.
Indicator of the progress of an experiment. The stage is Waiting before an experiment executes its start action, Initializing while running the start action, Running while the experiment has begun its first iteration and is progressing, Finishing while any finish action is running and Completed when the experiment terminates.
Number of completed iterations of the experiment. This is undefined until the experiment reaches the Running stage.
The latest observed split of traffic between versions. Expressed as percentage. Iter8 ensures that this field is current until the final iteration of the experiment. Iter8 will cease to update this field once a finish action is invoked.
Result of latest query to the Iter8 analytics service.
The version recommended for promotion. This field is initially populated by Iter8 as the baseline version and continuously updated during the course of the experiment to match the winner. The value of this field is typically used by finish actions to promote a version at the end of an experiment.
Determines the logic used to evaluate the app versions and determine the winner of the experiment. Iter8 supports two testing strategies, namely, Canary and Conformance.
Determines if and how traffic is shifted during an experiment. This field is relevant only for experiments using the Canary testing pattern. Iter8 supports two deployment patterns, namely, Progressive and FixedSplit.
An action is a sequence of tasks that can be executed by Iter8. ActionType is a string enum with three valid values: start, loop, and finish. The start action, if specified, is executed at the start of the experiment. The loop action, if specified, is executed during every loop of the experiment, after all the iterations within the loop have completed. The finish action, if specified, is executed at the end of the experiment after all the loops have completed. The actions field is used to specify all three types of actions.
Specification of a task that will be executed as part of experiment actions. Tasks are documented here.
Name of the task. Task names express both the library and the task within the library in the format 'library/task' .
Execute the task only if the specified condition is satisfied. Two built-in conditions are supported namely, WinnerFound() and CandidateWon(). They can be combined with predicates like not: not WinnerFound() and not CandidateWon() are also valid conditions.1
A list of metrics along with acceptable upper limits, lower limits, or both upper and lower limits for them. Iter8 will verify if app versions satisfy these objectives.
A list of metric references. Iter8 will collect and report the values of these metrics in addition to those referenced in the objectives section.
Note: References to metric resource objects within experiment criteria should be in the namespace/name format or in the name format. If the name format is used (i.e., if only the name of the metric is specified), then Iter8 searches for the metric in the namespace of the experiment resource. If Iter8 cannot find the metric, then the reference is considered invalid and the experiment will terminate in a failure.
Maximum number of loops in the experiment. In case of a failure, the experiment may be terminated earlier. Default value = 1.
Number of iterations per experiment loop. In case of a failure, the experiment may be terminated earlier. Default value = 15.
Duration of a single iteration of the experiment in seconds. Default value = 20 seconds.
Note: Suppose an experiment has maxLoops = x, iterationsPerLoop = y, and intervalSeconds = z. Assuming the experiment does not terminate early due to failures, it would take a minimum of x*y*z seconds to complete. The actual duration may be more due to additional time incurred in acquiring the target, and executing the start, loop and finishactions.
Variables are name-value pairs associated with a version. Metrics and tasks within experiment specs can contain strings with placeholders. Iter8 uses variables to substitute placeholders in these strings.
Conditions express aspects of the progress of an experiment. The Completed condition indicates whether or not an experiment has completed. The Failed condition indicates whether or not an experiment completed successfully or in failure. The TargetAcquired condition indicates that an experiment has acquired the target and is now scheduled to run. At any point in time, for any given target, Iter8 ensures that at most one experiment has the conditions TargetAcquired set to True and Completed set to False.
Type of condition. Valid types are TargetAcquired, Completed and Failed.