Knowledgebase

Thresholds


Background

This article explains Thresholds and how they are utilized within Load Impact Version 4.0 and k6.

Thresholds are used to specify test pass/fail criteria. Thresholds can be set on all metrics including custom ones and you can limit the data point of a metric that are being included in the evaluation by scoping the threshold with tags.

Note: When using Thresholds, you want to stay within the defined values. Thresholds that evaluate to false will result in a failed test.


Try It!

Add the following sample to the options section of one of your scripts (or our in app sample). It will set a threshold at 500 ms for the 95th percentile of all requests. That is, the test will be marked as failed by threshold if the value is exceeded.

1
2
3
thresholds: {
        "http_req_duration": ["p(95)<500"]
},

Threshold expressions

Thresholds can be specified in a short or full format. The short format looks like this:

export let options = {
    thresholds: {
        metric_name: [threshold_expression_string],
    }
};

The full format is covered further down on this page.

A threshold expression is a snippet of JS code that is expected to evaluate to true or false. Every time a threshold is being evaluated k6 injects one or more variables into the JS context. There are four metric types in k6, and each metric type provides its own set of variables that are injected and which can be used in threshold expressions.

Metric type Threshold expression variables
Counter count and rate
Gauge value
Rate rate
Trend avg, min, max, med and p(N) where N is a number between 0.0 and 100.0 meaning the percentile value to look at, eg. p(99.99) means the 99.99th percentile. The unit of these variables and functions are all in milliseconds.

Standard metrics

import { sleep } from "k6";
import http from "k6/http";

export let options = {
    thresholds: {
        "http_req_duration": ["p(95)<500"],
    }
};

export default function() {
    http.get("https://test.loadimpact.com/");
    sleep(3);
};

Custom metrics

Thresholds can also be set on custom metrics. The variable referred to in the threshold expression varies between the different custom metrics as you’ll see below.

Counter metric

To set a threshold on a Counter metric you:

  1. Create the custom metric and give it a name
  2. Specify the threshold condition for the Counter metric by referring to the count variable
  3. Add data points to the custom metric using the add() method
import http from "k6/http";
import { Counter } from "k6/metrics";

let CounterErrors = new Counter("Errors");

export let options = {
    thresholds: {
        "Errors": ["count<100"]
    }
};

export default function() {
    let res = http.get("https://loadimpact.com");
    let contentOK = res.html("h1").text().includes("Load Impact");
    CounterErrors.add(!contentOK);
};

Gauge metric

To set a threshold on a Gauge metric you:

  1. Create the custom metric and give it a name
  2. Specify the threshold condition for the Gauge metric by referring to the value variable
  3. Add data points to the custom metric using the add() method
import http from "k6/http";
import { Gauge } from "k6/metrics";

let GaugeContentSize = new Gauge("ContentSize");

export let options = {
    thresholds: {
        "ContentSize": ["value<4000"],
    }
};

export default function() {
    let res = http.get("https://loadimpact.com");
    GaugeContentSize.add(res.body.length);
};

Rate metric

To set a threshold on a Rate metric you:

  1. Create the custom metric and give it a name
  2. Specify the threshold condition for the Rate metric by referring to the rate variable
  3. Add data points to the custom metric using the add() method
import http from "k6/http";
import { Rate } from "k6/metrics";

let RateContentOK = new Rate("Content OK");

export let options = {
    thresholds: {
        "Content OK": ["rate>0.95"],
    }
};

export default function() {
    let res = http.get("https://loadimpact.com");
    let contentOK = res.html("h1").text().includes("Load Impact");
    RateContentOK.add(contentOK);
};

Trend metric

To set a threshold on a Trend metric you:

  1. Create the custom metric and give it a name
  2. Specify the threshold condition for the Trend metric by referring to one or more of the variables avg, min, max, med and p(N) (0<=N<=100)
  3. Add data points to the custom metric using the add() method
import http from "k6/http";
import { Trend } from "k6/metrics";

let TrendRTT = new Trend("RTT");

export let options = {
    thresholds: {
        "RTT": [
            "p(99)<300",
            "p(70)<250",
            "avg<200",
            "med<150",
            "min<100",
        ],
    }
};

export default function() {
    let res = http.get("https://loadimpact.com");
    TrendRTT.add(res.timings.duration);
};

Aborting a test using thresholds

By default thresholds don’t abort a test before the total duration or iterations have completed. But you can choose to abort the test, if a threshold is hit in the middle of the test:

export let options = {
    thresholds: {
        metric_name: [ { threshold: string, abortOnFail: boolean, delayAbortEval: string }, ... ],
    }
};
Name Type Description
threshold string This is the JS threshold expression string specifying the threshold condition to evaluate.
abortOnFail boolean Whether to abort the test if the threshold is evaluated to false before the test has completed.
delayAbortEval string If you want to delay the evaluation of the threshold for some time, to allow for more metric samples to be collected, you can specify the amount of time to delay using relative time strings like "10s", "1m" and so on.

See the k6 docs on thresholds for more information.

Next: Tags