Insights test comparison
The test comparison feature built-in to Insights allows you to compare the results of two different test runs of the same test. You can compare high-level metrics and individual checks and URL endpoints. You may wish to compare against a previous test run to look for differences in results. Or you may be comparing against a known baseline.
Setting a baseline test
Comparing results against a known baseline is a core part of the General Methodology for performance testing. Baseline tests are important as they allow you to compare against a control to look for differences. Baseline tests should produce enough load to contain meaningful data and ideal results. In other words, a heavy stress test isn’t a good baseline. Think much smaller.
In order to set your baseline, open up the results for the test run you wish to be your baseline. Click on the three dots in the top right corner -> set as Baseline test run. Baseline tests are exempt from data retention policies.
Selecting test run to compare against
To compare two test runs, open up one of the test runs. Then select the test run you want to compare it to using the select widget on the right side of the test result page, just above the Performance overview section. Note that you can only compare against other test runs of the same test.
Performance overview comparison
When in test comparison mode the performance overview section will show the overview chart for both test runs. This gives you a quick way to see if there are any performance differences on a high level.
Thresholds tab comparison
In the thresholds tab you’ll see an additional “Diff” column being added to the table. This column will show the current vs compared test run’s pass/fail status for each threshold. You’ll also see a separate threshold charts for each test run.
Checks tab comparison
In the checks tab you’ll see an additional “Difference in failure %” column being added to the table. This column will show the difference in failure rate between the current and compared test runs. You’ll also see a separate checks charts for each test run.
HTTP tab comparison
As in the Thresholds and Checks tab, you’ll see a new column in the table. This time the you’ll have a select box in the title where you can select based on what statistic you want to compare. It defaults to 95th percentile, but you also have the option to select min, avg, 99th percentile and max.
When you’ve made a selection you’ll also notice that the content of the corresponding statistic column will change to show the current vs compared test run value for that statistic. In the screenshot below you’ll see it in the 95th percentile column.
Clicking on a row will also show two separate charts, one for each test run.
Analysis tab comparison
When you use the “Add this graph to analysis tab” action in the other tabs, two charts will be added to the analysis panel, one for each test run. Same goes if you add a metric via the “Add metric to visualize” button, you’ll get two charts.
At this point metrics from the compared test run can't be added to the comparison chart.