Skip to content
Get Free Trial

Suite Analysis

Suite Analysis shows which steps caused the failures and healed steps in a suite run, which scenario files they affected, and how each file has done across earlier runs.

When a suite fails, a lot of time can go into going through the results to find the real problem. Often many failures come from very few steps: when you know that 23 failing scenario files all fail because of the same 3 steps, fixing and rerunning is quick. Suite Analysis finds those steps for you, and shows how each scenario file has done in earlier runs.

Open the suite report of a run, and click the Suite Analysis tab:

The Suite Analysis tab of the book_store_nightly.dd.csv suite report: Suite Details, the Test Scenario Summary with 7 in total, 5 passed, 1 error and 1 aborted timeout, and the start of the Failed Script History table

It starts with the Suite Details and the Test Scenario Summary, then has three parts: the Failed Script History, the Passed Scripts With Failure History, and the Failure & Healed Report.

The Failure & Healed Report lists each step that failed or was healed in this run, and how many scenario files it affected:

The Failure & Healed Report with Failure Report and Healed Report ticked: the step _click(_link("Forgot Password?")) Caused 1 Failures in 1 Scripts, with View Impact Tree outlined; the Impact Tree below goes from _click(_link("Forgot Password?")) at line 66 to OpenForgotPasswordPage at line 60 to login_errors.s.csv at line 8, and Affected Script Reports lists login_errors.s.csv
  • Step: the step that failed, such as _click(_link("Forgot Password?")). Click the arrow next to it to show or hide its impact tree.
  • Impact: how many failures the step caused, in how many scenario files, such as “Caused 1 Failures in 1 Scripts”.
  • View Impact Tree shows the path from the step to each scenario file it broke: the step and its line, the keywords or functions that called it, and the scenario files. A step inside a keyword used by many scenario files shows a branch for each one, so you can see that fixing this one step fixes all of them.
  • Affected Script Reports lists the scenario files that failed because of the step. Click one to open its script report.

Healed steps are listed the same way, as Autoheals, such as “Caused 1 Autoheals in 1 Scripts”. Each one is a step to update in your tests, before the page changes again:

The Failure & Healed Report for cart_button.s.csv: the step _click(_button("Add")) Caused 1 Autoheals in 1 Scripts, with View Impact Tree

Untick Failure Report or Healed Report at the top right to hide failures or healed steps.

Is a scenario file failing for the first time, or has it been failing for a while? The Failed Script History answers this for each file that failed in this run. Passed Scripts With Failure History does the same for files that passed this time but failed before, which can point to tests that pass and fail at random:

Failed Script History with slow_search.s.csv and login_errors.s.csv at 0% pass within the same suite and across all suites, with a row of red status icons for each; and Passed Scripts With Failure History with Register.s.csv (100% in the suite, 89.5% across all suites), books.s.csv (80% and 72.7%) and book_orders.s.csv (20% and 20%), with green and red status icons from recent to old

Each table shows the history twice:

  1. Within the same Suite: the runs of the file in this suite only.
  2. Across all Suites: every run of the file, in any suite.

For each, Pass % is the share of those runs that passed, and Status (Recent to Old) shows a status icon for each run, starting with the latest. A small number next to an icon means the file was retried in that run: it shows how many times it ran. For example, if a file fails, is retried once with Retry Count and fails again, its icon shows 2.

To change how many runs the tables show, set this in userdata/config/userdata.properties. The default is 10:

sahi.count.script_status_to_be_shown=10