Run, test and exploration statuses

Every test in a Testmode run is Queued, Running, Passed or Failed. A run takes its status from its tests, so it is Passed only when every test passed. A scheduled run that cannot start, because the organization cannot pay for it or none of its test cases has anything to check, is recorded as Skipped.

A test execution is one test case run once, in its own browser. It moves through these statuses:

Status	What it means
Queued	The test is waiting to start: for a free browser, for another test to sign in with the same credential, or for a free session on its credential.
Running	Testmode is opening the browser, signing in with the test’s credential, or the agent is working through the test.
Passed	The agent reported that the Expected Outcome was met.
Failed	Anything else. The Failure reason says why; see Common failure reasons.

Good to know:

• While a test waits, the run page says why. See Why a test is waiting. The Live Browser View appears once the browser is ready.
• A test can stay in Queued, or go back to it for a while: when no browser is free, while another test signs in with the same credential, when its credential’s session limit is reached, or when Testmode restarts. It then starts again from the beginning on its own, and the run page shows only the steps of the attempt that finishes.
• Tests have no Skipped or Canceled status. Every test that starts ends as Passed or Failed.

While a test is Queued, the run page shows one line that explains the wait. On a run with several tests, each queued test’s card also shows its reason under the test’s name.

Message	What it means
“Next in line for a browser”	The test starts as soon as a browser is free.
“Waiting for a browser (N ahead)”	N tests are ahead of it. The number refreshes on its own.
“Waiting for a browser”	Testmode tried to start the test, but no browser was free. It tries again shortly.
“Waiting for another test to sign in with the same credential” or Waiting for the login on credential "…"	Another test is signing in with this test’s credential. This test starts once that login is saved, and then reuses it.
Login limit for credential "…" reached	The credential’s Parallel browser sessions limit is in use by other tests or explorations. The test starts when one of them finishes. See Credentials and automatic login.
“Waiting for browser session…”	The test’s browser is starting, or Testmode has nothing more specific to show yet.

Browsers are shared fairly: organizations take turns, and so do the runs within one organization. The tests of a large run can therefore start in between the tests of other runs.

A run’s status is recalculated every time one of its tests changes status. Testmode applies these rules in order:

Run status	When
Running	At least one test is running, or some tests are still queued while others have finished.
Queued	Every test is still queued.
Failed	No test is running or queued, and at least one test failed.
Passed	Every test passed.
Skipped	A scheduled run that could not start. It has no tests; see below.

So a run can still show Running while some of its tests already show Failed. The run’s final status settles when its last test finishes.

In Pellbrook Digital’s Checkout regression run, “Long Nose Pliers can be added to the cart” fails because the product is out of stock, so Testmode stops it early on the product page. The other tests pass, and the run ends Failed.

The Required switch on a test case in a test plan does not change this. Any failed test makes the run Failed, including one marked Optional.

When a schedule fires and the run cannot start, Testmode records a run with status Skipped instead of starting any tests. That happens in two cases:

Case	Reason shown
The organization does not have enough credits for the run.	Starts with “Insufficient credits:” and says how many credits the run needed and how many the organization had.
None of the plan’s enabled test cases has steps or an Expected Outcome.	"…" has no steps or expected outcome to check. Add at least one step., or “None of the N test cases has steps or an expected outcome to check. Add at least one step.”

The run page shows “This run was skipped”, the reason, and a Manage credits link.

• A skipped run is not charged and sends no email report.
• It is not retried. The schedule’s next time tries again as normal.
• To find skipped runs, set the status filter on the Runs page to Skipped.

Some schedule times leave no run at all: when the schedule’s previous run is still Queued or Running, or when the plan has no enabled test cases. Runs you start yourself are never skipped. If the balance is too low, or nothing you chose has steps or an Expected Outcome, the Run tests dialog disables Run and no run is created.

When only some test cases in a run have no steps and no Expected Outcome, Testmode leaves those out, runs the rest and charges nothing for the ones it left out.

See How credits work and Schedule test plans.

Canceled appears in the status filter on the Runs page, but no run ends in it today. Runs and tests cannot be canceled: once a run starts, every test in it runs to a verdict. See Run tests.

Explorations are different. You can stop an exploration, and it then ends as Canceled.

Several pages summarize the latest result instead of showing every run:

Where	What it shows
Test Cases, Last Run column	The status of the test case’s most recent test execution and how long ago it started. It shows queued instead of a time when that test has not started yet, and Never when the test case has never run.
Test Plans, Last Run column	The status of the plan’s most recent run and how long ago it started, or Never.
Runs page, Status column	Each run’s status. A run with more than one test also shows how many of its tests have finished out of the total.
Overview, Recent runs	The status of each of the latest runs.

Each row also has a colored left edge that matches its status.

On Overview, the Pass rate · last 7 days card counts every test that has not passed as failed, including tests still queued or running. While runs are in progress, its failed count reads higher than the final result.

An exploration has its own statuses, shown on the Explore page and at the top of each exploration:

Status	What it means
Queued	Waiting for a browser. The heading reads “Waiting for a browser”.
Exploring	The agent is visiting pages. The heading reads “Exploring the application”.
Probing	In Interact mode only, the agent is trying dialogs, filters and form validation. The heading reads “Trying interactions”.
Analysing	The agent is comparing the pages it visited with your test suite and writing proposals. The heading reads “Analysing coverage”.
Ready for review	Proposals are ready for you to accept or reject.
Applied	Accepted proposals were written to the project. You can still accept and apply more.
Failed	The exploration could not finish. The heading reads “Exploration failed”, and a What went wrong box says why, for example “Login failed: …”.
Canceled	Someone stopped the exploration. It has no proposals.

A project can have only one exploration that is Queued, Exploring, Probing or Analysing at a time. If Testmode restarts while an exploration is in progress, the exploration goes back to Queued and starts again. See Start an exploration.

The Failure reason on a failed test is either the agent’s own explanation or a system reason:

Failure reason	What it means	Credits
The agent’s own summary	The test ran, and what the agent saw did not match the Expected Outcome.	Kept
“Login failed: …”	Automatic login did not succeed, so no steps ran.	Returned
“Session expired or login required”	A test with a credential reached a login page during the test.	Kept
“Stopped early: …”	The agent kept repeating itself or the page stopped changing, so Testmode ended the test before its time limit.	Kept
“The agent stopped without reporting a result”, or the agent’s last message	The agent ended without giving a verdict.	Kept
“The agent declined to continue: …”	The agent would not carry on with the test.	Kept
“The agent did not finish within …”	The agent used all the turns it has for the test before reaching a verdict.	Kept
“Timed out: the test did not finish within N minutes (X steps completed). …”	The test reached its time limit before a verdict: the test case’s Maximum duration, or the project’s default.	Returned if no action was recorded, otherwise kept
“Test execution error: …”	The browser or the agent hit an error during the test.	Returned if no action was recorded, otherwise kept
“This test has no steps or expected outcome to check. Add at least one step.”	When the test was about to start, it had no steps or Expected Outcome to check. No browser started.	Returned
“Test case not found” or “Environment not found”	The test case or environment was deleted after the run was created, before the test started.	Returned
“Test failed after … attempts”	Testmode could not set the test up, even after retrying automatically.	Returned only if no action was recorded

A test that ends without an error or a timeout but took no action, such as a check that only reads the page, still records an “Observed the page” step, so it keeps its charge.

Returned credits appear in the billing history as “Refund (test did not run)”, linked to the run. The test still shows Failed.

For how to act on each reason, see Troubleshoot failed tests and Troubleshoot automatic login.