How Testmode runs a test

When you start a run, Testmode gives each test its own fresh cloud browser. The browser logs in if the test has a credential and opens the test’s start page. An AI agent then works from your steps, within the test’s time limit, and decides whether the Expected Outcome was met. You get a Passed or Failed verdict, with screenshots, reasoning and browser logs as evidence.

1. You start a run
   You pick an environment and what to run in the Run tests dialog, then click Run. Or a schedule starts the run for you.
2. One test execution per test case
   The run holds one for each enabled test case it covers that has steps or an Expected Outcome. Each one then goes through these stages on its own:
   1. Queued
      The test waits as Queued until a browser is free. The run page says why it’s waiting.
   2. A fresh cloud browser
      If a credential applies, the browser loads the credential’s saved login, or signs in first. It then opens the environment’s Base URL plus the test case’s Start Path.
   3. The agent works from your steps
      It reads the page and acts on it until it reaches a verdict or the time limit. Each action is recorded with a screenshot and the agent’s reasoning.
   4. Verdict
      The test ends PASSED or FAILED, with a summary. A test that times out or is stopped early fails.
3. Results
   Each test gets its browser’s console messages and network requests. The run takes its status from its tests, and an email report goes out if you set one up.

Tests don’t wait for each other to finish, apart from the login waits described below.

A run leaves out test cases that have no steps and no Expected Outcome, and they cost nothing. The Run tests dialog tells you how many will be skipped before you start. If nothing is left to run, the run doesn’t start, and a scheduled run is recorded as Skipped.

Each test in a run starts on its own. Tests run in parallel when browsers are free, in no guaranteed order. That includes the tests in a test plan.

Tests share no cookies or data, with one exception. Tests with the same credential can start from that credential’s saved login: the cookies and site storage Testmode saved when the credential last signed in.

So each test has to stand on its own. In the Toolshop project, “Add a product to the cart” can’t rely on “Search finds hammers” having run first. It also can’t rely on anything another test put in the cart.

Tests that use the same credential are signed in as the same user at the same time. If they change the same data, such as a cart or a profile, give them different credentials, or let only one browser use the credential at a time (see Limit parallel sessions).

If a test needs to be signed in, Testmode logs in before the agent starts. You don’t write login steps.

Testmode decides which credential to use when the run is created:

• In a test plan, the test case’s Credential Override, if one is set.
• Otherwise, the test case’s Login Credential.
• If neither is set, the test runs without logging in.

After a credential signs in, Testmode saves the browser’s cookies and the site’s local and session storage for that credential. Later tests and explorations with that credential start with the saved login loaded:

• The saved login still works. The test skips the login form and starts signed in.
• The saved login no longer works. Testmode discards it, signs in again in the same browser and saves the new login.
• No login is saved yet. One test signs in. Other tests with the same credential wait as Queued, without a browser, and then reuse the login it saved.

Only one test signs in with a credential at a time. While another test signs in, a waiting test shows “Waiting for another test to sign in with the same credential” or Waiting for the login on credential "Jane Doe – demo customer".

For example, “Sign-in lands on My account” uses the “Jane Doe – demo customer” credential. The browser signs in to the Toolshop first, and the agent starts on /account already signed in.

If you change a credential’s username or password, Testmode clears its saved login, so the next test signs in with the new details. Changing only the name or the Login URL keeps the saved login.

If the login fails, the test ends as Failed with a reason that starts with “Login failed:”. The username and password are removed from that message. A login that takes too long fails the same way. The test isn’t retried, and because it never ran, its credits are returned. See How credits work.

If a test that started from a saved login still reaches a login page, it fails with “Session expired or login required”. Testmode then discards the saved login, so the next test signs in fresh.

Some applications allow only one active session per user and sign out the other browsers. For those, set the credential’s Parallel browser sessions (optional) to 1. Tests with that credential then run one at a time.

Tests over the limit wait as Queued with Login limit for credential "…" reached and start when a browser using the credential finishes. Running explorations with the credential count toward the limit. Leave the field empty for no limit. See Credentials and automatic login.

The start URL is the environment’s Base URL followed by the test case’s Start Path. With no Start Path, the test starts on the Base URL itself.

Environment Base URL	Start Path	Start URL
https://practicesoftwaretesting.com	/	https://practicesoftwaretesting.com/
https://practicesoftwaretesting.com	/contact	https://practicesoftwaretesting.com/contact

You choose the environment each time you run. The same test case can run against Production or Staging without changes.

When the test starts, Testmode fills in project variables in the steps, the Expected Outcome and the Start Path. For example, if {{Search Term}} is set to hammer, the agent sees hammer wherever your step uses that variable. Steps that are empty after variables are filled in are left out.

The agent gets the test name, all your steps and the Expected Outcome together, as one task. It doesn’t run your steps one at a time, and it doesn’t pass or fail them one by one. It reads the page, decides what to do next, and keeps going until it can give a verdict.

The agent can:

• click, fill in fields, type, press keys and choose options from dropdowns
• hover, scroll up and down, go back, and wait for content to load
• open other pages on the same site

It can only open pages directly on the start URL’s host and its subdomains. A test that starts on practicesoftwaretesting.com can’t navigate straight to another domain. The agent is also told to stay on the site and to fail the test if the task can’t be done there.

Each action the agent takes appears under Test Steps as soon as it’s done, with a screenshot of the page and the agent’s reasoning. These entries are the agent’s actions, not the steps you wrote. One of your steps often becomes several actions. Reading the page doesn’t add an entry.

If the agent reaches its verdict without taking any action, for example when it only checks what the start page shows, Testmode records one Observed the page entry with a screenshot and the agent’s reasoning.

After the verdict, Testmode attaches the browser’s console messages and network requests. They appear a few seconds after the verdict.

For everything Testmode stores and who can see it, see What Testmode records.

A test passes only when the agent reports that the Expected Outcome was met. Otherwise it fails, and the Failure reason explains why:

Failure reason	What happened
The agent’s own summary	The agent reached a verdict and the Expected Outcome wasn’t met.
“Login failed: …”	Automatic login didn’t succeed.
“Session expired or login required”	A test with a credential still reached a login page.
“Timed out: the test did not finish within N minutes (X steps completed). …”	The test reached its time limit before the agent gave a verdict.
“Stopped early: … Check that the steps match what this page offers.”	The agent stopped making progress, so Testmode ended the test before its time limit. “Long Nose Pliers can be added to the cart” fails this way: the product is out of stock, so trying to add it leaves the page unchanged.
“The agent did not finish within N steps”	The agent used all its turns before it gave a verdict.
“The agent stopped without reporting a result”	The agent ended without giving a verdict.
“The agent declined to continue: …”	The agent would not carry on with the test.
“Test execution error: …”	Something else went wrong while the test ran, for example the start page didn’t load.
“Test case not found” or “Environment not found”	The test case or environment was deleted before the test started.
“This test has no steps or expected outcome to check. Add at least one step.”	Nothing was left to check once variables were filled in.
“Test failed after N attempts”	Testmode couldn’t start the test, even after retrying it automatically.

For what to change in each case, see Troubleshoot failed tests.

Each test has a time limit. It starts when the browser opens the start page, after any login, and runs until the verdict. The limit is, in order:

1. The test case’s Maximum duration (minutes), if you set one.
2. Otherwise, the project’s Default test duration (minutes) under Settings > General.
3. Otherwise, 15 minutes.

Both fields take a whole number from 1 to 60. A test that reaches its limit fails with “Timed out: the test did not finish within N minutes (X steps completed). Raise the test’s maximum duration in its settings or split it into smaller tests.” See Maximum duration.

The agent also has a limited number of turns, and reading the page uses turns too. A longer time limit gives it more turns, but a long journey can still run out before it ends.

Testmode doesn’t wait for the time limit when a test is going nowhere. It ends the test early when the agent keeps repeating an action, acts without the page changing, keeps waiting on a page that doesn’t change, goes several turns without acting, or uses more model capacity than one run allows. The Failure reason then starts with “Stopped early:” and says what happened, usually with the page’s address, for example “The agent repeated ‘…’ N times without effect on …”. This usually means a step doesn’t match the page, such as a button label that changed.

Credits depend on whether the agent acted. A test that hits an error or runs out of time before the agent’s first action gets its credits back. Once the agent has recorded an action, or an Observed the page entry, the test stays charged. That includes tests that stopped early.

When no browser is free, a test waits as Queued. Browsers are shared fairly: organizations take turns, and within an organization its runs take turns. A small run isn’t stuck behind a large one, so the tests of one run can start in between the tests of other runs.

While a test waits, the run page shows why, above the results. The position in line refreshes every 10 seconds. On a run with several tests, each queued test’s card also shows the reason under the test’s name when Testmode knows it.

For example, “Waiting for a browser (N ahead)” means N tests are ahead of this one. Every message and what it means is listed in Why a test is waiting.

A test can return to Queued after it has started: when no browser turns out to be free, when another test is signing in with the same credential, or when Testmode restarts (its live view then closes). It starts again from the beginning without using up a retry.

If a test crashes before it gets going, for example while its browser is being set up, Testmode retries it automatically. Either way, the earlier attempt’s actions are removed, so you only see one attempt.

A verdict is final. A test that ran and failed isn’t retried.

• Keep tests independent. Don’t rely on another test’s data or on the order tests run in.
• Attach a credential instead of writing login steps. A test with a credential fails if it reaches a login page.
• Limit the credential if your app allows one session per user. Set Parallel browser sessions (optional) to 1.
• Use Start Path to begin on the right page instead of spending turns getting there.
• Name the exact text you expect in steps and the Expected Outcome, for example the message “Product added to shopping cart.”
• Keep each test short. Split long journeys into several test cases, or raise Maximum duration (minutes) when a test needs more time.
• Run tests that create real data, such as orders or messages, only against environments meant for testing.