How Testmode runs a test

A Testmode test is your plain-English steps and Expected Outcome, carried out in a real browser in the cloud. The first time a step runs, Testmode compiles it into a short list of browser operations and saves them on the step. Every later run replays those operations, so a test does the same thing each time until you change its wording.

1. Queued
   The run is created with one test per test case, credits are reserved, and each test waits for a browser.
2. Running
   Each test gets its own browser and works through these stages:
   1. Open the site
      A fresh cloud browser opens the environment’s URL, or its Start path.
   2. Log in
      When the test has a login credential, Testmode signs in first, or reuses a saved login.
   3. Compile
      Steps that are new or reworded are turned into browser operations, once.
   4. Replay
      The operations run in order, each one checked and recorded on the step timeline.
   5. Check
      Testmode reads the page and decides whether the Expected Outcome holds.
3. Passed or Failed
   The run rolls up its tests’ verdicts. Report emails go out, failures file findings, and Self-driving investigates a failed scheduled, CI or plan run.

When you start a run, Testmode creates it with one test per enabled test case, reserves the credits its tests are expected to use, and every test waits in a queue for a browser. Tests of the same run execute at the same time, each in its own browser, as browsers become free. On the Free plan, an organization runs one test at a time.

A test case marked Changes data is left out of a run on a production environment that does not allow such tests. See Pause testing and protect data.

While a test waits, its card says why:

Line on the card	Meaning
Waiting for a browser (N ahead)	Other tests are first in line.
Next in line for a browser	A browser is being prepared.
Waiting for the login on credential “…”	Tests that share a login wait while one of them signs in, then reuse that login. See Credentials and automatic login.
Login limit for credential “…” reached	The credential allows only so many browsers at once, and they are all in use.
One test at a time on the Free plan	Another test of the organization is running.
Retrying: …	Testmode’s AI model or browser failed on our side, and the test is tried again.

Each test gets a fresh browser in the cloud, at a desktop window size, with no cookies or history from earlier tests. It opens the environment’s URL, or the test case’s Start path appended to it. The browser reaches your site over the public internet, so your app must be reachable from there. See What Testmode can and cannot test.

The whole session is recorded. Once the test has finished, the run page plays it back as the Session replay. While it runs, editors can watch the live browser on the run page.

A test with a login credential starts signed in. Before step 1, Testmode opens the credential’s Login URL (or the environment’s URL), finds the login form, types the username and password and submits it, then checks that the login worked. Signing in is never one of your steps.

Tests that use the same credential share one login: the first test signs in and Testmode saves the session, and the other tests start with it, without a login of their own. When the login fails, the test fails with “Login failed: …” before any step runs, and it is not charged. See Troubleshoot login.

The first time a test case runs, Testmode turns each step into a few operations, the smallest things a browser test does:

Operation	What it does	Example from a step
Act	One action on one element: click, type, select, check, hover, scroll	Type “hammer” into the “Search” field.
Wait for	Waits up to 20 seconds for something to appear	Wait for the heading “Searched for: hammer”.
Assert	Checks that a statement about the page is true	Only “Buy milk” is listed.
Go to	Opens a path or URL, when the step says to	Go to /contact.
Reload	Reloads the page, when the step says to	Reload the page.
Press	Presses a key, when the step says to	Press Enter.
Wait	Waits a fixed number of seconds, when the step says to	Wait 3 seconds.

A step like Type "hammer" into the "Search" field and click "Search". becomes two operations: a type and a click. Compound steps are split, and nothing is added that the step doesn’t describe.

The operations are saved on the step. Every later run replays them as they are, which is why a test behaves the same way run after run. Rewording a step recompiles that step on its next run; the steps you didn’t touch keep their operations. Changing a variable’s value never recompiles anything, because operations keep {{variables}} as placeholders and fill them in when they run.

Each operation is one action, and so is the final check of the Expected Outcome: a run uses one credit per action. See How credits are counted.

Testmode runs the operations in order. For each action, it finds the element your words describe on the page as it is now and acts on it, so a button that moved still gets clicked. Then it checks what the action did:

• A typed value is read back from the field.
• A click that landed on the row, card or form around a control is done again on the control itself.
• A click that opened a confirmation dialog, such as “Delete” beside “Cancel”, confirms it, unless the next operation is about the dialog.

Each operation is saved to the step timeline as it completes, with its outcome and the page it ran on.

The first operation that fails ends the test, and the verdict says which step and operation stopped it, for example “Failed at step 6 (Click the “Confirm” button.)“. An action that finds no matching element, a Wait for that times out after 20 seconds and an Assert that is false all fail the test. Testmode takes a screenshot of the page at that moment, shown on the failed step.

An Assert and the Expected Outcome are checked on the page as it is when they run, after giving it a moment to finish loading. Testmode splits the statement into its parts, finds each part on the page and quotes what the page shows; the statement holds only when every part does. The Expected Outcome is also told what the asserts before it found, so it can compare with them: “the cart count is one more than in step 2”.

When the Expected Outcome holds, the test passes with a summary such as “Passed. 2 steps (3 actions) completed and the expected outcome was verified.” When it doesn’t, the test fails with “Expected outcome not met”, your Expected Outcome, and what Testmode observed instead.

Every test has a maximum duration: the test case’s own Maximum duration (minutes), or else the project’s Default test duration, which is 15 minutes unless changed in project settings. A test that runs longer is stopped and fails. The longest a test may run is 54 minutes.

Outcome	Test status	Charged
Every step ran and the Expected Outcome holds	Passed	The actions it ran
A step failed, the Expected Outcome didn’t hold, or the time ran out	Failed	The actions it ran
Login failed, or the site could not be reached at all, before the first step	Failed	Nothing
Testmode’s AI model or browser failed on our side	Retried; if it keeps failing, Failed	Nothing
The test case was switched off while the test waited	Skipped	Nothing
The run was canceled before the test started	Canceled	Nothing

A failure on our side is never a verdict about your app: it shows “Not charged: no credits used” on the run page.

The run’s status rolls up from its tests: Passed when every test passed, Failed when any failed. Then:

• Report emails go to the people who asked for them. See Email run reports.
• Findings are filed for the tests that failed for a reason of your app. See Findings.
• Self-driving investigates a failed run that a schedule, CI or a test plan started, when it is on. See How Self-driving investigates failed runs.
• The session replay of each test is ready on the run page. Browserbase, which hosts the browsers, keeps recordings for 31 days.
• Console and network logs are on each test’s page. See Console and network logs.