Troubleshoot failed tests

When a test fails, the first question is whether your app is broken or the test is. The Failure reason, the last screenshots under Test Steps and the test’s console and network logs usually answer it.

1. Open the run from Runs.

2. If the run has several tests, click Failures only, then click the failed test.

3. Read the red Failure reason box at the top.

The reason points you to the right section of this page:

The Failure reason says	Where to go next
An explanation in the agent’s own words	Did the app fail, or the test?
“Login failed: …”	Troubleshoot automatic login
“Session expired or login required”	Troubleshoot automatic login
“Stopped early: …”	The test stopped early
“The agent stopped without reporting a result”, or “The agent declined to continue: …”	The agent stopped without a result
“Timed out: the test did not finish within N minutes …”, or “The agent did not finish within …”	The test ran out of time or turns
“This test has no steps or expected outcome to check. Add at least one step.”	The test had nothing to check
“Test case not found” or “Environment not found”	The test case or environment was deleted after the run was created. Nothing ran, and the test’s credits were returned.
“Test execution error: …”	Run the test again. If the same error keeps coming back, check the site and its environment. If the error came before the agent’s first action, the test’s credits were returned.

For the full list of statuses and reasons, see Run, test and exploration statuses.

When the agent explains the failure in its own words, the test ran and what it saw did not match your Expected Outcome. Check what it saw:

1. Note what the Failure reason says the agent found instead.

2. Under Test Steps, expand the last few actions and look at their screenshots. They show the page as the agent saw it.

3. Compare the last screenshot with your Expected Outcome.

What the screenshots show	Likely cause	What to do
A real problem on the page, such as an error message, missing content or a wrong value	Your app	Report the bug. The test did its job.
The result you expected, worded differently from your Expected Outcome	The test	Quote the exact text the page shows in the Expected Outcome.
The agent on the wrong page, or acting on the wrong element	The test	Quote exact labels, set a Start Path, and split crowded steps.
The page unchanged after a click, as if nothing happened	Either	Check the console and network logs.
A challenge page, an access error or a blank page instead of your app	The environment	See The site blocked the browser.

“The agent stopped without reporting a result”, or a failure reason that is just the agent’s last message, means the agent ended without saying whether the test passed. “The agent declined to continue: …” means it would not carry on.

Most often, the test asks for something the agent can’t settle from the page. To fix it:

• Make the Expected Outcome observable. Describe text or elements on the screen, such as The page shows "Thanks for your message! We will contact you shortly.", not “the message is delivered”.
• Quote exact labels. Write Click "Send", not “submit the form”.
• Remove steps that don’t match the page. A step that refers to a button your app no longer has leaves the agent without a way forward.
• Keep to what a normal user would do on your site. If the agent declined, reword the step it stopped at.

“Stopped early: …” means the agent stopped making progress, so Testmode ended the test before its time limit instead of letting it use up the time. The reason says what the agent was doing, for example that it repeated the same action without effect, that its actions left the page unchanged, or that it made no progress for several turns. Most reasons name the page address it was on.

This usually means the steps don’t match the page the agent was on. To fix it:

• Open the page named in the reason and compare it with your steps. Check that the buttons, links and fields your steps name still exist there, with the same labels.
• Check the flow. If your app now needs an extra click, such as closing a dialog or choosing a tab, add that step.
• Quote exact labels and, if a step starts on the wrong page, set a Start Path.
• Split the test if the reason says the agent used more model capacity than a single run is allowed.
• Run it again if your steps do match the page, for example on a long form. Filling in fields can count as leaving the page unchanged.

For example, Pellbrook Digital’s test “Long Nose Pliers can be added to the cart” stops early on the product page. That page shows the product is out of stock, so “Add to cart” can’t succeed and “Product added to shopping cart.” never appears. The Toolshop is behaving as designed, so the test is what needs to change: pick a product that is in stock, or check the out-of-stock state instead.

A test that stopped early keeps its credits.

Each test has a time limit and a number of turns. The time limit is the test case’s Maximum duration, or the project’s default when the test case has none, which is 15 minutes unless someone changed it under Settings > General. The number of turns grows with the time limit. Reading the page also uses turns, so a long journey can run out before the agent reaches a verdict.

The failure reason then reads “Timed out: the test did not finish within N minutes (X steps completed). Raise the test’s maximum duration in its settings or split it into smaller tests.” or says the agent did not finish within its steps. If the agent recorded at least one action, the credits are kept; otherwise they are returned.

To fix it:

• Raise Maximum duration. With the Editor role or above, open the test case and set Maximum duration (minutes) to a higher whole number, up to 60. To change the limit for every test case without its own value, set Default test duration (minutes) under Settings > General and click Save Changes. See Write a test case.
• Split long journeys. Several short test cases, each with one clear outcome, finish well within the limit and tell you more when one fails.
• Start on the right page. Use Start Path instead of steps that navigate there.
• Let a credential sign in. Attach a Login Credential instead of writing login steps.
• Say what to wait for. Write “Wait until the product list updates” so the agent checks for the change instead of guessing.

“This test has no steps or expected outcome to check. Add at least one step.” means the test case had no steps and no Expected Outcome when the test was about to start. No browser started, and the credits were returned.

Testmode normally leaves such test cases out when a run is created: the Test Cases list marks them No steps, and the Run tests dialog lists them with “(no steps)” and doesn’t charge for them. Open the test case, add at least one step and an Expected Outcome, and run it again. See Write a test case.

When a click seems to do nothing, or the page shows a generic error, the browser logs often show the cause. Scroll below Test Steps to the Console and Network tabs:

• In Console, choose ERROR and look for an error that names the feature the test was using.
• In Network, choose ERRORS and look for a failed request to the part of your app that action touches, such as a cart, login or search endpoint.

The tabs appear a few seconds after the verdict, once the logs are collected. If the logs are clean and the screenshot shows the page working, the problem is more likely in the test’s wording. See Console and network logs.

If a step uses a variable that doesn’t exist in the project, Testmode leaves the placeholder as it is. The agent receives the literal text, such as {{Suport Message}}, and may type the braces into the form.

Signs of this:

• An action under Test Steps such as Typed '{{Suport Message}}'.
• The agent’s reasoning mentions a placeholder or odd text in a step.

To fix it:

• Check the name under Settings > Variables. Names must match exactly, including capitals and spaces, so {{Support Message}} is not the same as {{support message}}.
• Renaming or deleting a variable does not update the test cases that use it. Insert the variable again with Insert Variable.

See Variables.

Every test in a run starts in its own fresh browser, at the same time as the others when capacity allows, in no guaranteed order. That causes two kinds of failures that come and go:

• A test relies on another test. If “Add a product to the cart” expects a product that another test searched for or added, it fails whenever it runs first or alone. Each test must set up what it needs in its own steps.
• Tests change the same account. Tests that sign in with the same credential share whatever your app stores for that account, such as a cart or saved settings. When they run at the same time, one test can see the other’s changes.

Tests that use the same credential also share one sign-in: Testmode signs in once and loads that login into every test with the credential. Some apps allow only one active session per user and sign the other browsers out. Those tests then fail with “Session expired or login required”.

To fix it, make each test independent. When tests need separate accounts, give them different credentials, or set a Credential Override in the test plan. Or set the credential’s Parallel browser sessions to 1 so tests with that login run one at a time. See Credentials and automatic login and How Testmode runs a test.

Some failures come from the site or its setup rather than from your app’s behavior or the test’s wording. Look for these signs in the screenshots and the failure reason:

Sign	Cause	What to do
An error page, a page that never loads, or “Test execution error: …” before any step	The environment isn’t reachable from the public internet, for example localhost or a site behind a VPN	Test a publicly reachable environment, such as a staging site.
A challenge page, “Access denied” or a CAPTCHA	Bot protection, a firewall or CAPTCHA blocks cloud browsers	Allow test traffic and turn CAPTCHA off in the environment you test.
The agent reports it could not open a page on another domain	The flow moves to another domain, such as a hosted payment page	Keep the test on one site, for example with a payment method that stays on your site.
The test needs a code from email or SMS	Tests can’t read email or SMS	Use a test account without MFA or email verification.
The flow opens a new tab, uploads a file or needs drag and drop	Tests can’t do these	Start on the target page with Start Path, or test the flow another way.

See What Testmode can and cannot test.