Frequently asked questions

Straight answers to the questions people ask while they use Testmode: runs, cost, roles, test data and who sees what. Each answer links to the page with the full detail.

That depends on how your app is reached and how people sign in to it. What Testmode can and cannot test answers the common questions:

• Localhost, a VPN or an IP allowlist
• MFA, one-time codes, SSO or Google sign-in
• CAPTCHA, bot protection and firewalls
• Payment forms and embedded iframes
• Cookie consent banners
• Mobile and responsive layouts
• Sites that aren’t in English

No. Once a run starts, it can’t be stopped, and every test in it runs to a verdict with its credits reserved. The status filter on the Runs page lists Canceled, but runs don’t end in that status today.

Check the environment and What to run in the Run tests dialog before you click Run. Explorations are different: you can stop those. See Run tests.

Runs are paid for with your organization’s credits. A run costs one credit per step of each test case in it, with at least one credit per test case, and the credits are reserved when the run starts.

• The Run tests dialog shows the cost before you start, as “Uses N credits · M available”.
• Test cases with no steps and no Expected Outcome are left out of the run and cost nothing. The dialog warns you about them and leaves them out of the cost.
• A test that ends before the agent takes its first action gets its credits back, for example because login failed, or because an error or the time limit stopped it first. Every other test keeps its charge, whether it passed or failed.
• Owners and Admins buy credits on the Billing tab of Organization Settings, which shows the credit packs on sale and their prices.

For an overview of what Testmode costs, see Pricing. For the details, see How credits work and Credits and billing.

Not in the app today. You also can’t leave an organization, transfer ownership or delete an organization. To change a role or remove someone’s access, contact the Testmode team, for example by booking a call.

What you can do: Owners and Admins invite people with the role they need, and resend or cancel pending invitations. See Manage your organization and team.

Partly. You can attach a CSV or Excel file to a message in the Assistant as context for writing test cases. The Assistant receives the column headers, the first 5 rows and the total row count, not every row, so it won’t create one test per row of a large file. Files can have up to 500 rows and 100 columns.

A test case can’t read a file when it runs. For values you reuse across tests, such as a search term or a message, store them as variables and use them in steps as {{Name}}. See Variables.

Every test records a screenshot after each action, the actions themselves (including what the agent typed), the agent’s reasoning, its summary, and the browser’s console and network logs.

Every member of your organization can see these records, Viewers included. Schedule email reports can also go to addresses outside your organization. Passwords stored in credentials are never shown. See What Testmode records.

Common questions

Can I re-run only the failed tests of a run?

No. Re-run repeats the whole run on the same environment: a plan run re-runs the plan as it is now, and any other run re-runs the same test cases. To retry one failed test, start a Single test run for that test case from the Run tests dialog.

Can I start Testmode runs from CI, an API or a deploy hook?

Not today. Runs start from the Testmode app, or automatically from a schedule on a test plan. There is no CI integration, API trigger or deploy hook.

Do the tests in a run run one after another?

No. Each test in a run starts on its own in a fresh cloud browser and runs in parallel with the others when browser capacity allows, in no guaranteed order, even inside a test plan. Browsers are shared fairly between runs; while a test waits, its run page says why, for example how many tests are ahead. Write each test so it does not depend on another test's data. To run tests that share a login one at a time, set that credential's Parallel browser sessions to 1.

Can people outside my organization receive run reports?

Yes. A schedule can email its run reports to up to 20 addresses, including people who are not members of your organization. The link in a report opens only for members. Reports for runs you start yourself go only to you, if you turn them on in your profile.

Can my teammates see the passwords I store in credentials?

No. Credential passwords are stored encrypted and are never shown again after they are saved, to anyone, including the person who saved them. Teammates see a credential's name, username and Login URL. Variable values are plain text that every member can read, so keep passwords out of variables.