How credits are counted
A credit is one action a test runs: a click, a typed value, a check. How a run reserves credits and settles them, what tests give back, what explorations, reviews, investigations and the assistant cost, and what happens when credits run out.
Everything in Testmode is paid for in credits from your organization’s plan. See Plans and billing for how many each plan includes.
What uses credits
Section titled “What uses credits”| What | Credits |
|---|---|
| An action a test runs | 1 |
| A message to the assistant | 5 |
| A page an exploration reads | 10 |
| An investigation of a failed run | 50 |
| A Self-driving review | 100 |
A typical test runs about ten actions, which is how the app turns credits into “about N test runs”.
Actions
Section titled “Actions”The first time a test runs, Testmode turns each of its steps into browser operations: a click, a typed value, a wait, a check, opening a page. Each operation is an action, and so is the final check of the Expected outcome. Every action costs one credit; the time a test takes doesn’t count.
Here is a real run of Pellbrook Digital’s Checkout regression plan on its production environment:
| Test case | Steps | Actions, with the final check | Credits |
|---|---|---|---|
| Add a product to the cart | 4 | 6 | 6 |
| Long Nose Pliers can be added to the cart (failed) | 3 | 5 | 5 |
| Sign-in lands on My account | 1 | 2 | 2 |
| Checkout with cash on delivery (changes data, left out on production) | 7 | 0 | 0 |
| The whole run | 13 |
The step count says little: one step such as “Search for hammer and open the first result” is several actions, and the expected-outcome check is one more. A test’s summary counts the actions its steps took (“4 steps (5 actions)”), and its run page lists them.
When credits are reserved
Section titled “When credits are reserved”- When the run is created, it reserves each test’s estimate: one credit for every action its steps compiled to, one for a step not compiled yet, and one for the expected outcome. When there aren’t enough credits for the whole run, no run is created: the Run tests dialog says so and disables Run, and the API answers with an error.
- Before a test’s browser opens, its reservation is topped up to every action its steps compiled to, since a step can compile into several. A test that would need more credits than are left doesn’t start, and gives back what it reserved.
- When the test ends, it pays for the actions it ran. Your History on the Billing tab shows a run’s reservation as Used, and anything above the estimate as Actions beyond the estimate.
Scheduled runs work the same way: a schedule that fires without enough credits records a Skipped run with the reason, rather than running half the plan. See Schedule test plans.
What tests give back
Section titled “What tests give back”| The test… | Pays? |
|---|---|
| Passed | The actions it ran |
| Failed on a step, on the expected outcome, or by running out of time | The actions it ran, up to the failure |
| Failed because the page stopped responding | The actions it ran: that is the site failing |
| Failed at login, before any step ran | Nothing |
| Never started: the site could not be reached, or its test case or environment was deleted first | Nothing |
| Ended on Testmode’s side: our AI model or browser failed | Nothing |
| Was skipped: its test case was switched off while it waited, or the run was canceled first | Nothing |
| Was left out because it changes data on production | Nothing |
A test that gave its credits back says Not charged: no credits used on its run page, and the History shows Returned (did not run).
Explorations, reviews, investigations and the assistant
Section titled “Explorations, reviews, investigations and the assistant”- An exploration costs 10 credits for each page it read, charged when it ends. Its exploratory tests and analysis are included. A stopped exploration pays for the pages it read; a failed one pays nothing.
- A Self-driving review costs 100 credits when it completes. The trial runs it makes of new and changed tests pay for their actions like any test.
- An investigation of a failed run costs 50 credits when it completes. Its re-runs and draft runs pay for their actions like any test.
- The assistant costs 5 credits a message, charged before it answers. Only Editors and above can send it messages, so read-only members spend nothing.
Self-driving’s levels keep its explorations and reviews within a known cost a month, and the schedules it plans may use at most 70% of your organization’s monthly credits. See Turn on Self-driving.
When you run out
Section titled “When you run out”The plan’s name next to the organization turns amber when few credits are left, and Owners and Admins are emailed at 80% and 100% of a period’s credits.
- On Pro and Startup, pay as you go keeps everything running past the plan’s credits, up to your spending cap.
- On Free, during a trial, or with pay as you go off, runs, explorations, reviews and the assistant stop until the credits renew. The Run tests dialog explains why and disables Run. Owners and Admins get a link to the Billing tab; Editors are told to ask an organization admin.
Credits come back with the next billing period, or the next month on Free. Unused ones don’t roll over.
Common questions
Section titled “Common questions”Does a longer test cost more?
Section titled “Does a longer test cost more?”A test with more actions does: each click, typed value, wait or check is one credit. Time doesn’t count, so a slow page costs nothing extra.
Do I pay for a test that failed?
Section titled “Do I pay for a test that failed?”You pay for the actions it ran up to the failure, since finding what broke is the point. A test that never reached a verdict, such as one whose login failed or that failed on Testmode’s side, gives everything back.