What is AI testing?

AI testing is the use of artificial intelligence to create, run and maintain software tests. In its most common form today, you describe what should happen in plain English — “log in, add a laptop to the basket, and check the total updates” — and an AI carries that out in a real browser, deciding for itself which elements to click and what a correct result looks like.

That is the short version. The term is used loosely enough that it is worth being precise about what falls inside it.

Two different things are called “AI testing”

The phrase means two almost opposite things, and search results mix them constantly:

• Using AI to test software. The subject of this page. AI writes, runs or maintains tests against an application.
• Testing software that contains AI. A different discipline: evaluating model outputs, checking for hallucination, measuring quality of generated text. This is usually called evaluation or evals.

They share almost no tooling. If you arrived here looking for the second, the tools below will not help you.

What the AI actually does

“AI-powered” is claimed by nearly every testing product now, and it covers at least five separate jobs. Knowing which ones a tool actually does is most of what a comparison is worth.

Authoring. Turning an intention into a test. This is the largest change: instead of a developer writing a script with explicit selectors, someone writes a sentence. It is the reason non-developers can now create tests at all.

Suggesting coverage. Some tools examine an application and propose what is worth testing. Useful when you are starting from nothing and do not know where to begin.

Locating elements. Traditional automation targets an element by a selector — a CSS path or an ID — which breaks when the markup changes. AI approaches identify elements by what they are: the button that submits the form, the field labelled “email”.

Maintenance and self-healing. When the interface changes, something has to update the test. Self-healing repairs a broken locator automatically. Tools that work from a description have less to repair, because the description was never tied to the markup in the first place.

Triage. Reading a failure and saying what probably caused it, rather than handing you a stack trace and a screenshot.

How it differs from traditional test automation

Traditional automation is not going away, and it is not inferior. It is a different trade.

The honest summary of that table: code gives you determinism and demands engineering time. AI gives you speed and access, and asks you to accept that something else is deciding how to carry out your intention.

For a concrete version of this trade-off, see Testmode vs Playwright — Playwright being the strongest example of the code-based approach.

Where AI testing fits in software development

Testing is a feedback loop throughout development. A useful setup starts with an agreed outcome, checks the change before release, and brings failures back to the people who can fix them. AI can help execute and investigate those checks; the team still defines correct behaviour and decides what to ship.

Use each layer for the question it answers best. The pyramid shows relative scope, not a required ratio of tests.

Before a pull request is merged, focus on the feature being changed and the journeys it could affect. A checkout change might need checks for discounts, shipping and payment failures. Selecting tests by risk can shorten feedback time, but a small passing selection cannot prove that unrelated flows still work.

Before a release, run broader regression coverage against a known build, with predictable accounts and test data. Browser tests complement the faster unit and integration tests that check calculations, services and contracts. A passing browser journey alone does not establish security, accessibility or performance across the application.

After deployment, a small set of controlled smoke checks can catch issues with configuration or dependencies. Use dedicated test accounts and safe test transactions; a production checkout check should not create real customer orders. Investigate failures and feed confirmed bugs back into the next development cycle.

End-to-end, regression and exploratory testing

These terms answer different questions. End-to-end describes the scope of a test: a complete journey through connected parts of the application. Regression describes its purpose: checking that a change has not broken existing behaviour. Exploratory describes an approach: learning about the application and choosing the next check based on what you find.

One test can be both end-to-end and regression coverage. Exploration discovers paths to add to the saved suite.

An end-to-end checkout test can also be part of a regression suite. Regression coverage can include unit and API tests too; it does not require a browser. Exploration helps discover cases that the saved suite does not yet cover. The terminology follows the distinctions in the ISTQB testing glossary.

AI can translate a journey into actions, rerun described checks, or suggest new paths to explore. The expected result still needs an independent basis: an acceptance criterion, a business rule or a known calculation. An agent reaching a confirmation page is insufficient if the order total is wrong. This focus on observable behaviour also applies to conventional automation; see Playwright’s testing guidance.

From a failed test to a verified fix

A red result is the start of an investigation. It might indicate a product bug, a stale expectation, unavailable test data or an environment problem. Useful AI testing produces enough evidence to tell those cases apart.

Classify the cause before changing the test. If it remains unexplained or fails again, return to investigation; preserve the original evidence.

For example, suppose a voucher should reduce a $100 basket by $10, but checkout still shows $100. Capture the voucher, basket state and observed total. After fixing the discount calculation, verify the $90 result and check the no-voucher case as well. Save the failing scenario so the same defect is easier to catch next time.

Self-healing should repair how a test finds an element, not silently redefine what success means. If a required discount disappears, changing the assertion to accept the full price hides the bug. Review changes to expectations just as you would review changes to product requirements.

Where AI testing works well

• Regression testing of stable business flows. Checkout, sign-up, search, permissions. High value, tedious to check manually, easy to describe.
• Applications nobody on the team can fully explain. Software generated by AI coding tools, assembled on a low-code platform, or delivered by an outside supplier. You cannot write unit tests for a codebase you do not understand, but you can still describe what it is supposed to do.
• Interfaces that change constantly. A description survives a redesign that would break every selector in a scripted suite.
• Teams with no QA function. This is the largest practical shift. Where the alternative is a person clicking through the site before each release — or nobody checking at all — the comparison is not with a better tool.

Where it still fails

Worth stating plainly, because vendors rarely do:

• Ambiguous expectations. “Check the page looks right” is not a test. If two people would read your sentence differently, so will the AI.
• Pixel-level and canvas interaction. Drag-and-drop, drawing tools, map interactions and games are usually easier to demonstrate by recording than to describe.
• Anything needing an exact assertion. If the requirement is that a value is precisely 1,247.50, say so. Descriptions invite approximation.
• Non-determinism. An AI deciding how to carry out a step can, occasionally, decide differently. Code-based tools are more repeatable, which is exactly why regulated environments still prefer them.
• Audit requirements. If you must evidence precisely what ran, tools built for that — see Testmode vs Virtuoso QA — carry machinery that general-purpose tools do not.

Choosing an approach

Three questions settle most of it.

Who will write the tests? If the answer is developers, code-based frameworks are free, excellent, and fully under your control. If it is a product manager, an operations lead or a founder, plain-English authoring is not a convenience — it is the difference between having tests and not.

What does the tool need before it can help you? Some need a codebase and a pipeline. Some need a QA practice to own them. Some need only a running application and a URL. That constraint, more than any feature list, decides what is realistic for your team.

What happens in six months? Every test suite is a maintenance commitment. Ask what breaks when the interface is redesigned, and who fixes it.

For the tools themselves, sorted by what each one needs from you, see the current roundup of AI testing tools.