Guide
Natural language test automation
Writing automated tests as plain English sentences instead of code. How the two generations of it differ, how to write a description that actually works, and where the approach breaks down.
Last reviewed 29 August 2026
Natural language test automation is automated testing where the test is a sentence rather than a script. You write what should happen — “log in as a standard user, add a laptop to the basket, and check the total updates to include VAT” — and the tool determines which elements to interact with and whether the result was correct.
The significance is not that it saves typing. It is that it changes who is allowed to write a test. A scripted suite can only be authored by people who write code, which in most organisations is a small and heavily committed group. A described suite can be authored by whoever knows what the software is supposed to do.
Two generations, often confused
Products in this space work in one of two quite different ways, and the difference matters more than any feature comparison.
Structured natural language. The English you write is drawn from a defined command vocabulary. The tool parses your sentence into known actions and resolves elements through locators. testRigor and Testsigma work this way.
Free-form description. You write the sentence however you like, and an AI agent interprets the intent at run time, deciding for itself how to carry it out. Testmode and Functionize work this way.
| Structured grammar | Free-form description | |
|---|---|---|
| What you write | English from a documented command set | Any sentence |
| Learning curve | A vocabulary to learn | None |
| Consistency | High — a test reads the same way twice | Depends on how you write |
| Determinism | Higher | Slightly lower |
| Behaviour after a redesign | Locators may need updating | Intent is re-interpreted |
| Ambiguity | Rejected at authoring time | Resolved at run time, sometimes wrongly |
Neither is better in the abstract. Structured grammars trade flexibility for predictability, which is the right trade for a large regulated suite. Free-form description trades some predictability for the fact that nobody has to learn anything, which is the right trade when the alternative is having no tests.
How to write a description that works
This is the part vendor documentation tends to skip, and it is where most of the practical difficulty lives. A natural-language suite is only as good as its sentences.
Name the starting state. “Add an item to the basket” assumes a logged-in user, a product in stock and an empty basket. Say so. Most surprising failures are unstated preconditions.
Describe the outcome, not the mechanics. Write “the order appears in the order history” rather than “click the third row of the table”. Mechanics are what breaks in a redesign; outcomes are what you actually care about.
Be exact where exactness matters. If the total must be €1,247.50, write the number. A description invites approximation unless you close it off.
One flow per test. A sentence describing six things fails as a unit and tells you nothing about which of the six broke.
Write it so a new colleague could follow it. This is the reliable test. If a competent human reading your sentence would have to ask a clarifying question, the AI has the same problem — it just will not ask.
Where the approach breaks down
- Genuinely ambiguous expectations. “Check the page looks right” is not a test in any methodology.
- Pixel-level interaction. Drag-and-drop, canvas drawing, map manipulation and games are easier to demonstrate by recording than to describe. Tools built around recording, such as Autify, handle these better.
- Exhaustive data-driven cases. Two hundred permutations of a pricing rule are a loop in code, not two hundred sentences.
- Hard audit requirements. Where you must evidence precisely what executed, the determinism of code or of a structured grammar is worth more than the convenience of prose. See Testmode vs Virtuoso QA.
What it does not remove
Natural language removes the scripting. It does not remove the thinking, and teams that expect it to are disappointed.
You still have to decide which flows matter, what correct behaviour actually is, what to do when something fails, and how often to run the suite. Those are product and risk judgements. What has changed is that making that judgement no longer requires being able to express it in TypeScript.
Tools that work this way
Structured: testRigor, Testsigma, Virtuoso QA.
Free-form: Testmode, Functionize.
Adjacent, but not natural language: Autify and Testim start from a recording; Rainforest QA drafts steps with AI and has you edit them visually.
The full field, sorted by what each tool needs from you before it can help, is in the roundup of AI testing tools.
Common questions
What is natural language test automation?
It is automated testing where the test is written as an ordinary sentence rather than as code. You describe the flow — log in, add an item to the basket, check the total — and the tool works out which elements to interact with and what a correct result looks like. It removes the requirement that a developer write and maintain a script for every flow.
Is natural language testing reliable?
For well-described, deterministic flows, yes. Reliability tracks the precision of the description far more than the capability of the tool: a sentence two people would read differently will also be carried out two different ways. Vague expectations, not the AI, are the usual cause of a flaky natural-language suite.
What is the difference between structured and free-form natural language testing?
Structured tools such as testRigor and Testsigma accept English drawn from a documented command set, so tests read consistently and behave predictably, at the cost of learning the vocabulary. Free-form tools such as Testmode take the sentence as written and let an AI agent interpret intent at run time, which is more flexible and slightly less deterministic.
Do you still need developers for natural language testing?
No, and that is the point of it. Anyone who understands the workflow can author a test — product managers, operations staff, founders. Developers are still useful for deciding what is worth testing and for wiring results into a delivery process, but they are no longer a bottleneck on writing tests at all.