How Self-driving works

Self-driving is Testmode’s agent for one project. You give it an environment to work on and a goal in your own words; it does the testing work a QA engineer would do between releases, and hands back only what needs a person. This page explains the moving parts. To switch it on, see Turn on Self-driving.

1. It explores. A little every day, Self-driving opens your app in a real browser and reads it page by page. Each page joins the map: what the screen is for, a screenshot, the links to and from it.
2. It reviews the project. After an exploration, a deploy, a change it noticed, or when you ask, it reads the goal, the map, the test suite and the latest runs, and decides what the suite lacks.
3. It changes the suite. It adds the tests its explorations suggest, writes and fixes test cases, switches off the ones that no longer make sense, groups them in plans and schedules the plans.
4. Your tests run. The scheduled plans run like any other, alongside the runs you and your CI start.
5. It investigates what fails. When a scheduled, CI or plan run fails, it re-runs the failures and says what each one is: a bug, a flaky test, a test that needs updating, or an environment problem.
6. It reports. What needs a decision goes to the Inbox. What it did on its own is listed under Activity, each change with Undo.

An exploration is one visit to your app. It runs for about 20 minutes at most, in four parts:

• The crawl opens pages and reads them, following links within the environment. It fills in no form except the login, clicks nothing but a cookie banner and leaves logout links alone. Besides reading, it measures what a glance misses: error statuses, slow pages, broken images and links, script errors, text such as undefined on screen, figures that don’t add up.
• Journey walks reach screens no link opens, such as the checkout after a product is in the cart, by taking a few steps from pages it read.
• Exploratory tests try things on the pages it read, the way a tester hunting bugs does: sort a list and check the order, search in lower case, enter an invalid value and check it is refused, reload and check a choice was kept. On a production environment that does not allow tests that change data, they stop short of anything that would keep data.
• The analysis compares what it found with your test suite and suggests new test cases, changes to existing ones, plans and variables, plus recommendations about defects it confirmed.

Self-driving starts its own explorations: up to the day’s pages while it learns your app, then one a day to keep up. You can start one yourself with Explore something specific. See Start an exploration.

A review is Self-driving thinking about the project as a whole. It:

• researches the product on the web when the map does not say enough (its own site, docs, pricing, changelog) and writes a short summary of the product;
• works out the journeys users take and ranks them by your goal (critical, high, normal, low), keeping any ranking your team gave;
• marks tests that change real data, which then never run on a production environment unless that environment allows it;
• starts an exploration focused on what the map lacks, at most one a day;
• adds the exploration suggestions that serve the goal, writes, fixes and switches off test cases, and tries each new or changed test once before it joins a plan;
• groups tests in plans by purpose (Smoke for the critical journeys, Regression for the rest) and schedules them within your credits;
• asks your team what only you know, as a question in the Inbox.

It ends with an update: a short account of what it did and every change it made to the suite, shown under Activity.

When a run started by a schedule, by CI or for a test plan ends Failed, Self-driving looks at every failed test: what the test says, what each step did, the screenshot of the failure, the browser’s logs and the test’s history. It re-runs the test in a fresh browser, and files one verdict per test. For a test that needs updating, it rewrites the steps, tries them in a draft run, and offers them for you to apply when they pass. See Investigate failed runs.

• It never changes a test without saying so. Every change is a row under Activity with what was there before, and Undo puts it back. Choose Suggest them in its settings and it only suggests.
• It never runs a data-changing test on production unless that environment allows tests that change data. See Pause testing and protect data.
• It never fills in forms while it explores, apart from the login.
• It never works while testing is paused, or while it is off.
• It never goes past your credits. Its levels have a known cost a month, and the schedules it plans may take at most 70% of your organization’s monthly credits.

• The goal. What matters most, in your own words. Every exploration, review and investigation reads it.
• Answers. When it asks a question in the Inbox, your answer is kept as a note every later run reads.
• Notes. Teach it about the app under What it knows on the map: test accounts, flows to leave alone, things that only happen in one region.
• Journeys. Rank a journey yourself and later reviews keep your ranking. Delete a journey or a screen that is wrong.
• Undo. Take back any change it made to the suite.
• The assistant. Ask it about the map, the Inbox or the goal, or have it start a review. See Ask the assistant.

There are no locks: you and Self-driving work in the same test cases. Test cases shows who made each test case’s last change, and marks the ones Self-driving wrote this week New. A test case’s editor lists Self-driving’s changes to it, each with Undo. When you edit a test case, your version stands; Self-driving works from it next time.