Acceptance testing software built by a supplier

You paid an agency, a contractor or an offshore team to build something. It has been delivered. The demo went well.

Now: how do you know it works?

This is an awkward position that a great many organisations occupy and few talk about. You have no source code, or rights to it you cannot use. Nobody internally can read it if you did. The supplier says it is finished and is waiting for sign-off, and the only evidence you have is a demonstration they controlled.

Why the usual advice does not apply

Standard testing guidance assumes you own the codebase and employ people who understand it. Write unit tests, get coverage up, wire it into CI. None of that is available here, and suggesting it to a buyer in this position is not useful.

What you do have is more valuable than it looks: you know what the software is supposed to do. You specified it. That is precisely the knowledge acceptance testing requires, and it is knowledge the supplier cannot supply on your behalf.

Write the criteria before delivery

The single highest-leverage thing you can do costs nothing and happens before any code exists: write your acceptance criteria as concrete flows.

A usable criterion names three things — a starting state, an action, and an observable outcome:

As a customer with one item in the basket, completing checkout with a valid card produces an order confirmation email and an order visible in the account history.

Compare that with what most contracts actually contain: “the checkout process must function correctly”. The first can be verified by someone watching a screen. The second is an argument waiting to happen.

The test for a good criterion is simple. If you cannot check it by looking at the application, it is not an acceptance criterion. It might be a legitimate requirement — performance targets, security posture, code quality — but it needs a different kind of verification and should be handled separately.

Fifteen to thirty of these will cover most commissioned projects. Attach them to the contract.

Verify them yourself

Do not accept a demonstration as evidence. A demo is a rehearsed path through software, performed by the people who built it, on data they chose. It tells you the happy path works on a Tuesday.

Run the criteria yourself, against the delivered application, on a system you control, with data you chose. Because end-to-end testing works through the interface, this requires no source access and no cooperation beyond a URL and credentials.

Three things worth checking that suppliers rarely demonstrate:

• The unhappy paths. An expired card. A duplicate email. A required field left blank. This is where commissioned software is weakest, because it is where the specification was thinnest.
• Permissions. Log in as one customer and try to reach another’s data. This is the failure that turns into a disclosure obligation.
• The second time. Run the same flow twice. Software that assumes a clean state frequently breaks on the repeat.

Keep the tests after sign-off

Acceptance testing is usually treated as an event: verify, sign, move on. That wastes most of the value.

The criteria you wrote are a regression suite. If the supplier stays on a maintenance retainer, they now describe what must not break in future work. If they do not, and the software sits untouched for two years, the suite is how you find out that a browser update, an expired certificate or a changed payment provider broke something nobody was watching.

The cost of keeping them running is small. The cost of discovering a two-month-old breakage from a customer is not.

When something fails

Report failures as reproductions rather than opinions. “Checkout is broken” is disputable; a specific flow with specific data, run twice, with a recorded result is not. It moves the conversation from whether there is a problem to when it will be fixed.

This is worth more than it sounds in a commercial relationship. Suppliers are not usually acting in bad faith — they are working from a specification that was vaguer than either party realised. Concrete, reproducible criteria remove the ambiguity that turns delivery into a dispute.

Tools that fit this situation

The requirement is unusual and narrows the field sharply: the tool must run against an application you cannot modify, with no source access, no build pipeline and probably no engineers.

That eliminates code-based frameworks, anything requiring CI, and most QA platforms. What is left are tools that need only a URL and a description of the flow: Testmode, testRigor, Rainforest QA, Functionize and Autify.

Plain-English authoring matters more here than usual, for a specific reason: the person who knows what was commissioned is the person who wrote the specification, not a developer. If writing the test requires code, the knowledge and the capability sit in different heads, and something is lost in the handover.

See the roundup of AI testing tools for the whole field, sorted by what each one needs from you, and regression testing without a QA team for what to do with the suite after sign-off.