Run tests from CI and AI agents
Testmode’s API lets GitHub Actions, any other CI pipeline and AI agents over MCP start runs and read results, with a project API key or by signing in. An overview of the options and where to start.
Testmode runs from outside the app too. A deploy pipeline can run your smoke tests after every release and fail the build when one fails, and tell Self-driving what it just deployed; an AI coding agent can run the tests for the feature it just changed and read the results. Three front doors share one API, which comes with every paid plan:
| Front door | For | Signs in with |
|---|---|---|
| GitHub Action | GitHub Actions workflows | A project API key |
| REST API | Any other CI: GitLab, CircleCI, Jenkins, Bitbucket, a script | A project API key |
| MCP server | AI agents: Claude, ChatGPT, Cursor, VS Code, Claude Code | A project API key, or signing in as you |
Everything is served on https://api.testmode.ai: the REST API under /v1/, the MCP server at /mcp.
Where to start
Section titled “Where to start”Settings > API & CI in your project has everything: your API keys, and copy-paste snippets for GitHub Actions, other CI and MCP, filled in with the environment and test plan you pick.
- Create an API key.
- Pick your front door:
What callers can do
Section titled “What callers can do”- Read the project’s environments, test plans, test cases and runs.
- Start runs of a test plan, test cases by id, or test cases by tag, on any environment.
- Report a deploy (REST API), so Self-driving reviews what changed. See Report a deploy.
They can’t read credentials or variable values, change the test suite, or create API keys. Runs they start appear in Runs with the key’s name or the agent as their trigger, use credits like any other run, and send report emails to whoever asked for them. While testing is paused for the project, they can’t start runs.
When a run started from CI fails, Self-driving investigates it, as it does a scheduled run. See How Self-driving investigates failed runs.
Connect an AI agent over MCP
Section titled “Connect an AI agent over MCP”Add https://api.testmode.ai/mcp as a custom connector or MCP server in your AI app and sign in. The agent can then list your projects and tests, start runs and read the results. See Connect an AI agent over MCP.
Common questions
Section titled “Common questions”Does a run started from CI cost the same as one started in the app?
Section titled “Does a run started from CI cost the same as one started in the app?”Yes. Each action its tests run uses one of your organization’s credits, exactly as from the Run tests dialog, and tests that never run are given back.
What happens to a key when its creator leaves the organization?
Section titled “What happens to a key when its creator leaves the organization?”It stops working for good. A key only works while the member who created it is an Editor, Admin or Owner, so removing them or making them a Viewer disables every key they created. Create a new key and update your CI secret.
Can an AI agent change or delete my tests?
Section titled “Can an AI agent change or delete my tests?”No. Agents can list projects, environments, test plans, test cases and runs, and start runs. They can’t edit the test suite, and they never see passwords or variable values.
Why does my CI job keep running after a test fails?
Section titled “Why does my CI job keep running after a test fails?”The job waits for the whole run, so it reports every test, not just the first failure. Set wait to false to return as soon as the run is queued.