# Run tests from CI and AI agents

URL: https://testmode.ai/docs/api-ci-mcp/

> Testmode's API lets GitHub Actions, any other CI pipeline and AI agents over MCP start runs and read results, with a project API key or by signing in.

Testmode's API lets tools outside the app start runs and read their results. Use it to run your tests after every deploy from **GitHub Actions** or any other **CI pipeline**, or to let an **AI agent** such as Claude, ChatGPT, Cursor or VS Code look at your tests and run them over the Model Context Protocol (MCP).

Everything lives on one address, `https://api.testmode.ai`: the REST API under `/v1/`, the MCP server at `/mcp`.

**Before you start:** you need the Editor role or above to create API keys and to start runs.

## Create an API key

Pipelines and agents that you configure by hand sign in with a **project API key**.

1. Open **Settings** for the project and choose the **API & CI** tab.
2. Click **Create API key**, give it a name you will recognize later, such as `GitHub Actions`, and choose when it expires: 30 days, 90 days, 1 year or never.
3. Copy the key. It starts with `tmk_` and is shown **only once**. Store it as a secret in your CI, for example as the GitHub Actions secret `TESTMODE_API_KEY`.

A key works in **one project** and acts for the member who created it. It can read that project's environments, test plans, test cases and runs, and start runs. It cannot change your tests or read credentials and variables.

The **API & CI** tab lists every key with the start of the key, when it was created, when it was last used and when it expires. To stop a key, click its delete button and confirm **Revoke**: pipelines using it fail from their next request.

A key also stops working when the member who created it leaves the organization or becomes a Viewer.

**Caution**

Treat a key like a password. Keep it in your CI's secret store, never in a repository, and revoke it if it leaks.

## Run tests from GitHub Actions

The [Testmode action](https://github.com/amber-digital-bv/testmode-ai-action) starts a run, waits for it, prints each test's result, adds a summary table to the job page and fails the job when a test fails.

1. Add your API key to the repository as the secret `TESTMODE_API_KEY` (**Settings → Secrets and variables → Actions**).
2. Add a workflow, for example `.github/workflows/testmode.yml`:

**.github/workflows/testmode.yml**

```yaml
name: Testmode
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: amber-digital-bv/testmode-ai-action@v1
        with:
          api-key: ${{ secrets.TESTMODE_API_KEY }}
          environment: Staging
          test-plan: Smoke
```

3. Push. The run appears on the **Runs** page, triggered by `API key: <key name>`, and the job links to it.

The **API & CI** tab shows this workflow filled in for the environment and test plan you pick. To test a deployment, run the job after your deploy job with `needs: deploy`.

| Input | Default | What it does |
| --- | --- | --- |
| `api-key` | required | Your project API key, from a secret |
| `environment` | the project's default environment | Environment name or id |
| `test-plan` | | Test plan name or id |
| `test-case-ids` | | Test case ids, separated by commas or spaces |
| `tags` | | Every enabled test case that has any of these tags |
| `name` | workflow, run number, branch and commit | The run name in Testmode |
| `wait` | `true` | `false` starts the run and ends the step at once |
| `timeout-minutes` | `30` | Stop waiting after this long. The run itself goes on |

Give exactly one of `test-plan`, `test-case-ids` or `tags`. The action sets the outputs `run-id`, `run-url` and `status` (`PASSED`, `FAILED`, `CANCELED` or `SKIPPED`), which later steps can read.

## Run tests from other CI

Any pipeline that has `curl` and `jq` can start a run and wait for it. Put the key in the variable `TESTMODE_API_KEY`:

```bash
# Start the run
RUN=$(curl -sf -X POST https://api.testmode.ai/v1/runs \
  -H "Authorization: Bearer $TESTMODE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment": "Staging", "testPlan": "Smoke"}')
RUN_ID=$(echo "$RUN" | jq -r .id)

# Wait for it to finish, then fail the build unless it passed
while :; do
  RUN=$(curl -sf https://api.testmode.ai/v1/runs/$RUN_ID -H "Authorization: Bearer $TESTMODE_API_KEY")
  [ "$(echo "$RUN" | jq -r .finished)" = true ] && break
  sleep 15
done
echo "$RUN" | jq -r '"\(.status): \(.passedTests)/\(.totalTests) passed. \(.url)"'
[ "$(echo "$RUN" | jq -r .status)" = PASSED ]
```

This works the same in GitLab CI, CircleCI, Bitbucket Pipelines, Jenkins and Azure Pipelines.

### API reference

Send the key as `Authorization: Bearer tmk_…` on every request.

| Request | Returns |
| --- | --- |
| `GET /v1/project` | The project with its environments and test plans |
| `GET /v1/environments` | The project's environments |
| `GET /v1/test-plans` | The project's test plans |
| `GET /v1/test-cases?tag=smoke&search=login` | The project's test cases, optionally filtered |
| `POST /v1/runs` | Starts a run and returns it |
| `GET /v1/runs` | The most recent runs, `?limit=` up to 100 |
| `GET /v1/runs/{id}` | A run with every test's status and summary |

`POST /v1/runs` takes `environment` (name or id; the default environment when left out), exactly one of `testPlan` (name or id), `testCaseIds` or `tags`, and an optional `name`. A run carries its `status` (see [Run and test statuses](https://testmode.ai/docs/run-and-test-statuses/)), `finished`, counts of passed and failed tests, a `url` to open it in Testmode and, per test, the summary of what was checked or why it failed.

Errors come back as `{"error": "…"}`. `401` means the key is missing, wrong, expired or revoked, `402` that the organization's balance does not cover the run, `403` that the key's creator can no longer edit the project, and `429` that the key sent more than 60 requests in a minute.

## Connect an AI agent over MCP

Over MCP, an AI agent can list your projects and tests, start a run when you ask for one, and tell you what passed and why a test failed.

**Claude, ChatGPT and other apps**

Add a custom connector (in Claude: **Settings → Connectors → Add custom connector**) with this address:

```text
https://api.testmode.ai/mcp
```

The app opens Testmode. Sign in if needed and check the page: it shows the app's name, where you will be sent back to, and what the app will be able to do. Click **Connect**. The app then works in every project you can use, with your role in each: it can start runs where you are an Editor or above.

**Claude Code**

Sign in with your Testmode account:

```bash
claude mcp add --transport http testmode https://api.testmode.ai/mcp
```

Or use an API key, for one project:

```bash
claude mcp add --transport http testmode https://api.testmode.ai/mcp \
  --header "Authorization: Bearer $TESTMODE_API_KEY"
```

**Cursor and VS Code**

Add the server to your MCP configuration. Without `headers`, the editor asks you to sign in to Testmode; with them, it uses an API key for one project.

```json
{
  "mcpServers": {
    "testmode": {
      "url": "https://api.testmode.ai/mcp",
      "headers": { "Authorization": "Bearer <your API key>" }
    }
  }
}
```

The agent gets these tools: `list_projects`, `get_project` (environments and test plans), `list_test_cases`, `start_run`, `get_run` and `list_runs`. Try asking it to "run the Smoke plan on staging and tell me what failed".

**Note**

Only connect an app you started connecting yourself, just now. Apps choose their own names, so check the address Testmode says it will send you back to.

### Disconnect an app

Apps you connected by signing in are listed under **Profile → Connected apps**. Click **Disconnect** to take away an app's access at once; connecting it again asks for your approval.

## Common questions

### Does a run started from CI cost the same as one started in the app?

Yes. Each test the run executes counts against your organization's balance exactly as it would from the Run tests dialog, and tests that never run are given back.

### What happens to a key when its creator leaves the organization?

It stops working at once. A key only works while the member who created it is an Editor, Admin or Owner of the organization, so removing them or making them a Viewer disables every key they created. Create a new key and update your CI secret.

### Can an AI agent change or delete my tests?

No. Agents can list projects, environments, test plans, test cases and runs, and start runs. They cannot edit the test suite, and they never see saved passwords or variable values.

### Why does the job keep running after a test fails?

It waits until every test in the run has finished, then fails the job if any test did not pass. Set wait to false if you only want to start the run.

## Related

  - [Run tests](https://testmode.ai/docs/run-tests/): Start runs from the app.
  - [Read run results](https://testmode.ai/docs/read-run-results/): What a run records and how to find out why a test failed.
  - [Roles and permissions](https://testmode.ai/docs/roles-and-permissions/): Who can create keys and start runs.
  - [How credits work](https://testmode.ai/docs/how-credits-work/): What a run costs and when it is given back.