Run tests from CI and AI agents

Testmode’s API lets tools outside the app start runs and read their results. Use it to run your tests after every deploy from GitHub Actions or any other CI pipeline, or to let an AI agent such as Claude, ChatGPT, Cursor or VS Code look at your tests and run them over the Model Context Protocol (MCP).

Everything lives on one address, https://api.testmode.ai: the REST API under /v1/, the MCP server at /mcp.

Before you start: you need the Editor role or above to create API keys and to start runs.

Pipelines and agents that you configure by hand sign in with a project API key.

1. Open Settings for the project and choose the API & CI tab.
2. Click Create API key, give it a name you will recognize later, such as GitHub Actions, and choose when it expires: 30 days, 90 days, 1 year or never.
3. Copy the key. It starts with tmk_ and is shown only once. Store it as a secret in your CI, for example as the GitHub Actions secret TESTMODE_API_KEY.

A key works in one project and acts for the member who created it. It can read that project’s environments, test plans, test cases and runs, and start runs. It cannot change your tests or read credentials and variables.

The API & CI tab lists every key with the start of the key, when it was created, when it was last used and when it expires. To stop a key, click its delete button and confirm Revoke: pipelines using it fail from their next request.

A key also stops working when the member who created it leaves the organization or becomes a Viewer.

The Testmode action starts a run, waits for it, prints each test’s result, adds a summary table to the job page and fails the job when a test fails.

1. Add your API key to the repository as the secret TESTMODE_API_KEY (Settings → Secrets and variables → Actions).

2. Add a workflow, for example .github/workflows/testmode.yml:

   name: Testmode
   on:
     push:
       branches: [main]
     workflow_dispatch:
   
   jobs:
     e2e:
       runs-on: ubuntu-latest
       steps:
         - uses: amber-digital-bv/testmode-ai-action@v1
           with:
             api-key: ${{ secrets.TESTMODE_API_KEY }}
             environment: Staging
             test-plan: Smoke

3. Push. The run appears on the Runs page, triggered by API key: <key name>, and the job links to it.

The API & CI tab shows this workflow filled in for the environment and test plan you pick. To test a deployment, run the job after your deploy job with needs: deploy.

Give exactly one of test-plan, test-case-ids or tags. The action sets the outputs run-id, run-url and status (PASSED, FAILED, CANCELED or SKIPPED), which later steps can read.

Any pipeline that has curl and jq can start a run and wait for it. Put the key in the variable TESTMODE_API_KEY:

# Start the run
RUN=$(curl -sf -X POST https://api.testmode.ai/v1/runs \
  -H "Authorization: Bearer $TESTMODE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment": "Staging", "testPlan": "Smoke"}')
RUN_ID=$(echo "$RUN" | jq -r .id)

# Wait for it to finish, then fail the build unless it passed
while :; do
  RUN=$(curl -sf https://api.testmode.ai/v1/runs/$RUN_ID -H "Authorization: Bearer $TESTMODE_API_KEY")
  [ "$(echo "$RUN" | jq -r .finished)" = true ] && break
  sleep 15
done
echo "$RUN" | jq -r '"\(.status): \(.passedTests)/\(.totalTests) passed. \(.url)"'
[ "$(echo "$RUN" | jq -r .status)" = PASSED ]

This works the same in GitLab CI, CircleCI, Bitbucket Pipelines, Jenkins and Azure Pipelines.

Send the key as Authorization: Bearer tmk_… on every request.

POST /v1/runs takes environment (name or id; the default environment when left out), exactly one of testPlan (name or id), testCaseIds or tags, and an optional name. A run carries its status (see Run and test statuses), finished, counts of passed and failed tests, a url to open it in Testmode and, per test, the summary of what was checked or why it failed.

Errors come back as {"error": "…"}. 401 means the key is missing, wrong, expired or revoked, 402 that the organization’s balance does not cover the run, 403 that the key’s creator can no longer edit the project, and 429 that the key sent more than 60 requests in a minute.

Over MCP, an AI agent can list your projects and tests, start a run when you ask for one, and tell you what passed and why a test failed.

https://api.testmode.ai/mcp

claude mcp add --transport http testmode https://api.testmode.ai/mcp

claude mcp add --transport http testmode https://api.testmode.ai/mcp \
  --header "Authorization: Bearer $TESTMODE_API_KEY"

{
  "mcpServers": {
    "testmode": {
      "url": "https://api.testmode.ai/mcp",
      "headers": { "Authorization": "Bearer <your API key>" }
    }
  }
}

The agent gets these tools: list_projects, get_project (environments and test plans), list_test_cases, start_run, get_run and list_runs. Try asking it to “run the Smoke plan on staging and tell me what failed”.

Apps you connected by signing in are listed under Profile → Connected apps. Click Disconnect to take away an app’s access at once; connecting it again asks for your approval.