Run tests from GitHub Actions and other CI

Run your Testmode tests from CI, for example after every deploy to staging, and fail the build when a test fails. You need a project API key.

1. Create an API key and add it to your repository as the secret TESTMODE_API_KEY (Settings > Secrets and variables > Actions in GitHub).

2. Save a workflow such as .github/workflows/testmode.yml:

   name: Testmode
   on:
     push:
       branches: [main]
     workflow_dispatch:
   jobs:
     e2e:
       runs-on: ubuntu-latest
       steps:
         - uses: amber-digital-bv/testmode-ai-action@v1
           with:
             api-key: ${{ secrets.TESTMODE_API_KEY }}
             environment: Staging
             test-plan: Smoke

3. Push. The job starts the run, waits for it, prints each test’s result, and fails when a test fails.

Settings > API & CI shows this snippet filled in with the environment and plan you pick. The action is Testmode AI on the GitHub Marketplace.

Give exactly one of test-plan, test-case-ids or tags. Names match exactly, ignoring case; pass the id when two share a name.

The action sets run-id, run-url and status (PASSED, FAILED, SKIPPED, or QUEUED when not waiting), and writes a job summary with each test. The step succeeds when the run passed, or once it is queued with wait: false; it fails when the run didn’t pass or waiting timed out, and when the run couldn’t be started or read.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy-staging.sh
  e2e:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - uses: amber-digital-bv/testmode-ai-action@v1
        with:
          api-key: ${{ secrets.TESTMODE_API_KEY }}
          environment: Staging
          tags: smoke

The action’s script runs unchanged in any CI with bash, curl and jq. Set the key and what to run as environment variables:

export TESTMODE_API_KEY=tmk_...        # from your CI's secret store
export TESTMODE_TEST_PLAN="Smoke"      # or TESTMODE_TAGS / TESTMODE_TEST_CASE_IDS
export TESTMODE_ENVIRONMENT="Staging"  # optional
curl -fsSL https://raw.githubusercontent.com/amber-digital-bv/testmode-ai-action/v1/run.sh | bash

It exits 0 when the run passed, 1 when it didn’t pass or waiting timed out, and 2 when the run couldn’t be started or read. The script’s header lists every setting.

Prefer your own calls? Use the REST API directly.

• Run against a test environment. Tests do what their steps say, including orders and messages.
• Give busy pipelines their own keys. One key allows 60 requests a minute, and a waiting job polls every 15 seconds.
• Name runs after the change, such as the pull request, so Runs shows which change a result belongs to. The action does this by default.
• Every test counts one of your plan’s test runs, like a run from the app.