SmoothQA Run Tests — GitHub Action

Trigger SmoothQA test suite runs from your CI/CD pipeline. The action calls the Pipeline API, waits for results, and optionally posts a summary comment on your pull request.

Quick start

- name: Run E2E tests
  uses: your-org/smoothqa-run-tests@v1
  with:
    api_key: ${{ secrets.SMOOTHQA_API_KEY }}
    base_url: ${{ vars.SMOOTHQA_URL }}
    suite_ids: "suite-uuid-1, suite-uuid-2"

Local development: Use uses: ./github-action to reference the action from within this monorepo.

Inputs

Input	Required	Default	Description
api_key	yes	—	Pipeline API key (starts with sqpk_). Create one in Settings → Pipeline API keys.
base_url	yes	—	Base URL of your SmoothQA instance.
suite_ids	yes	—	Comma-separated test suite IDs to run (max 10).
timeout_ms	no	600000	Total timeout in milliseconds (range: 30 000 – 1 800 000).
include_details	no	false	Include individual test case results in the output and PR comment.
comment_on_pr	no	true	Post a summary comment on the pull request.
github_token	no	${{ github.token }}	Token for posting PR comments. The default GITHUB_TOKEN works for most cases.
fail_on_test_failure	no	true	Fail the workflow step when any test suite fails. Set to false to treat test failures as informational.

Outputs

Output	Description
overall_status	PASSED or FAILED
total_suites	Number of suites that ran
passed_suites	Number of suites that passed
failed_suites	Number of suites that failed
total_duration_ms	Total wall-clock duration in milliseconds
result_json	Full JSON response from the API (compact)

Using outputs in subsequent steps

- name: Run tests
  id: smoothqa
  uses: your-org/smoothqa-run-tests@v1
  with:
    api_key: ${{ secrets.SMOOTHQA_API_KEY }}
    base_url: ${{ vars.SMOOTHQA_URL }}
    suite_ids: "suite-uuid-1"
    fail_on_test_failure: "false"

- name: Check results
  run: |
    echo "Status: ${{ steps.smoothqa.outputs.overall_status }}"
    echo "Passed: ${{ steps.smoothqa.outputs.passed_suites }}/${{ steps.smoothqa.outputs.total_suites }}"

PR comments

When comment_on_pr is true (the default) and the workflow is triggered by a pull_request event, the action posts a formatted comment with:

• Overall pass/fail status
• Per-suite breakdown (status, pass count, duration)
• Failed test case details (when include_details is true)

If the action runs multiple times on the same PR, it updates the existing comment instead of creating duplicates.

Full workflow example

name: E2E Tests

on:
  pull_request:
    branches: [main, develop]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run SmoothQA tests
        uses: your-org/smoothqa-run-tests@v1
        with:
          api_key: ${{ secrets.SMOOTHQA_API_KEY }}
          base_url: ${{ vars.SMOOTHQA_URL }}
          suite_ids: ${{ vars.SMOOTHQA_SUITE_IDS }}
          include_details: "true"
          timeout_ms: "300000"

Publishing

The action source lives in github-action/ in this monorepo. A CI workflow (.github/workflows/publish-action.yml) automatically publishes it to a separate repo for external consumption.

How to release a new version

1. Tag the monorepo commit:

   git tag action-v1.0.0
   git push origin action-v1.0.0

2. The workflow copies the action files to the target repo, creates a v1.0.0 tag, and updates the rolling v1 tag.
3. Users reference your-org/smoothqa-run-tests@v1 — always pointing to the latest v1.x.x.

One-time setup

1. Create the target repo (e.g. your-org/smoothqa-run-tests). It can be empty.
2. Create a Personal Access Token (or fine-grained token) with contents:write on the target repo.
3. Add it as a repository secret named ACTION_REPO_TOKEN in this monorepo.

Issuing a Pipeline API key

1. Open Settings in the SmoothQA web UI.
2. Scroll to Pipeline API keys.
3. Click Issue new API key and give it a name (e.g. "GitHub Actions").
4. Copy the token immediately — it is only shown once.
5. Add it as a repository secret named SMOOTHQA_API_KEY.

Requirements

• curl and jq must be available on the runner (both are pre-installed on ubuntu-latest).
• The SmoothQA instance must be reachable from the runner.