Table of contents

GitHub Actions

We’ve optimized the Checkly CLI to work in any CI/CD workflow. Here are the basics you need to know that will come in handy when adapting the examples we give you to your own, specific setup.

1. For authentication, make sure to set the CHECKLY_API_KEY and CHECKLY_ACCOUNT_ID parameters as environment variables in your CI/CD platform.
2. Set the reporter you want to use for the test command using the --reporter flag, i.e. --reporter=dot.
3. To store a test session with full logging, traces and vides, set the --record flag for the test command.
4. Use the --force flag on the deploy and / or destroy commands to skip the normal confirmation steps.

When using the --record flag, the CLI will attempt to parse git specific information from the environment to display in the recorded test session as metadata. However, you can also set these data items specifically by using environment variables.

Check the CLI command line reference for more options.

GitHub Actions workflows can run on different events. In most cases, you will want to trigger the Checkly CLI after a deployment is done based on a recent git push to a branch, when a branch is merged or when a pull request is created.

GitHub created the deployments API for this and many 3rd party integrators like Vercel and Heroku use this to relay deployment status to GitHub. However, you can also call this API yourself and trigger a deployment_status event.

Make sure to set your CHECKLY_API_KEY and CHECKLY_ACCOUNT_ID as secrets in your GitHub Actions settings before you get started.

The deployment_status event is the preferred event to trigger a GH Actions workflow that executes your checks. However, this comes with some peculiarities that are native to GH Actions and a bit different from using the push event.

• The deployment event holds core information about your deployment, i.e. the environment name and an optional ENVIRONMENT_URL.
• The full git repo with full history is not available. We have to jump through some hoops to properly set the branch name for instance.
• We have no access to the original pull request that triggered the deployment event.

# .github/checkly.yml
name: 'checkly'
on: [deployment_status]

# Set the necessary credentials and export variables we can use to instrument our test run. Use the ENVIRONMENT_URL
# to run your checks against staging, preview or production.
env:
  CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
  CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
  ENVIRONMENT_URL: ${{ github.event.deployment_status.environment_url }}
  CHECKLY_TEST_ENVIRONMENT: ${{ github.event.deployment_status.environment }}
jobs:
  test-e2e:
    if: github.event.deployment_status.state == 'success' # Only run when the deployment was successful.
    name: Test E2E on Checkly
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v3
        with:
          ref: "${{ github.event.deployment_status.deployment.ref }}"
          fetch-depth: 0
      - name: Set branch name # workaround to detect branch name in "deployment_status" actions
        run: echo "CHECKLY_TEST_REPO_BRANCH=$(git show -s --pretty=%D HEAD | tr -s ',' '\n' | sed 's/^ //' | grep -e 'origin/' | head -1 | sed 's/\origin\///g')" >> $GITHUB_ENV
      - uses: actions/setup-node@v3
        with:
          node-version-file: '.nvmrc'
      - name: Restore or cache node_modules
        id: cache-node-modules
        uses: actions/cache@v3
        with:
          path: node_modules
          key: node-modules-${{ hashFiles('package-lock.json') }}
      - name: Install dependencies
        if: steps.cache-node-modules.outputs.cache-hit != 'true'
        run: npm ci
      - name: Run checks # run the checks passing in the ENVIRONMENT_URL and recording a test session.
        id: run-checks
        run: npx checkly test -e ENVIRONMENT_URL=${{ env.ENVIRONMENT_URL }} --reporter=github --record
      - name: Create summary # export the markdown report to the job summary.
        id: create-summary
        run: cat checkly-github-report.md > $GITHUB_STEP_SUMMARY
      - name: Deploy checks # if the test run was successful and we are on Production, deploy the checks
        id: deploy-checks
        if: steps.run-checks.outcome == 'success' && github.event.deployment_status.environment == 'Production'
        run: npx checkly deploy --force

If you are running a mono repo, you might bump into issues that multiple deployment_status events will randomly trigger your workflow. GitHub Actions does not have a great way to filter for this, but there are two strategies you can follow.

If part of the URL for your deployments deterministically maps to one of the apps in your repo, you can just extend the if statement in your workflow, as shown below.

jobs:
  test-e2e:
    if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment_url, 'webapp')
    name: Test E2E on Checkly
    runs-on: ubuntu-latest
    timeout-minutes: 10

In many cases, the above solution with if statements is not enough. When using Vercel and mono repos for instance, GitHub Actions will recognize any “skipped” statuses as “passed”. This can cause havoc. A way to sidestep this issue is by setting a branch protection rule on your repo that is generated by your GH Action workflow.

You do this by adding the LouisBrunner/checks-action@v1.6.0 action to your workflow and assigning some name. This name can be fixed or dynamic. See the example below.

name: 'Run Checkly MaC Checks'
on: [deployment_status]
env:
  ENVIRONMENT_URL: ${{ github.event.deployment_status.environment_url }}
  CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
  CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
run-name: Run MaC checks against ${{ github.event.deployment_status.environment_url }}
jobs:
  test-e2e:
    if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment_url, 'webapp')
    name: Test E2E on Checkly
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      # All steps as normal...
      - uses: LouisBrunner/checks-action@v1.6.0
        if: always()
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          name: Test E2E Passed Notification
          conclusion: ${{ job.status }}

The will generate a custom status for your called Test E2E Passed Notification.

Then, add a branch protection rule for your repo.

Together with any optional if statements, your GH Actions will now run only against the apps you want in your mono repo.

You can print a user friendly summary report to your GitHub Actions Job Summary by adding the --reporter=github flag to the test command and export the resulting checkly-github-report.md file to the $GITHUB_STEP_SUMMARY variable.

In the example, you can see this in the following step:

- name: Run checks
  id: run-checks
  run: npx checkly test
    -e ENVIRONMENT_URL=${{ env.ENVIRONMENT_URL }}
    --reporter=github
    --record
- name: Create summary
  id: create-summary
  run: cat checkly-github-report.md > $GITHUB_STEP_SUMMARY

Vercel Deployment Protection Jenkins

Last updated on August 16, 2024. You can contribute to this documentation by editing this page on Github