Visual regression testing Drupal using BackstopJS - Part 2

20 Apr 2024

#testing #backstopjs #github-actions

See Part 1 for an example how to set up BackstopJS.

You can see this in action here:

• Failing tests: https://github.com/tuutti/visual-regression-example/pull/2
• Passing tests: https://github.com/tuutti/visual-regression-example/pull/3

Prerequisites

The workflow works something like this:

1. The reference images are generated on commit to main branch and stored as Actions artifact.
2. Opening a pull request will then download the artifact, extract the images as reference, run tests against said reference images and compare what has changed.
3. The HTML test result is uploaded to GitHub Pages, so you can visually preview the changes.
4. Merging the pull request will mark the changes as “approved” and generate new reference images.

Drupal is set up using Docker Compose and Stonehenge. You can check the used Docker compose file here.

Using Stonehenge is completely optional, and it should be relatively easy to adjust this to work with any other stack, like DDEV or Lando. You can also run Drupal directly on the host runner using shivammathur/setup-php Action.

NOTE: We’ve found that the MySQL deadlocks on cache_config table a lot when running tests in parallel. Due to this, it’s highly recommended to set up Redis or some other key-value cache storage to minimize the chance of random failures.

Setting up GitHub Actions

Create .github/workflows/visual.yml file:

on:
  pull_request:
    types: [opened, reopened, synchronize, ready_for_review]
    paths-ignore:
      - '**.md'
  push:
    branches:
      - main

name: Visual regression tests

# The concurrency group is used to make sure only one visual regression test
# can be run at a time. Running multiple tests in parallel can cause a race
# condition with GitHub Pages deployments.
# Due to GitHub's limitation, only one test suite can be queued and run at
# a time; any additional run will be canceled automatically and must be
# re-started manually.
concurrency:
  group: visual-regression

jobs:
  tests:
    # Don't run tests against Draft pull requests.
    if: github.event.pull_request.draft == false

    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Update
        run:  sudo apt update

      - name: Install and start Stonehenge
        run: |
          git clone -b 4.x https://github.com/druidfi/stonehenge.git ~/stonehenge
          cd ~/stonehenge && make up

      - name: Build project
        run: composer install

        # Store the files folder in cache, so we don't have to install Drupal from
        # scratch every time.
        # You can force new re-installation by manually deleting the Actions cache.
      - name: Restore files folder
        id: drupal-cache
        uses: actions/cache@v4
        with:
          path: web/sites/default/files
          key: drupal-cache

      - name: Prepare Drupal setup
        run: |
          mkdir web/sites/default/files/styles -p && \
          chmod 777 web/sites/default -R

        # Start the project using Docker compose and wait until the database server
        # is up.
      - name: Start the project
        run: |
          docker compose up -d --wait
          # Wait for Drupal to respond.
          for i in {1..5}; do docker compose exec app bash \
            -c "drush sqlq 'SHOW TABLES;' -q" && break || sleep 5; done

        # Install the site from existing dump if the cache restoration was successful.
      - name: Install Drupal from existing dump
        if: steps.drupal-cache.outputs.cache-hit == 'true'
        run: |
          docker compose exec app bash -c "mysql \
            --user=drupal \
            --password=drupal \
            --database=drupal \
            --host=db \
            --port=3306 -A < /app/web/sites/default/files/latest.sql"

          docker compose exec app bash -c "drush deploy"

        # Install the site from scratch using existing configuration if we failed
        # to restore the cache.
        #
        # Dump the database into the files folder, so we can speed up the
        # installation process and install Drupal using that dump from now on.
      - name: Install Drupal from scratch
        if: steps.drupal-cache.outputs.cache-hit != 'true'
        run: |
          docker compose exec app bash -c "drush si --existing-config -y"
          docker compose exec app bash -c "drush sql-dump --result-file=/app/web/sites/default/files/latest.sql"

        # You can change the Node.js version by creating a '.nvmrc' file in
        # your Git root with Node.js version in it.
      - name: Setup Node.js dependencies
        run: |
          export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
          nvm install && npm install && npx playwright install

        # Attempt to restore previously generated reference images from
        # Actions artifact.
      - name: Restore bitmaps
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: gh run download -n bitmaps -D web/sites/backstop/bitmaps_reference || true

        # Generate new reference images if:
        # - Something is merged into the main branch.
        # - Reference images do not exist yet
        # - Pull request has 'recreate-reference-images' label.
      - name: Evaluate if we should re-create reference images
        id: evaluate-reference-images
        if: |
          contains( github.event.pull_request.labels.*.name, 'recreate-reference-images') ||
          hashFiles('web/sites/backstop/bitmaps_reference/') != '' ||
          github.ref == 'refs/heads/main'
        run: echo "generate-references=true" >> $GITHUB_OUTPUT

      - name: Generate new reference images
        if: steps.evaluate-reference-images.outputs.generate-references == 'true'
        run: node backstop/backstop.js reference

        # Save reference images as Actions artifact.
      - uses: actions/upload-artifact@v4
        if: steps.evaluate-reference-images.outputs.generate-references == 'true'
        with:
          name: bitmaps
          path: web/sites/backstop/bitmaps_reference
          overwrite: true
          compression-level: 0

      - name: Run tests
        id: run-tests
        # Skip tests if we generated new reference images on this run since tests
        # should never fail.
        if: steps.evaluate-reference-images.outputs.generate-references != 'true'
        run: |
          if ! node backstop/backstop.js test; then
            echo "result=:warning: Visual regression found! Please check if this change is wanted or accidental. " >> $GITHUB_OUTPUT
          else
            echo "result=✅ Tests passed!" >> $GITHUB_OUTPUT
          fi
          echo "report_url=You can check the output here: https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/pull/${{ github.event.pull_request.number }}/html_report/" >> $GITHUB_OUTPUT

        # Deploy the HTML report to GitHub Pages, so we can easily compare the
        # results.
        # You might want to use an external repository to store the test results
        # to prevent your main repository from getting bloated with bitmap data.
        # @see https://github.com/peaceiris/actions-gh-pages?tab=readme-ov-file#%EF%B8%8F-deploy-to-external-repository-external_repository
      - name: Deploy to PR preview
        uses: peaceiris/actions-gh-pages@v4
        if: steps.evaluate-reference-images.outputs.generate-references != 'true'
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: web/sites/backstop/
          destination_dir: pull/${{github.event.number}}

      - name: Update comment
        if: steps.evaluate-reference-images.outputs.generate-references != 'true'
        uses: thollander/actions-comment-pull-request@v2
        with:
          comment_tag: status
          pr_number: ${{ github.event.number }}
          message: "${{join(steps.run-tests.outputs.*, '  ')}}"

      - name: Export container logs
        run: docker compose logs app > /tmp/container.log

      - name: Upload container logs
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: container-log
          path: /tmp/container.log
          retention-days: 1

Setup GitHub Pages

Create an orphan gh-pages branch if you don’t have one already: git checkout --orphan gh-pages. This will create a new branch with no parents, you can then clear the working directory with: git rm --cached -r ..

Create and commit an empty .nojekyll file. This will tell GitHub Pages to skip the build process since we’re only deploying generated HTML content.

Go to your repository’s Settings -> Pages and choose gh-pages branch from the Branch dropdown.

Workflow permissions

You must grant “Read and write” permissions to the GITHUB_TOKEN so Actions can push to gh-pages branch and trigger the deployment.

Go to your repository’s Settings -> Actions -> General -> Workflow permissions and choose “Read and write permissions”.

Cleaning up old test previews

You can either use a pull_request: [closed] or schedule event. The pull_request event seemed pretty unreliable for multiple reasons, but mainly because the action can fail for whatever reason and the preview is never deleted unless you re-run the failed action manually.

Create .github/workflows/visual-regression-cleanup.yml file:

name: Delete old BackstopJS preview pages
on:
  # This allows the workflow to be triggered manually from the Actions tab.
  workflow_dispatch:
  # Run once a day at 04:05.
  schedule:
    - cron: '5 4 * * *'

concurrency:
  group: visual-regression

jobs:
  visual-regression-cleanup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: gh-pages

      - name: Setup Git user
        run: |
          git config --global user.name github-actions
          git config --global user.email github-actions@github.com

      - name: Remove stale preview pages
        env:
          GH_TOKEN: ${{ github.token }}
        # This will:
        # - Loop through all preview folders (pull/{{ number }}) and parse the pull request number
        # - Check if the pull request is still open using GitHub CLI tool.
        # - Remove the folder if the pull request is not open
        run: |
          for d in pull/*; do
            id=$(echo $d | cut -d / -f2)
            state=$(gh pr view $id --json state --jq .state)

            if [ "$state" != "OPEN" ]; then
              rm -r pull/$id
            fi
          done

          if [[ $(git status --porcelain) ]]; then
            git add .
            git commit -m 'Automated commit'
            git push
          fi