Playwright Test gives you the ability to do visual regression testing and snapshot testing. With Checkly, you can run these tests against production, 24x7. Visual regression testing is useful for testing the visual appearance of your applications and making sure production-specific issues like bad deploys, flaky 3rd party dependencies, CMS changes and user-generated content do not impact to general layout and visual consistency of your core screens on production. Have a look at this video for a quick explainer: The TL;DR is that you can:
  • Use the .toHaveScreenshot() assertion to visually compare a screenshot of your page to a golden image / reference snapshot.
  • Use the .toMatchSnapshot() assertion to compare any string or Buffer value to a golden image / reference snapshot.
Visual regression & snapshot testing is available on our Team and Enterprise plans.

​
Visual regression testing

Starting with visual regression testing takes just three easy steps:
  1. Add a single expect(page).toHaveScreenshot() line of code to your browser check script, like the example below.
    visual-comparison.spec.ts
    import { test, expect } from '@playwright/test';

test('Playwright homepage', async ({ page }) => {
   await page.goto('https://playwright.dev')
   await expect(page).toHaveScreenshot()
})
  2. Run your browser check. The first time you run it, you will get an error indicating that no golden image / reference snapshot exists yet. 
    A snapshot doesn't exist at /tmp/19g67loplhq0j/script.spec.js-snapshots/Playwright-homepage-1-chromium-linux.png.
  3. Generate a golden image / reference snapshot by clicking the “Run script and update golden image” option in the “run script” button.
This will generate the golden image, which you can inspect in the “golden files” tab in the editor. You can now save your check and on each check run the golden image will be compared to the actual screenshot.
[!CLI] If you are using the Checkly CLI, you can also generate a golden image / reference snapshot by running the following command in your terminal: 
npx checkly test --update-snapshots
Now, when your check fails due to a visual difference, you will see a diff of the golden image and the actual screenshot in your check result.

​
Configuring visual regression testing

To create accurate and actionable screenshot comparisons, Playwright gives you a ton of options to tweak how the .toHaveScreenshot() should behave. What are acceptable differences in color, size, position, etc.? Do you want to match the full screen, or ignore some dynamic elements that might screw up your comparison? Let’s look at some examples, or check the official reference docs.

​
Example 1: setting pixel ratios and color thresholds

You can control the margin of error you find acceptable between your golden image and the actual image using the following options:
  • maxDiffPixelRatio: An acceptable ratio of pixels that are different to the total amount of pixels, between 0 and 1
  • maxDiffPixels: A total acceptable amount of pixels that could be different.
  • threshold: An acceptable perceived color difference in the YIQ color space between the same pixel in compared images, between 0 (strict) and 1 (lax).
thresholds.spec.ts
import { test, expect } from '@playwright/test';

test('Playwright homepage', async ({ page }) => {
   await page.goto('https://playwright.dev')
   await expect(page).toHaveScreenshot({ maxDiffPixelRatio: 0.2 })
   await expect(page).toHaveScreenshot({ maxDiffPixels: 1000 })
   await expect(page).toHaveScreenshot({ threshold: 0.2 })
})

​
Example 2: ignoring specific screen elements

A typical homepage can have dynamic elements that change on each page load, or change based on the geographical region. Think of a “latest blog posts” section, a cookie banner or a region / language selector. Playwright allows you to ignore these elements when doing a visual comparison using the mask option and using one or more page.locator() selectors. The example below hides the cookie banner and optional CTA popup from Intercom on the Checkly docs pages.
ignoring.spec.ts
import { test, expect } from '@playwright/test';

test('Ignore cookie banner & cta popup', async ({ page }) => {
   await page.goto('https://docs.checklyhq.com')
   await expect(page).toHaveScreenshot({
      mask: [
         page.locator('.optanon-alert-box-wrapper'),
         page.locator('#intercom-container-body')
      ]
   })
})

​
Example 3: disabling animations

In some cases, any ongoing animations can cause a visual difference between your golden image and the actual screenshot. You can disable any CSS animations and transitions using the animations option.
disable-anims.spec.ts
import { test, expect } from '@playwright/test';

test('Disable animations', async ({ page }) => {
   await page.goto('https://playwright.dev')
   await expect(page).toHaveScreenshot({ animations: 'disabled' })
})

​
Snapshot testing

Snapshot testing, using the expect().toMatchSnapshot(snapshotName) assertion, is a great way to test the output of any arbitrary string or Buffer value. Note that it is not optimized for visual regression testing.
snapshot.spec.ts
import { test, expect } from '@playwright/test'

test('Match hero text', async ({ page }) => {
   await page.goto('https://playwright.dev')
   expect(await page.textContent('.hero__title')).toMatchSnapshot('hero.txt')
})
Creating or updating the golden image / reference snapshot works the same as with visual regression testing. Check the official reference docs for all options.

​
Embedding in your CI/CD workflow

Using the Checkly CLI you can code and configure visual regression and snapshot testing on your local machine and deploy any changes either directly from your local machine or from your CI/CD pipeline of choice. In a typical scenario, you would follow the steps below:
  1. Create or update a browser check with visual regression or snapshot testing on your local machine.
  2. Generate the golden image / reference snapshot(s). 
    npx checkly test --update-snapshots
    The resulting files are stored in a some-file-prepend.ts-snapshots folder next to your browser check script.
  3. Commit the browser check script and the golden image / reference snapshot(s) to your version control system.
  4. Push your code to your CI/CD pipeline.
  5. In your CI/CD pipeline, optionally run your checks again. Maybe add the --record flag to record the test in Checkly. 
    npx checkly test --record
  6. If your tests pass, deploy your checks to production. The CLI will push your snapshot to the Checkly cloud automatically. 
    npx checkly deploy
Learn more about setting up the Checkly CLI for your CI/CD pipeline 👇

​
Known limitations

  • Checkly currently only supports the Chromium and Chrome browsers.

​
Automatic screenshots on error

Whenever your Playwright script encounters an error, we will automatically snap a screenshot the moment the error occurs. This is different than Visual Regression Monitoring Here is an example from real life!
1

Script Action

Your script clicks on a button using a selector wait page.click(".my-button-class").
2

Element Issue

For some reason, that button does not exist or is not clickable.
3

Playwright Error

Playwright waits for the button with the selector to appear. It does not and Playwright throws an error.
4

Automatic Screenshot

Checkly automatically calls page.screenshot() a screenshot. The screenshot indicates that that specific button was missing.

​
Playwright resources