Use case · Tests to production monitors

Promote twelve tests, not four hundred

Your Playwright suite can run against production. The work is choosing which specs deserve to, and making them safe to run there forever. This is that rollout: tagging, fixtures, secrets, ownership, and the pipeline that ships it.

Start for freeBook a demo
Tag-based promotionOne suite, two jobsShipped from CI
__checks__/production.check.ts
1// __checks__/production.check.ts
2import { PlaywrightCheck, Frequency } from 'checkly/constructs'
3
4// Not the whole suite. The specs you tagged as production-worthy.
5new PlaywrightCheck('production-critical', {
6 name: 'Production critical paths',
7 playwrightConfigPath: './playwright.config.ts',
8 pwTags: ['@production'],
9 frequency: Frequency.EVERY_5M,
10 locations: ['us-east-1', 'eu-west-1'],
11})

Trusted by teams who already had the tests

Vercel
Carhartt
CrowdStrike
Airbus
Fanatics
Mistral
ServiceNow
GoFundMe
Hopper
1Password
Fastly
Total Wine
“We’ve saved a lot of time by using Playwright with Checkly.”
Keegan Donley·Senior Front-End Engineer · Kizen

Three ways this rollout goes wrong

The idea is easy and the execution has three well-known failure modes. All of them are avoidable if you decide up front.

all 400

Promoting the whole suite is the fastest way to fail

Four hundred specs every five minutes produces noise, cost, and a rotation that mutes the channel by Thursday. The value is in choosing the twelve that mean something.

fixtures

CI tests assume a world production does not have

A seeded database, a reset between runs, a mocked payment provider, a user created in a fixture. Every one of those assumptions has to be replaced before the spec can run against the real thing.

unowned

A monitor nobody owns becomes a muted channel

The promotion succeeds and then the first flake arrives at 2am with no obvious owner. Six weeks later the alert is filtered and the coverage is theoretical.

For why your Playwright tests can serve as monitors at all, see monitoring with Playwright. This page is about doing it without regretting it.

“Checkly CLI has enhanced our engineering team’s ability to quickly build, validate and deploy an entire suite of checks from their local development environment.”

Tavares Chambless

Manager, Quality Assurance · Loyal Health

Tag the specs that would wake you up

The selection test is simple. If this spec went red at 2am, would you want someone paged? Tests that assert on copy, layout, or a rarely-used edge case are valuable at merge time and pure noise as an alert. Tag the ones that clear the bar and promote only those.

  • A tag is the whole selection mechanism: no separate list to maintain
  • Start with sign-in, signup, and the core action; add slowly
  • Untagged specs keep running in CI exactly as they do today
  • Changing what is monitored is a one-line diff in a pull request
e2e/checkout.spec.ts
1// e2e/checkout.spec.ts
2import { test, expect } from '@playwright/test'
3
4// @production earns its place: it fails only when a user would be blocked.
5test('guest checkout completes @production', async ({ page }) => {
6 await page.goto(`${process.env.BASE_URL}/store`)
7 await page.getByRole('button', { name: 'Add to cart' }).click()
8 await page.getByRole('link', { name: 'Checkout' }).click()
9 await expect(page.getByText('Order confirmed')).toBeVisible()
10})
11
12// Stays in CI. Asserts on copy that marketing changes twice a quarter —
13// valuable at merge time, pure noise as a 2am page.
14test('empty cart shows the upsell banner', async ({ page }) => {
15 await page.goto(`${process.env.BASE_URL}/cart`)
16 await expect(page.getByText('You might also like')).toBeVisible()
17})

Make the spec safe to run forever

A CI test runs against a world it controls and then throws it away. A monitor runs against the real world every five minutes, indefinitely. Three changes usually cover the gap, and they are the substance of the migration.

  • Replace seeded fixtures with a persistent probe account you can filter from analytics
  • Replace hardcoded hosts with an environment variable, so one spec targets preview or production
  • Make it idempotent: clean up what it creates, or it accumulates forever
  • Point transactional steps at gateway test mode rather than real money
e2e/support/production.ts
1// e2e/support/production.ts
2import { test as base } from '@playwright/test'
3
4// A CI spec assumes a world it controls. Production is not that world.
5export const test = base.extend({
6 // 1. No seeded fixtures — use a dedicated probe account that persists.
7 account: async ({}, use) => {
8 await use({
9 email: process.env.PROBE_USER!,
10 password: process.env.PROBE_PASS!,
11 })
12 },
13
14 // 2. No hardcoded host — the target is injected per environment.
15 baseURL: async ({}, use) => {
16 await use(process.env.ENVIRONMENT_URL ?? 'https://acme.com')
17 },
18})
19
20// 3. Idempotent by construction: the spec cleans up what it creates,
21// so running every 5 minutes forever does not accumulate garbage.
22export async function cleanup(request, id: string) {
23 await request.delete(`/api/projects/${id}`)
24}

Ship the promotion through the pipeline

The promotion itself should be a deploy step, not a console session. Run the tagged specs against the preview URL to gate the merge, then deploy the same files as monitors when it lands. Monitors change in the pull request that changes the code they cover.

  • Checks run against each preview deployment before the merge
  • npx checkly deploy promotes the same specs to scheduled monitors
  • A monitor change is reviewed like any other code change
  • Native support for GitHub Actions, GitLab CI, and Vercel
.github/workflows/deploy.yml
1# every pull request: run the tagged specs against its preview URL
2- run: npx checkly test --record --reporter=github \
3 --env ENVIRONMENT_URL=${{ steps.deploy.outputs.url }}
4
5# on merge: the same specs become scheduled monitors
6- run: npx checkly deploy --force
7 if: github.ref == 'refs/heads/main'

The part CI never had to give you

In CI a failing spec blocks a merge and waits politely for someone to look at it. In production the same spec has to decide whether to wake a person at 3 a.m., reach the team that owns it rather than whoever is nearest, and tell your customers. None of that existed in your pipeline, and it is the real work of promotion.

  • Retries with backoff, cross-region, so the flake a CI rerun hid does not page anyone
  • Escalation policies: the first failed run, the third, or minutes of sustained failure
  • Groups route to the team that owns the spec, not to one firehose channel
  • The promoted flow can drive the status page it deserves to drive

More on alerting →

__checks__/promoted.check.ts
1// __checks__/promoted.check.ts
2import {
3 PlaywrightCheck, CheckGroupV2, SlackAlertChannel,
4 RetryStrategyBuilder, AlertEscalationBuilder,
5 StatusPageService, Frequency,
6} from 'checkly/constructs'
7
8// The spec is unchanged. This is everything around it.
9const checkout = new CheckGroupV2('checkout-team', {
10 name: 'Checkout',
11 tags: ['team:checkout'],
12 alertChannels: [
13 new SlackAlertChannel('slack-checkout', {
14 url: new URL(process.env.SLACK_CHECKOUT_WEBHOOK!),
15 channel: '#checkout-oncall',
16 }),
17 ],
18})
19
20new PlaywrightCheck('checkout-critical', {
21 name: 'Checkout',
22 group: checkout,
23 playwrightConfigPath: './playwright.config.ts',
24 pwTags: ['@critical'],
25 frequency: Frequency.EVERY_5M,
26
27 // A CI rerun used to hide this. Now it resolves on its own.
28 retryStrategy: RetryStrategyBuilder.exponentialStrategy({
29 baseBackoffSeconds: 10,
30 maxRetries: 2,
31 sameRegion: false,
32 }),
33
34 // Two consecutive real failures before anyone's night ends
35 alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(2),
36
37 triggerIncident: {
38 service: new StatusPageService('checkout', { name: 'Checkout' }),
39 severity: 'MAJOR',
40 name: 'Checkout degraded',
41 description: 'The checkout flow is failing.',
42 notifySubscribers: true,
43 },
44})

It plugs into the stack you already run

Checkly is the synthetic layer, not another silo. Results flow out to your metrics, traces, and incident tooling through native integrations and raw webhooks.

Playwright monitoring

The capability behind this page: your specs, unchanged, running as production monitors.

Learn more

Playwright checks

Point Checkly at your existing playwright.config.ts and pick the tags to promote.

Learn more

GitHub Actions

Gate the merge on the same specs, then deploy them as monitors from the same job.

Learn more

GitLab CI

Same flow if your pipelines live in GitLab.

Learn more

Vercel

Run the promoted specs against every preview deployment automatically.

Learn more

Checkly CLI

Test, validate, and deploy checks from your terminal before they ever reach CI.

Learn more
Slack

Slack

Route each promoted group to the team that owns the spec.

Learn more

Dashboards

Watch pass rate per spec so you can tell a real regression from a flake.

Learn more

Monitoring as code

Monitors reviewed in the pull request that changes the code they cover.

Learn more

Alerting

The part CI never needed: retries, escalation policies, and per-group routing so a promoted spec pages the right team.

Learn more

Status pages

The specs you promoted become the uptime your customers see, on your own domain.

Learn more

Traces

A promoted spec that fails links to the backend span behind it, not just the assertion.

Learn more

Rocky AI

Reads the failed run and its trace, which is how you tell a real regression from a tired selector.

Learn more

Tell a regression from a flake

This is the question that decides whether promotion survives contact with your team. Get it wrong often enough and people mute the channel, which is how the whole effort quietly dies. A promoted spec keeps its evidence, its history, and its trace, so the answer arrives with the alert instead of after somebody reruns it by hand.

  • Pass-rate history per spec: 288/288 reads very differently from 40/288
  • The trace goes past the assertion into the request that actually hung
  • Screenshot and video replay of the run, not of a rerun that passed
  • Rocky AI weighs all of it and says whether this is real

See how Rocky AI reads a failure →

Teams who made the promotion

“The ease of integration in our pipeline was remarkable. We have everything defined in a CI/CD pipeline using GitHub Actions using the CLI.”

Tobias Deekens

Principal Frontend Engineer · commercetools

“Checkly is super easy to use and fits perfectly in our toolchain. We monitor our Vercel projects on preview and production to catch issues earlier and we can trust that we are the first ones to know about problems in production.”

Andreas Wixler

CTO & Founder · FINN

“Checkly was incredibly easy to integrate. Even our initial traces setup was quick, and the documentation was better than OTel’s own.”

Brian Stack

Infrastructure Engineer · Render

“Checkly is incredible: It combines Pingdom, Ghost Inspector and Assertible all in the same app, and the insights are much more detailed.”

Leo Lamprecht

SVP Product · Vercel

“A real advantage that Checkly gives us is that we’re not waiting for our users to report an issue with a product, or waiting on a staff member to file a ticket. Checkly gives us real-time feedback on what is and isn’t working.”

Thomas Reither

Associate Consultant · Solutia

“We’ve been using Checkly for the past months. It’s been phenomenal. Super easy to get setup, works flawlessly and intuitively. The team are super receptive & quick to help.”

Jake Cooper

Founder / Engineer · Railway

Frequently asked questions

Choosing specs, replacing fixtures, cost, secrets, and who owns the alert.

The ones that fail only when a user is genuinely blocked. Sign-in, signup, the core action, checkout. A good filter: if this test went red at 2am, would you want someone woken up? If the honest answer is no, it stays in CI. Most teams start with fewer than a dozen specs and add slowly.

That is the real work of the migration, and it is worth doing deliberately. Replace fixtures with a dedicated probe account that persists, replace hardcoded IDs with lookups, and make the spec clean up whatever it creates so it can run indefinitely. Anything that cannot be made idempotent probably belongs in CI rather than production.

It would if you promoted everything, which is the main reason not to. Promote a small set of high-value specs, run the truly critical ones every five minutes and the rest less often, and keep the long tail in CI where it already runs for free on every pull request.

Encrypted environment variables at the check, group, or account level, referenced from the spec as process.env values. The same spec then runs locally, in CI, and as a production monitor without a credentials file moving around, and rotating a password is a settings change rather than a code change.

Whoever owns the code it covers, which is the point of keeping checks in the service repo. Group promoted specs by owning team and give each group its own alert channels. Decide this during the rollout rather than after the first 2am alert lands in a channel nobody reads.

Retries do most of it. A failed run can be retried with fixed, linear, or exponential backoff, from the same region or a different one, which absorbs the transient failure a CI rerun would have quietly hidden. Then an escalation policy decides whether a confirmed failure is worth a page: the first failed run, the third, or after minutes of sustained failure. If a spec still needs that machinery to stay quiet, it is telling you it was not ready for promotion.

They are the best thing to drive it. You promoted them precisely because they represent a user being blocked, which is exactly the signal a status page should reflect rather than whether the homepage returns 200. Wire a promoted check to a status page service and the incident opens when it fails, notifies subscribers, and resolves when it recovers, all from the same spec that gates your merges.

Yes. This is not a migration away from CI, it is the same files doing a second job. The specs still gate merges exactly as they do now; a tagged subset additionally runs against production on a schedule. There is no second suite to keep in sync because there is no second suite.

Start with twelve specs this afternoon

Tag the flows worth waking someone for, point Checkly at the config you already have, and deploy them from the pipeline you already run.

Start for freeBook a demo