Use case · Reliability for AI-generated code

Your agents ship faster than anyone can review

The bottleneck moved from writing code to being confident in it. Verify what agents ship against production continuously, and give them a monitoring surface they can author, run, and read themselves.

Start for freeBook a demo
Checks as codeMCP serverAgent skills
__checks__/agent-shipped.check.ts
1// __checks__/agent-shipped.check.ts
2import { PlaywrightCheck, Frequency } from 'checkly/constructs'
3
4// Whoever wrote the change — a person or an agent — the same
5// contract holds: it stays verified against production.
6new PlaywrightCheck('critical-paths', {
7 name: 'Critical paths',
8 playwrightConfigPath: './playwright.config.ts',
9 pwTags: ['@critical'],
10 frequency: Frequency.EVERY_5M,
11 locations: ['us-east-1', 'eu-west-1'],
12})

Trusted by teams shipping at machine speed

Vercel
Carhartt
CrowdStrike
Airbus
Fanatics
Mistral
ServiceNow
GoFundMe
Hopper
1Password
Fastly
Total Wine
“A real advantage that Checkly gives us is that we’re not waiting for our users to report an issue with a product, or waiting on a staff member to file a ticket. Checkly gives us real-time feedback on what is and isn’t working.”
Thomas Reither·Associate Consultant · Solutia

Generated code fails differently

Not more often, necessarily. Differently — and in ways that reading the diff is unusually bad at catching.

review queue

Generation outran review

An agent can open more pull requests in a day than a team can meaningfully read in a week. The bottleneck moved from writing code to being confident about it, and review alone no longer closes that gap.

plausible

It compiles, it passes, it is subtly wrong

Generated code fails differently from hand-written code. It is syntactically clean and locally reasonable, and it breaks an assumption three services away that nothing in the diff mentions.

who fixes it

The author is not around at 2am

When an agent wrote the change, there is no engineer holding the context in their head. Whoever is on call starts from zero unless the failure arrives with its own evidence.

For the wider picture — Checkly as monitoring infrastructure that agents operate directly — see AI reliability.

“Checkly Traces helped us resolve issues faster by showing exactly how long database calls are taking. This insight was crucial in pinpointing N+1 issues and optimizing caching. Having a vendor like Checkly levels up your team to the point where it starts to feel like an unfair advantage.”

James Hall

AWS Hero & Founder · Parallax

The monitor ships in the same pull request

Checks are TypeScript in your repo, so an agent that can write the service can write the monitor for it, in the same change, for a human to review next to the diff. Coverage stops being a follow-up ticket that never gets picked up.

  • Agent skills give a coding agent your conventions, not a generic guess
  • The check is reviewed exactly like the code it covers, in one diff
  • A refactor that changes behaviour has to change its assertions to merge
  • Nothing ships to production with monitoring listed as a later task
__checks__/payments.check.ts
1// __checks__/payments.check.ts
2//
3// Generated by the same agent that shipped the payments refactor,
4// in the same pull request, reviewed by a human alongside the diff.
5import { ApiCheck, AssertionBuilder, Frequency } from 'checkly/constructs'
6
7new ApiCheck('refunds-create', {
8 name: 'POST /v1/refunds',
9 frequency: Frequency.EVERY_1M,
10
11 degradedResponseTime: 800,
12 maxResponseTime: 3000,
13
14 request: {
15 method: 'POST',
16 url: 'https://api.acme.com/v1/refunds',
17 body: JSON.stringify({ payment_id: '{{TEST_PAYMENT_ID}}', amount: 100 }),
18 assertions: [
19 AssertionBuilder.statusCode().equals(201),
20 // The behaviour the refactor was supposed to preserve
21 AssertionBuilder.jsonBody('$.state').equals('pending'),
22 AssertionBuilder.jsonBody('$.ledger_entry_id').isNotNull(),
23 ],
24 },
25})

A failure an agent can actually read

A screenshot is for a human. An agent needs the failing check, its trace, and a diagnosis as structured data it can act on. Over MCP it gets exactly that, which is the difference between an agent that reports a problem and one that opens a fix with the failing case attached.

  • Check results, traces, and AI root cause analysis exposed over MCP
  • The same CLI your pipeline runs is the one an agent can run
  • Failures carry the evidence, so the fix starts from the cause not the symptom
  • Humans get the screenshot and the video; agents get the structured data

Watch the full alert-to-resolution loop →

the loop
1# the agent ships, then verifies its own work
2$ npx checkly test --record --env ENVIRONMENT_URL=$PREVIEW_URL
3 ✓ 8 checks passed against preview
4
5# a production check fails an hour after merge
6 ✗ POST /v1/refunds — 500, eu-west-1
7
8# the agent reads the diagnosis over MCP, not a screenshot
9$ checkly mcp get_check_result --id 8f21c
10 root_cause: ledger_entry_id null when payment
11 is captured but not settled
12 trace: api-gateway → payments → ledger
13
14# → agent opens the fix PR with the failing case attached

Write down what must stay true

The most useful thing to monitor in a codebase being edited by machines is its invariants. A refund still produces a ledger entry. An order still reaches fulfilled. These are the assertions you would want a reviewer to make on every diff, and no reviewer has time to.

  • Encode behaviour that must survive any refactor, whoever performs it
  • Generated from a config, so adding an invariant is a one-line change
  • Runs against production continuously, not only at merge time
  • A broken invariant names the behaviour, not just the endpoint
__checks__/guardrails.check.ts
1// __checks__/guardrails.check.ts
2import { ApiCheck, AssertionBuilder, Frequency } from 'checkly/constructs'
3import { invariants } from './invariants.config'
4
5// Behaviour that must survive any refactor, whoever performs it.
6// These are the assertions you would want a reviewer to make and
7// nobody has time to make on every diff.
8for (const rule of invariants) {
9 new ApiCheck(`invariant-${rule.slug}`, {
10 name: `Invariant — ${rule.name}`,
11 frequency: Frequency.EVERY_5M,
12 request: {
13 method: rule.method,
14 url: rule.url,
15 assertions: [
16 AssertionBuilder.statusCode().equals(rule.expectStatus),
17 ...rule.expectFields.map((f) => AssertionBuilder.jsonBody(f).isNotNull()),
18 ],
19 },
20 })
21}

Built for humans and agents

Every surface Checkly exposes to a person is available to an agent as code, CLI, API, or MCP.

AI reliability

The wider picture: Checkly as monitoring infrastructure that agents operate directly.

Learn more

MCP server

Agents read check results, failures, and diagnoses through a typed interface.

Learn more

Agent skills

Give a coding agent the context to author and deploy checks the way your team would.

Learn more

AI root cause analysis

A failure arrives with a read on why, which is what an agent needs to open a fix.

Learn more

Checkly CLI

The same commands your pipeline runs are the ones an agent can run.

Learn more

Monitoring as code

Checks arrive in the pull request, so a generated monitor is reviewed like generated code.

Learn more

Traces

Connect a failing check to the span three services away that the diff never mentioned.

Learn more

GitHub Actions

Gate agent-authored pull requests on the same checks that monitor production.

Learn more

Dashboards

Watch whether reliability moves as generation volume goes up.

Learn more

Evidence, in a form either can use

When nobody on call wrote the change, the failure has to explain itself. Every run keeps the evidence a human needs to judge it and the structure an agent needs to act on it.

full screenshotvideo replaytraceAI root cause analysisMCP

See how AI analysis reads a failure →

Teams building with Checkly

“Checkly is a fabulous developer tool! The flexible features and developer-friendly API made the integration super easy. Bonus: their support is super friendly and knowledgeable!”

Connor Hicks

Lead Developer · 1Password

“Checkly CLI has enhanced our engineering team’s ability to quickly build, validate and deploy an entire suite of checks from their local development environment.”

Tavares Chambless

Manager, Quality Assurance · Loyal Health

“The ease of integration in our pipeline was remarkable. We have everything defined in a CI/CD pipeline using GitHub Actions using the CLI.”

Tobias Deekens

Principal Frontend Engineer · commercetools

“Checkly was incredibly easy to integrate. Even our initial traces setup was quick, and the documentation was better than OTel’s own.”

Brian Stack

Infrastructure Engineer · Render

“Checkly is super easy to use and fits perfectly in our toolchain. We monitor our Vercel projects on preview and production to catch issues earlier and we can trust that we are the first ones to know about problems in production.”

Andreas Wixler

CTO & Founder · FINN

“We’ve saved a lot of time by using Playwright with Checkly.”

Keegan Donley

Senior Front-End Engineer · Kizen

Frequently asked questions

What changes with generated code, what agents can do themselves, and what review is still for.

The practice is the same; the volume and the failure mode are not. Generated code arrives faster than review scales, and it tends to fail by being plausible rather than by being obviously broken — correct in the diff, wrong against an assumption elsewhere. Continuous verification against production catches the class of problem that reading a diff does not.

Yes, and it is the natural place to start. Checks are TypeScript in your repo, so an agent that can write a service can write the monitor for it, in the same pull request, for a human to review alongside the diff. Agent skills give it the conventions to do that the way your team would.

Through the MCP server, rather than by looking at a screenshot. It can pull the failing check, its result, its trace, and the AI root cause analysis as structured data, which is what it needs to open a fix with the failing case attached rather than a guess.

The same way you stop them generating hundreds of low-value tests: review. Checks arrive as code in a pull request, so a generated monitor is subject to exactly the review a generated function is. Keeping monitors in the repo is what makes that possible at all.

Invariants. The behaviour that must survive any refactor regardless of who performs it: that a refund still produces a ledger entry, that an order still reaches fulfilled, that authentication still returns the right account. These are the assertions you would want a reviewer to make on every diff and nobody has time to.

No, and treating it that way would be a mistake. It changes what review is for. Reviewers keep judging design and intent, while the question of whether the change actually works in production gets answered continuously by something that never gets tired at 5pm on a Friday.

Ship at machine speed. Verify at machine speed.

Give your agents a monitoring surface they can author, deploy, and read — and give your team the confidence to let them.

Start for freeBook a demo