Checkly Blog: Monitoring Insights & Trends

We Let AI Agents Rewrite a 92M-Message-a-Day Service in Go. Zero Incidents.

Edvinas Janusevicius — Tue, 01 Sep 2026 00:00:00 GMT

Our Results Daemon processes about 92 million messages a day. We recently rewrote it from Node.js to Go, and we let Claude Code write it.

We wanted to know whether we could trust an agentic rewrite for a critical, high-throughput production service rather than a prototype. It shipped with zero incidents, a 70% reduction in running pods, and a lighter database load. Go's stronger type system also proved a better fit for agents than JavaScript, adding protection against regressions and letting us ship faster and with more confidence.

What made it work was the test harness we built before the agent wrote a line. Here's how we designed it, and the principles you can reuse on your own legacy services.

The problem

Checkly is a monitoring platform that runs synthetic checks, automated scripts that emulate real users, and uptime checks that confirm a system component is operational. A runner component executes all of these and produces a result that has to be processed, stored, and alerted on.

Since we introduced uptime checks, and with the company's overall growth, the volume of checks run on our platform has doubled over the last year. Some components started degrading under that load. The most notable one was Results Daemon, a Node.js component written in vanilla JavaScript.

Results Daemon is a background worker. It consumes results from our runner, writes them to databases, determines the check outcome, issues alerts, and schedules retries as needed. It also publishes WebSocket updates to our CLI and UI. In total, this component processes approximately 92,000,000 messages every day, around 40,000,000 of them check results and the rest WebSocket publishes.

At that scale, it was becoming a bottleneck. It paged our on-call engineers more often, and limited type safety made every change harder to land safely. So we decided to rewrite Results Daemon in Go using agentic engineering.

Designing a test harness

We built the harness before we started the rewrite. If an agent is going to write the code, something other than a human reviewer has to define what correct means.

We built it on these design principles:

The harness tests the component as a black box. There is zero coupling between the code or language of the system under test and the harness itself.
Every test case provides an input and expects a deterministic output, with all outputs recorded in "golden files." These are generated against the legacy system and later used by the rewrite to assert byte-to-byte parity.
Non-deterministic fields, such as UUIDs or timestamps generated during the test itself, are written as <uuid> or <timestamp> and are still type-checked, to minimize the risk of differences in behavior slipping through.
Surrounding components (databases, queues, caches, other services) are categorized as boundaries. These are managed strictly by the harness, and the system under test is only pointed at them using environment variables.
Boundaries use real instances of the service in testing. If data is written to PostgreSQL, the harness uses a real PostgreSQL container rather than an emulated one.
For simpler boundaries such as SQS queues, we built our own emulator instead of using ElasticMQ or LocalStack. We found it more performant and simpler, both in our tests and in our assertions.
A set of "oracle" classes fetches the test output and asserts whether the test failed. For example, PostgresOracle.expectResultToMatchSnapshot(testId) fetches the relevant output and asserts its byte-level accuracy against the established golden file.

Three technical choices carried the harness:

Playwright, a testing framework built for reliability, with strong tooling for network interception and parallel test execution. Our own synthetic monitoring offering is built on Playwright, so we already knew it well, and its black-box model fit what we were doing here.
Docker Compose, the simplest way to start and tear down containers for our boundaries, both locally and in CI.
Toxiproxy, a TCP proxy that emulates network conditions and let us test how the system behaves when surrounding infrastructure fails.

Building test cases

With the architecture settled, the next step was building actual test cases. Results Daemon's outputs depend on two things:

The check result, the data object representing the outcome of a check execution. It holds the outcome state of the check run (Failed | Degraded | Success) and other metadata.
The check configuration at the time the result was received: retry rules for rescheduling, alert rules for notifications, and so on.

From there it followed quickly that behavior coverage depends directly on the diversity of the inputs. A harness that covers every combination of check result and configuration covers every possible code path, with zero coupling to the implementation. That gave us our first principle:

The quality of the harness depends on the quality of the inputs you can provide. The more diverse and realistic the inputs, the greater the coverage of code paths and behaviors. In our case, the number of outcomes is represented by (accountConfigs × groupConfigs × checkConfigs × resultOutcomes).

We generated those inputs from our internal data lake. We extracted all account, group, and check configurations along with every result outcome from the last 24 hours, which is the longest interval we schedule checks at. We loaded all of it into a ClickHouse instance, and each dataset was "collapsed" into a unique set of configurations and outcomes, each tagged with its number of occurrences. We then used that data to seed realistic scenarios and pin them to business rules. For example, a re-dispatch takes its runtime from the account when the job pins none.

After generating the test cases, we used code coverage reports to estimate how effective the harness was, targeting between 90% and 100%. We also fed those reports back to the agent to review, identify gaps, and propose test cases for anything missed. We expected some gaps to remain, but with all relevant files sufficiently covered and every scoped feature included, we considered the harness ready for a test run.

Code coverage is a great metric to track as you start out with your initial set of test cases, since high code coverage means the core behaviors are covered. However, it does measure direct system actual behavior and is therefore likely to miss edge cases.

We also built a separate suite of tests that caught failure modes when infrastructure failures occur, e.g., PostgreSQL going down, using Toxiproxy. These were focused on how behavior changes and what data is lost when a piece of infrastructure goes down.

The agentic rewrite

A prerequisite for this approach was an earlier migration to a monorepo running on Tilt, which lets us spin up the whole platform end to end in a local development environment.

We dispatched an instance of Claude Code, using Fable, and gave it one instruction: "build a Go service that consumes data from input queues, processes the messages, and writes the outputs to downstream applications such as databases, caches, and other queues," with the legacy implementation available as a reference. The main acceptance criterion was that the test harness passed against the new implementation.

The agent ran overnight and produced a deployable service of about 13,000 lines of application code, architecturally mirroring the legacy implementation. It also kept token usage within the daily limits of a $200 subscription.

We ran an earlier attempt with the same instructions using Opus. That implementation did not meet our bar and was discarded.

After reviewing the implementation, we deployed the application to every environment except production, using an internal account to push results so we could get feedback quickly. The review also surfaced anti-patterns we wanted gone and improvements we wanted in:

Removing configuration generated at runtime. The legacy application derived its configuration from partial values provided via environment variables. That has been a pain point in the past when debugging and making configuration changes, especially under pressure during incidents.
Improving end-to-end observability. The legacy service had high-level observability, but given the increase in throughput, there was clear room to do better.

We implemented both with human supervision, which gave us two more principles:

Human intervention is sometimes required, especially to find and fix anti-patterns an agent inherited from the legacy application or introduced itself. For us this meant removing all configuration derivations and making static environment variables the only way the application is configured, plus improving observability. That improved the system's operability and simplified the code in both the rewrite and the harness.
Take the rewrite as an opportunity to improve things. In our case that meant expanding end-to-end observability. At this message volume, recording low-level metrics such as database connection utilization, CPU, memory, and timings of individual code execution steps was a necessary addition for long-term operational excellence.

With those in place, the next step was production.

First deployment: what we got wrong

Before deploying, we had to decide how to migrate customers. The simplest approach was to spin up separate infrastructure (queues) and patch the consumers to push data to the new queue for accounts with a specific feature flag enabled.

Once the infrastructure was ready, we deployed the Go application across all environments and migrated internal accounts. We ran into issues almost immediately, mainly retries failing.

The root cause was a critical gap in the harness: our local environment's surrounding infrastructure did not match production. The harness modeled a simplified local queue topology instead of the real one. Locally, retries were routed to 3 queues based on check type alone. In production, routing also depends on factors like priority and hosting type, for a total of 18 possible queues per region. Because the harness assumed the local topology, the agent built its retry logic around it, and the gap made its way into the new application.

We made that mistake during design. The initial boundary classification was too high-level and assumed production behavior that proved wrong.

We updated the harness to align with production, removed all code branches that only ran in the local environment, and followed up with a human-supervised refactor of the retry module. A day later, the daemon was processing results for our internal accounts, around 3% of total platform load.

The retry-queue gap left us with three principles for the harness going forward:

The harness environment must align with production as closely as possible.
Every boundary must be established down to its lowest unit. Every single queue, every DB table, accounted for.
Every assumption about the surrounding infrastructure must be written down and verified, not inferred.

Rolling out to customers

The migration strategy was deliberately boring. Before migrating any customer, the new daemon had already been through the harness and a full rollout across our own internal accounts on real production traffic. On top of that, we migrated customer accounts in stages to reduce blast radius, so each cohort benefited from what we learned on the previous one:

Free accounts, on a trial period or hobbyists.
Paid accounts, on a monthly subscription.
Enterprise accounts, with signed enterprise deals.

Migrating a cohort meant flipping a feature flag, which instantly switched their traffic from the legacy daemon to the new one. After each flip, we monitored for 24 to 48 hours to confirm no results were dropped, failure rates across the platform stayed consistent, and no support escalations pointed back to the daemon before moving on.

During this period, both daemons were processing real traffic, which meant both had to stay in sync. A change in one needed to land in the other. To enforce that without slowing anyone down, we updated CI to run the harness against both the legacy application and the new daemon, blocking any pull request that failed for either.

When customers found or flagged bugs, we used a simple TDD loop to fix them: reproduce the gap in the harness first, then fix the underlying issue. That let us close an issue in about an hour, including CI and deployment time.

The issues we hit were minor, and nearly all of them were edge cases the harness hadn't covered. Beyond those, none of our customers noticed they had been migrated.

After a week of migrating customers, monitoring dashboards, and fixing small bugs, the migration was complete, and we decommissioned the legacy workload.

Results

Better database performance

The tooling we chose to talk to the database, sqlc, produced more performant queries.
Row locking dropped because Go's concurrency model executes operations inside transactions faster.
Together, that gave us 60% fewer total average active sessions (AAS) on our database and a roughly 15% reduction in database CPU.

Improved efficiency and operability

The new daemon freed up around 15 vCPU and 45GB of memory, as the new application needed far fewer pods to support the workload.
The additional observability built into the new application helps with triaging during any alerts or incidents and enables us to make informed decisions when refactoring the code.

Improved developer experience

Go's type system works much better with agents and adds another layer of protection against regressions.
The test harness guarantees consistent behavior before any rollout. That confidence shows up as faster feature and bug-fix shipping, more frequent deployments, and fewer alerts for our on-call engineers.

With a harness that validates behavior at this level of rigor, we can now ship agent-written code faster than before, trusting the harness to catch regressions instead of relying on manual review alone.

Summary

Rewriting legacy applications is never easy, even with agents doing the heavy lifting. We got there by designing and building a strict test harness first, then letting the agent work inside it. Rigorous testing, strict boundary definition, and knowing when to step in as a human are what made this work, and those principles will hold up as the tooling keeps changing.

If you want to see the testing philosophy behind this in a product rather than a blog post, it is the same one behind Checkly: Monitoring as Code, real Playwright scripts running against production, and results you can trust enough to act on. Start monitoring for free, or read the docs to see how it fits your stack.

]]>

Metabase Security Incident

Mon, 10 Aug 2026 00:00:00 GMT

On 3 August 2026, an attacker exploited a zero-day vulnerability in Metabase, the third-party analytics tool we use internally, and gained read access to a database holding a copy of Checkly operational data. The attacker bypassed authentication and obtained an administrator session on our Metabase Cloud instance. Metabase has since blocked the attack, patched the vulnerability, and published a security update. The attacker did not access our production platform.

If you put authentication values directly in your check configurations, or you use OTEL API keys for trace collection, or you want to be extra safe about potential data being accessed in an encrypted state, please rotate your credentials now.

The rest of this post explains what was exposed, what was not, and what we are doing about it.

We are sorry this happened. The vulnerability was in a vendor's product, but protecting your data is our job, and this incident put some of it at risk.

What happened

Metabase is a business intelligence tool we use to analyze product usage and operational metrics. It connects to a data warehouse that holds a copy of Checkly operational data, separate from our production platform.

The zero-day allowed the attacker to create an administrator session without valid credentials. No password was stolen, and no Checkly system was directly breached. With that session, the attacker queried the connected data warehouse through Metabase for about 26 minutes on August 3rd, 2026. All activity was read-only.

Metabase discovered the attack against Metabase Cloud the same day, blocked the endpoints the attacker used, and patched the vulnerability. They notified customers, law enforcement, and engaged a third-party forensics firm on August 6th. On August 10th, we concluded our investigation of the attacker's activity in our instance using detailed logs Metabase provided to complete our own analysis of the affected systems.

What may have been exposed

The data warehouse contained check configuration and account data, potentially including authorization keys or headers if your checks send them. Specifically:

Credentials you store as Checkly secrets are encrypted at rest and injected only at run time. These were not exposed in plaintext.
Checkly stores custom values added directly in a check's configuration (custom headers, cookies, query string parameters, or authorization headers) in plaintext. You should consider these values exposed.
Cryptographic hashes of OTEL API keys used for trace collection were also accessed.
Encrypted tokens used in integrations with Vercel, Coralogix, Slack, and Prometheus.

What was not affected

Access was read-only and lasted about 26 minutes on 3 August. The attacker did not modify, disable, or create any checks, alerts, or settings.
The attacker made no changes to your account, users, or API keys, and created no new ones.
The attacker did not access our production platform; neither the production database nor the Checkly platform, including check execution infrastructure and alerting, were ever breached.
We reviewed our cloud audit logs and found no evidence of anyone reaching the data warehouse from outside our known analytics tooling.

What you need to do now

Rotate any credentials and secrets added directly to a check configuration (custom headers, cookies, query string parameters, or authorization headers).
Rotate all OTEL API keys in use.
Review the systems those credentials protect for unauthorized use since 3 August 2026.
Reauthorize Vercel, Prometheus, Slack, and/or Coralogix integrations

If your checks only use Checkly secrets for authentication, you do not need to rotate them. For future security, we recommend storing secrets and sensitive data with Checkly Secrets.

What we did

We rotated the credentials Metabase uses to reach the data warehouse.
We reset all Metabase user passwords, removed the compromised administrator account, and rotated all Metabase API keys.
We audited the admin accounts on the instance and confirmed the attacker created none.
We verified through cloud audit logs that only our known analytics tooling used the warehouse credentials.
We are contacting affected customers directly.

What we are changing

Rotating credentials fixes the immediate problem. It does not fix the reason this hurt: our analytics environment held more sensitive data and had broader access than it needed. So we are making these changes:

Extensive audit of analytics and processing tools to limit the exposure surface of any potentially sensitive data.
Improve our sanitization process when storing check configurations and data for analytic processing by scoping down data access so only required information is accessed.
Vendor security notices will page our on-call engineers directly.
Future product features around proactively communicating to customers when potentially sensitive data is exposed in their check configurations.

We will update this post if Metabase's ongoing forensic investigation or our own review turns up anything that changes the guidance above.

If you have questions or see suspicious activity on systems your checks touch, contact us at support@checklyhq.com.

Hannes Lenke CEO, Checkly

Frequently Asked Questions

Am I affected?

Sensitive data from your account was potentially exposed if:

You hard-coded sensitive information in check configurations, like API check custom request headers, query string parameters, authorization headers / cookies, and in Browser check and Multistep check scripts.
You set up an OpenTelemetry integration with Checkly. Cryptographic hashes of OTEL API keys used for trace collection were accessed.

You should:

Rotate any credentials and secrets added directly to a check configuration (custom headers, cookies, query string parameters, or authorization headers).
Rotate all OTEL API keys in use.
Review the systems those credentials protect for unauthorized use since 3 August 2026.
Reauthorize integrations (including Vercel, Prometheus, Coralogix, etc)

Were environment variables and secrets exposed?

Per-check: created and saved in the check configuration

Plain environment variables: name, metadata, and values were exposed.
Locked environment variables: name, metadata, and encrypted values were exposed. Values were not exposed in plain text.
Environment secrets: name, metadata, and encrypted values were exposed. Values were not exposed in plain text.

Per-group: created and saved in the group configuration

We do not have any evidence that the attacker queried for the group configuration data. However, these were accessible via Metabase, which the attacker had access to, so we recommend rotating those credentials to be on the safe side.
Plain environment variables: name, metadata, and values were potentially exposed
Locked environment variables: name, metadata, and encrypted values were potentially exposed. Value was not exposed in plain text.
Environment secrets: name, metadata, and encrypted values were potentially exposed. Values were not exposed in plain text.

Global account environment variables: created for your entire account on the ‘Global environment variables and secrets’ page

Plain environment variables: Name and metadata exposed. Values were not exposed, as they were not accessible via Metabase.
Locked environment variables: Name and metadata exposed. Values were not exposed, as they were not accessible via Metabase.
Environment secrets: Name and metadata exposed. Values were not exposed, as they were not accessible via Metabase.

For all encrypted values that were potentially exposed, we recommend rotating those credentials to be on the safe side and protect against decrypting them in the future, once more sophisticated mechanisms like quantum computing are available.

Were service API keys, user API keys, or private location API keys exposed?

No. However, we recommend you rotate them to be on the safe side.

Do I need to change my Checkly account password?

No, these are stored on a completely separate system and were not exposed.

Were alert channels exposed?

No. Webhook URLs, Slack tokens, PagerDuty keys, and alert email addresses were not exposed. We recommend rotating those credentials to be on the safe side.

Were Playwright and other scripts exposed?

A subset of API check setup and teardown scripts, Browser check scripts, and Multistep check scripts were exposed.

You should rotate any hard-coded sensitive information in those scripts.

Playwright Check Suite scripts were not exposed since they were stored in a separate system and not accessible in Metabase.

Plain request details were exposed. Other result details and artefacts were not exposed.

Do I need to rotate API check ‘basic authentication’ information?

If you added basic authentication credentials to your API check using our dedicated ‘Basic Authentication’ feature, the usernames are stored as plain text but the passwords are encrypted.

We recommend rotating those credentials to be on the safe side and protect against decrypting them in the future, once more sophisticated mechanisms like quantum computing are available.

How do I rotate the Prometheus integration?

On the Integrations page, remove the integration entirely with the ‘Remove Prometheus integration’ button. Then, create the integration again. This will generate a new bearer token for you to use.

We have an SSO connection with Checkly. Do we need to rotate secrets or certificates associated with that?

No, these are stored on a completely separate system and were not exposed.

]]>

CLIs are more token-efficient than MCP. Or are they?

Stefan Judis — Thu, 30 Jul 2026 00:00:00 GMT

MCP servers have a reputation: they eat your context window. CLIs paired with skills, on the other hand, are more token efficient. But is this still true? I dropped all my MCP servers five months ago. Five months is a long time in AI land.

When Anthropic came up with the concept of skills, many people stopped using MCP servers in favor of CLI tooling and skills. CLIs integrate nicely with coding agents and generally skills were supposed to be lighter on tokens because things were pulled in lazily and on demand. CLIs plus skills became the default advice for coding agents, and MCP got labeled as the token-hungry way to do the same thing.

We at Checkly are heavily involved in the Playwright ecosystem. Playwright offers an MCP server and a CLI, too. There's this "famous" YouTube video from the Playwright team making exactly the case for the CLI, and I bought it, and switched from MCP to the CLI.

The video was released in February 2026 and it aged poorly. There's no big difference in using CLIs or MCPs these days.

Why is MCP supposed to eat up your context window?

There were two real problems if you tried MCP servers in 2025.

The major problem: the moment your agent connected to an MCP server, it pulled every tool definition into context. All of them, upfront, whether the conversation would ever touch them or not. For the Playwright MCP server that meant 48 tool schemas at 5.9k tokens before you typed a single prompt. This was the problem skills were supposed to solve.

And Playwright is a mild case. If you wanted to use GitHub's MCP server it wasted 17.8k tokens "for nothing".

If you switched to skills, agents only loaded the skill files' one-line description (roughly 30-50 tokens) and read the full files if the agent actually needed them. For established tooling like the gh CLI, you often don't even need a skill. Modern LLMs know their way around it pretty well with common CLI tools.

The second problem, and this one might be a bit more Playwright specific, was that every single MCP action, every click and every keystroke, came back with the entire page snapshot inlined. The context window was filling up with YAML wall after YAML wall. This is the part that produced the scary comparison numbers floating around.

So yes, back in 2025 the criticism was fair and it seemed like a no-brainer to use CLIs.

What changed since then

Both problems got fixed. On the upfront side, modern agent harnesses defer MCP tool loading and optimize tool calls with some tricks. Claude Code, for example, only keeps the tool names around and pulls a full tool definition in when a tool is actually called. Anthropic digs into the underlying idea of reading tool definitions on demand in "Code execution with MCP: Building more efficient agents". The key fact is that MCP tool definitions get pulled in lazily now, similar to how CLI skills work, and stay out of your context window until the agent actually needs them.

And on the per-action side, an MCP specification change allowed Playwright to stop inlining snapshots. Previously, the MCP spec was fairly strict about the communication flow. All content had to be included in the tool call results. This is the reason why the MCP always returned the current page snapshot. But with resource links the MCP server changed its general behavior and now writes snapshots to disk, responding with a resource URL. If the agent decides that it needs the current page snapshot it can go ahead and read it. But it won't be included if it isn't needed.

Writing page state to disk and letting the agent grep into it is exactly the pull model the CLI was praised for. By now, playwright-cli commands literally invoke the MCP tools by name under the hood (click calls browser_click, find calls browser_find). The Playwright CLI and MCP use the same backend, same snapshot files on disk.

There's no difference anymore?

But let's look at some numbers. I gave the same task to two fresh Claude Code sessions, one with the Playwright MCP server, one with only the playwright-cli and the installed CLI skill:

Please open my example demo shop and search for "liquid". Click the first product and add the product to the cart. Validate that the cart is showing the correct amount and close it.

The result after running the same task three times was that MCP and skill-based CLI flows have more or less the same token consumption. The MCP session ended at 48k to 50k tokens of context, the CLI session at 45k to 48k. I didn't go much deeper but while seeing what both agent flows were doing it became obvious that they're both doing the exact same thing. There might be small differences, but the CLI output and MCP tool results look almost identical, so I didn't see much value in running the experiment again. I also ran this comparison live in our Playwright webinar; if you want to watch the measurement happen in real time, the recording is on-demand.

AI advice comes with an expiry date

It's exhausting but every piece of advice in the AI or agent space has a shelf life measured in months. "MCP loads all tools upfront" was true, then harnesses shipped deferred loading. "Playwright MCP inlines a snapshot per action" was true, then snapshots moved to disk. The viral Playwright video that made me switch was accurate when it was recorded and stale within a quarter.

So, "don't use MCP, it's too token hungry" is old news.

If a product provides a CLI and MCP, compare features and functionality first, because they often don't match. Take our own tooling at Checkly as an example: the Checkly CLI is built for the Monitoring as Code flow to create, test and deploy your monitoring from a local project. The Checkly MCP server is for live account operations like inspecting check health, investigating failures and kicking off Rocky AI root cause analyses.

Only when both sides genuinely match, like Playwright's do, does it come down to preference. And yes, I've since brought some of my MCP servers back. I prefer to use MCP in Claude Desktop and CLIs in Claude Code these days. But it's really up to you!

The only thing to keep in mind is that if there's a "common" best practice about token consumption, check your context window every now and then. What was true a couple of months ago probably isn't anymore.

]]>

How Upstash Monitors Every Redis Replica with Checkly

Ilter Kavlak — Fri, 10 Jul 2026 00:00:00 GMT

This is a guest post by Ilter Kavlak, Site Reliability Engineer at Upstash. It's a companion to his post on the Upstash blog, which covers the internal side of the same monitoring stack.

There's a support ticket every SRE dreads: "is something wrong with my database?" The outage is bad enough. Worse is the possibility that the customer knew first.

At Upstash, we treat that scenario as two failures rather than one: the incident itself, and the uptime monitoring gap that let a customer beat us to it. We write a postmortem for the gap, too.

This post is about the layer we built to make sure that never happens: a fleet of Checkly checks that watches our platform from the outside, the way a customer would. One check per replica, running every minute from 18 locations around the world, all of it defined in Terraform. I'll walk through what we check, how a failing check reaches a human, and how we monitor a product where "reachable" and "working" are very different claims.

Why every replica gets its own uptime check

Upstash Redis is multitenant and multizone. Every database is backed by at least two replicas, and with the Prod Pack those replicas live in different availability zones. When a replica dies, the survivors elect the most up-to-date among themselves and the database keeps serving; customers notice nothing.

That resilience is exactly what makes replica-level monitoring easy to skip. If your checks only watch the database endpoint, a dead replica is invisible: the endpoint stays green while you quietly run with less redundancy than you promised your customers. And the next failure is no longer a non-event.

So our rule is blunt. A dead replica is an incident even when no customer can feel it. Every single replica gets its own uptime check, and losing any one of them wakes the oncall.

A couple hundred URL monitors, all in Terraform

Because Upstash Redis speaks HTTP alongside the Redis protocol, health checking stays simple. No Redis client, no connection pooling, no scripted scenario. Each replica exposes a /ping endpoint over HTTPS, and a 200 means that replica is serving its databases the same way it would serve a real client.

We use Checkly URL monitors for these, defined in Terraform. Each replica is one module block:

module "check-eu-central-1-r2" {
  source = "../modules/checkly_url_monitor"
  name              = "eu-central-1-r2"
  base_url          = "https://<replica-endpoint>.upstash.io/ping"
  frequency_minutes = 1
  locations         = var.default_locations
  alert_channel_ids = local.default_alert_channel_ids
}

This is Monitoring as Code, and it's the feature that made us commit to Checkly. Our monitoring configuration lives in the same repository as the rest of our infrastructure, and a single apply deploys or updates the whole fleet. By now, the file reads like a census: one block per replica, a couple of hundred in total, spanning AWS and GCP regions from Tokyo to São Paulo. When a new replica joins a cluster, a new block gets written, reviewed, and applied like any other infrastructure change. No monitor is ever created by clicking around a dashboard; if it isn't in git, it doesn't exist in Checkly.

That discipline would be optional at two checks. At two hundred, it's the only way to be sure nobody forgot one.

Redis isn't the only product in that file. Upstash Vector and Search replicas get the same per-replica treatment, except their checks go beyond a ping: each one upserts a record and runs a query against it, because an index that accepts connections but can't answer queries is not "up" in any sense a customer cares about. Keep that idea in mind for the QStash section below.

From 18 locations, every minute

The hardest-working parameter in that module block is locations. Upstash serves clients globally, with replicas placed close to their readers, so a replica answering from one vantage point proves very little. Every monitor runs from all 18 of the Checkly locations we've configured, covering North and South America, Europe, Africa, and Asia-Pacific. A routing problem that only affects clients in Singapore is a full outage for a client in Singapore, so our monitoring treats it as one.

Each monitor runs every minute, and a failed request is retried after a short backoff before anything alerts, since a single dropped packet somewhere on the public internet is no reason to wake a human. And we refuse to conflate slow with down: a response over five seconds marks the monitor degraded, and past ten it counts as failed. Two different signals, routed two different ways.

How a failing check reaches a human

Every monitor is wired to the same set of alert channels, also defined in Terraform:

Opsgenie pages the oncall on failure. This is the loud one.
Slack receives both failures and recoveries, giving the rest of the team context without waking anyone.
Degraded-state notifications land in a Slack channel of their own. Slowness deserves eyes but not a 3 a.m. page, and it shouldn't drown in a busy alerts channel either.

When a replica check fails, the oncall is looking at it within a couple of minutes, whatever the hour. None of that is remarkable. The habit worth writing down is how incidents end: we don't call anything resolved because internal dashboards look healthy. Resolved means the Checkly check is green again, from all 18 locations. Internal metrics can tell you a process is running; only the external check can tell you a client on the other side of the planet is actually being served. The second claim is the one customers experience, so it's the one that closes the incident.

QStash: proving messages actually get delivered

For QStash, our message queue and task scheduler, a health endpoint would prove almost nothing. QStash exists to deliver messages; it could happily return a 200 for every publish, deliver none of them, and a ping check would stay green the whole time.

So instead of a ping, QStash gets a Checkly API check that uses the product the way a customer would. The check publishes a real message through the QStash publish API and points it at an Opsgenie heartbeat endpoint, so every successful delivery resets the heartbeat's timer.

One message tests the whole pipeline, and the two ways it can fail are distinguishable:

The publish fails. QStash is unreachable or rejecting requests. The Checkly check turns red and pages the oncall.
The publish succeeds but nothing arrives. QStash accepted the request and then failed to deliver it. The heartbeat falls silent, Opsgenie notices the silence, and the page goes out.

Either way a human is investigating within minutes, and the alert already names which half of the flow broke before anyone opens a dashboard. That's a serious head start.

My favorite property of this check is that it's indistinguishable from a customer. Same publish API, same delivery path, same retries as real traffic. There is no separate test mode that could pass while production fails.

What Checkly doesn't cover, on purpose

External checks answer one question: would a customer be served right now, from where they are? Two other layers answer the questions they can't.

Internally, a federated Prometheus stack watches the clusters themselves and catches pod-level trouble. And in every region, a fleet of benchmark clients (deployed deliberately outside the clusters they measure) pushes a steady mix of reads and writes around the clock, recording every latency it observes. That fleet is our second, independent paging path, and its historical dashboards are where slow-moving problems like release regressions and noisy neighbors show up as trend lines long before they'd trip any single alert. Both layers are managed the same way as the Checkly monitors: Terraform modules in the same repository, one workflow for everything. I've written about that side of the stack in detail on the Upstash blog.

Layer	Question it answers	Who gets woken
Checkly per-replica checks (Redis, Vector, Search)	Is each replica serving traffic, from every region, right now?	Oncall
Internal cluster monitoring	Is the software healthy on the inside?	Oncall
Continuous latency benchmarks	Is each replica as fast as it should be?	Oncall on failure, internal alerts on drift
QStash end-to-end check	Does the product do its real job, end to end?	Oncall, with the broken half already identified

Every layer exists to cover a blind spot in the others. But when internal and external monitoring disagree, the external answer wins the argument, because it's the only one describing what customers actually experience.

The philosophy, in five sentences

Watch every replica, because redundancy you can't verify isn't redundancy. Watch from wherever your users actually are. Treat ‘slow’ as its own signal rather than a milder form of down. Test the thing the product promises to do, not just whether it answers. And don't declare victory until the outside world agrees.

None of this prevents outages; hardware still fails, zones still have bad days, and the internet remains the internet. What it buys is time and clarity: minutes from failure to a human engaged, an alert that names what broke, and a client's-eye definition of "fixed." The ticket asking "is something wrong?" should only ever get one reply: "yes, we know, and we're already on it."

Upstash runs a couple hundred Checkly monitors, every one of them defined in code. If you want the same workflow, start with Monitoring as Code: define your monitors in TypeScript with the Checkly CLI, or use the Terraform provider like Upstash does. Start monitoring for free.

]]>

Autoscaling Checkly Private Location Agents in Kubernetes with KEDA

Edvinas Janusevicius — Thu, 02 Jul 2026 00:00:00 GMT

Monitoring load is not always steady. A team might add a new batch of checks or run several ad hoc tests during a rollout. When that happens, your Private Location agents need to pick up more work at once.

If there aren’t enough agents available during a burst, checks start piling up in the queue, which can delay or disrupt check execution. But solving this by running a high number of agents around the clock has the opposite problem: most of that capacity sits idle until the next busy period.

Autoscaling solves both ends of that problem. You scale agents up when checks queue, scale back down when the burst clears, and stop guessing in between. This guide covers two things that landed together: the new Private Location metrics that show you whether your agents are running efficiently, and the KEDA setup that automatically scales your agent deployment.

How do you know if a Private Location has enough agents?

You can now monitor Private Location health with Checkly, so capacity stops being a guessing game. Open any Private Location in the UI and you'll find three metrics that tell you exactly how your agents are doing:

Queue duration: how long check runs wait before an agent picks them up. This is the number to watch. If it creeps up, your agents may be falling behind on work.
Agent count: how many agents are currently reporting for the location.
Scheduled check runs: how many runs are queued for the location right now.

These metrics are also available via our Prometheus v2 integration.

Together, these signals help you understand whether your agents are healthy and whether your Private Location has enough capacity. For example, if two agents are reporting but queue duration keeps climbing and scheduled runs are stacking up, you may be under-provisioned. If ten agents are reporting and queue duration stays near zero all day, you may be running more capacity than you need. Those are the two states autoscaling exists to fix, and now you can see which one you're in.

Scaling on the right signal

Checkly agents are stateless: they sit in your network, pull scheduled checks from a shared queue, run them, and send the results back. You can add or remove agents freely without losing or duplicating work. That's the property that makes scaling safe.

To develop a practical approach to autoscaling, we studied a range of large Private Location setups across customers, including DVAG, LinkedIn, Sherwin-Williams, and Auto Trader. Looking at how those fleets handle different checks and workloads helped us identify the signal that best reflects a location’s current demand.

For teams running agents on Kubernetes, that signal is checkly_private_location_check_runs. This gauge reports the number of check runs routed through a Private Location, broken down by state.

Two states matter for capacity:

queued: the check run is scheduled, but no agent has picked it up yet.
inflight: an agent is running it right now.

Add those two and you have the total live load on the location, which is the signal you want driving replica count. To use this metric, make sure Prometheus V2 metrics are enabled for your account.

ℹ️ If your existing KEDA config still uses

checkly_private_location_oldest_scheduled_check_run, consider switching to checkly_private_location_check_runs with the queued and inflight states. The older metric is useful for spotting backlog, but queued plus inflight gives KEDA a more direct signal for current load.

Setting up KEDA

KEDA (Kubernetes Event-Driven Autoscaling) reads a Prometheus metric and scales a Kubernetes deployment up or down based on that value. You'll need KEDA installed in the cluster, your agents deployed via the Checkly agent Helm chart, and Prometheus V2 enabled for your Checkly account.

Here's a ScaledObject with sensible defaults. Adjust the bounds to your workload:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: checkly-agent-autoscaler
spec:
  scaleTargetRef:
    namespace: <namespace_for_agent_deployment>
    name: <agent_deployment_name>
  minReplicaCount: 2
  maxReplicaCount: 10
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
        scaleDown:
          selectPolicy: Min
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: checkly_private_location_check_runs
        threshold: "1"
        query: sum(checkly_private_location_check_runs{state=~"queued|inflight", private_location_slug_name="<slug>"})

The query targets a single Private Location using private_location_slug_name. If you run multiple Private Locations, create one ScaledObject for each. Because the slug name is immutable, renaming the location in the UI won’t affect your scaling config.

If you deploy agents with the Helm chart, template the ScaledObject alongside your chart values. This keeps the autoscaler tied to the agent deployment instead of managing it as a separate manifest.

For a Prometheus instance outside the cluster, add an authenticationRef that points to a TriggerAuthentication with the required credentials.

How many pods will you get?

Set the threshold to match your agent JOB_CONCURRENCY. This defines how many checks one agent pod can process at the same time. With the default JOB_CONCURRENCY of 1, use threshold: "1". That means KEDA will try to keep roughly one queued or in-flight check run per agent pod, within your min and max replica limits.

For example, with minReplicaCount: 2 and maxReplicaCount: 10:

Queued + in-flight check runs	Resulting pods
0	2 (idle floor)
1	2
3	3
7	7
20	10 (capped)

Your browser does not support the video tag.

Autoscaling in action: a burst of checks spikes the queue, the agent count scales from 1 to 11 to clear the backlog, then settles back to a right-sized floor.

A few rules of thumb for the bounds:

Keep minReplicaCount at 2 or higher. One agent is a single point of failure, and if it dies your whole location goes dark until a replacement spins up.
Set maxReplicaCount above your expected peak load, because check runs that pile up past the cap get dropped after the queue's six-minute TTL.
Leave threshold at "1" unless you've raised JOB_CONCURRENCY; packing more checks per pod can delay long-running checks.
If you want to scale to zero when a location is completely idle, set minReplicaCount: 0 and tune cooldownPeriod, which controls how long KEDA waits after load drops before scaling all the way down.

Don't kill checks mid-run

Scaling down means evicting pods, and an evicted pod might be running a check. Checkly handles this for you: an in-flight check on a terminating pod gets rerun on another agent after a 300-second timeout, so a scale-down never silently loses a result. To give a pod room to finish its current work before Kubernetes sends SIGKILL, set terminationGracePeriodSeconds above that window on the agent pod spec:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 330

Set it to match your longest check type. Most checks finish fast, but Playwright Check Suites can run up to an hour, so bump the grace period accordingly:

Check type	Maximum runtime
API, TCP, DNS, ICMP	30 seconds
Browser	4 minutes
Multistep	4 minutes
Playwright Check Suite	60 minutes

If you run Playwright Check Suites in this location, terminationGracePeriodSeconds can go as high as 1800.

What this gets you

Your checks stay reliable under load. When a deploy or a traffic spike floods the queue, agents scale up to clear it instead of letting runs wait. That means faster, more dependable detection from inside your own infrastructure.

You also avoid paying for idle capacity. Rather than keeping enough agents running all day to handle occasional peaks, you can maintain a smaller baseline and scale up only when needed. Across multiple Private Locations, that unused capacity can add up quickly.

Get started

Private Locations are available on Checkly's Team and Enterprise plans. If you're already running agents on Kubernetes, follow the Private Location Autoscaling guide for the complete setup, including how to verify that KEDA is receiving the metric. Enable Prometheus V2, apply in the ScaledObject, and run a batch of checks to see how your agent deployment scales with demand.

Already using Private Locations? Open a location in the Checkly UI to view its queue duration, agent count, and scheduled check runs. These metrics help you quickly identify whether you have too little or too much capacity.

New to Checkly? See pricing to find the plan that fits, or start for free to try the rest of the platform.

]]>

Rocky reads your stack, not just your check

Stefan Judis — Thu, 04 Jun 2026 00:00:00 GMT

You get an infrastructure alert. Something's off, so you task your agent to investigate, and it ends up grepping through the logs. You both wonder if there's customer impact.

There’s a simple rule: AI agents in the incident loop are only as good as the signal you feed them. And logs alone aren't a great signal.

Synthetic monitoring at least solves the impact question. When a check fails, something user-facing is broken. You know what broke, not why. OpenTelemetry traces have answered that why for a while, and Checkly supports them natively. But there was an unacceptable catch in the agentic world: a human still had to correlate the two by switching tabs, finding the right service, copying a trace ID, and walking the spans. By the time they had an answer, the next check had already run.

"We have a lot of traces collected, but people maybe don't know they exist… I think Checkly is a way to integrate traces into developers' workflow."
— Observability lead, leading public-safety tech company

Rocky AI, Checkly's agent, now does that investigation walk for you. If a check fails, Rocky reads your OpenTelemetry traces and walks the spans behind the failure. When you're on call, "Where do I even start looking?" becomes "Let me validate my agent's fix!".

Your browser does not support the video tag.

Rocky Analysis using OTEL traces

What Rocky AI reads, per check type

Good signal in, good answer out. So Rocky's context comes in three layers:

The failure itself (whatever the check type produces: request, response, headers, traceroute, assertions, test source, etc.)
A set of on-demand tools Rocky reaches for when the picture is incomplete (the last passing run, PCAP data for API checks, Playwright trace files for browser failures)
Context provided by a human (the check description field, the tags, the "add context" box in the RCA UI)

OTEL traces are the newest tool in that on-demand layer.

Source	API	Playwright	Browser	MultiStep	DNS	TCP	ICMP	Notes
Check Result Payload	✅	✅	✅	✅	✅	✅	✅
Check Description	✅	✅	✅	✅	✅	✅	✅	Description context
User Context	✅	✅	✅	✅	✅	✅	✅	UI text box
Tags	✅	✅	✅	✅	✅	✅	✅	Tag names
Last Passing Result	✅	✅	✅	✅	✅	✅	✅	On-demand tool
OTEL traces (new!)	✅	✅	✅	✅	✅	✅	✅	On-demand tool
PCAP	✅	-	-	-	-	-	-	API-Checks only
Playwright Trace	-	✅	✅	✅	-	-	-	On-demand tool

What Rocky AI does with OTEL traces

Once you set up traces, every check Checkly runs injects a W3C traceparent header (tagged tracestate: checkly=true) on its outgoing requests. Your backend's OpenTelemetry instrumentation continues that same trace, so your spans share the trace ID Checkly generated. When those traces are sent back to Checkly, it stitches them to the originating check run by trace ID. That's what puts your backend traces right alongside a failing check, giving Rocky the evidence to pinpoint the root cause.

A 500 error on an API or Playwright check might trace back to a downstream identity provider returning DNS errors, a Clickhouse server timing out, or a third-party payment API throwing 502s through your load balancer. The same check failure could have countless reasons, different stories in the spans, and multiple teams that own the fix.

Rocky classifies (INFRASTRUCTURE_ERROR, DOWNSTREAM_DEPENDENCY), names the offending service, and writes a root cause, user impact, and suggested fix. It all lands in whatever alert channel you already use, Slack, PagerDuty, a webhook.

There's no separate add-on. If your traces are flowing into Checkly, Rocky reads them.

Remember the agent you tasked at the top of this post? The one grepping logs? It can read this analysis too.

The same root cause analysis, for your coding agent

Rocky's analysis is also available via the Checkly CLI, and who loves using CLI tools? Your agent!

Make sure to install the Checkly skills, point your coding agent at a failing check, and ask it to investigate. It picks up the right CLI commands on its own and reads the same classification, root cause, user impact, and suggested fix. Including, when it's available, the OTEL trace Rocky walked.

Your browser does not support the video tag.

Agent Conversation using Rocky's Root Cause Analysis

That's the agentic layer in action. There's no magic. Just CLI commands and agents helping each other out with good signal. And what about you?

You don't need to post screenshots into Slack threads, or investigate by hand. You're only in charge of validating the fix! Welcome to the new world.

Wire up OTEL and inspect the why

If you're already on Checkly with OTEL flowing in, Rocky is already walking those traces. Enjoy! Open a failing check and read the analysis.

If you're not yet sending traces to Checkly, the setup docs get you there.

Without traces, Rocky's intelligence stops at the border of your app. With them, your check failures come with the why attached so you can stop hunting for it. And your agent will happily stop grepping logs. Win-win!

]]>

Two AI agents, one incident: Rocky AI comes to the terminal

Stefan Judis — Wed, 29 Apr 2026 00:00:00 GMT

A Playwright Check fails at 2 am. The login flow is broken. Until today, that alert triggered a human to get up, open the Checkly dashboard, copy Rocky AI root cause analysis (RCA), and then tell an agent to get to work.

There were two AI agents, one incident, and no way for them to talk to each other.

The extended checkly checks and new checkly rca CLI commands close that gap. Your coding agent can now pull Rocky AI's analysis into its ongoing work, read the diagnosis, and go fix the code. Everything happens in the same session.

There's no poor soul on call required to pass around information.

Meet the two agents

One sees production from the outside. The other sees the code from the inside.

Rocky AI is the one watching your production site. It knows about failed checks, error messages, monitoring scripts, logs, metrics, network waterfalls, packet captures, and Playwright traces. Its entire and only job is to come back with a classification, a plain-English root cause, a user-impact assessment, evidence, and a suggested fix.

Your coding agent, on the other hand, is the one handling your application. If it uses the agentic layer, it can query databases, tail logs, hit APIs, and poke at your entire infrastructure. It can see production from the inside, the way the code sees it.

What it couldn't reach, until now, was Rocky AI analysis itself. Rocky sits a layer above the raw signals. It's a specialist who's already analyzed everything your monitoring produces into one diagnosis. For your coding agent, that's gold. Your agent skips the evaluation and already knows the kind of bug plus the evidence to reason from. Without Rocky's analysis, your agent has to redo the triage from scratch, which only burns tokens and risks the wrong fix. Especially on production alerts, every minute counts.

Until now, those two agents have never met. Rocky AI analysis stayed inside the Checkly UI. Your coding agent had plenty of reach, except for your alert analysis. This changes now.

Rocky AI analysis, in your terminal

If you have automatic Rocky analysis enabled, the root cause analysis is already embedded in the check result. Your agent doesn't need a new command. checkly checks already returns the results:

# access the failing checks
npx checkly checks list --status failing

# access check results
npx checkly checks get <check-id>

# access the root cause analysis for the current error group
npx checkly checks get <check-id> --error-group <group-id>

The error group's output includes the root cause, user impact, cited evidence (HTTP traces, assertions, timings), and reference links. Everything Rocky already checked and analyzed is available for you and your agent.

When should you use `checkly rca` directly?

There are two cases where you reach for the dedicated rca command:

You've turned off automatic analysis to control your invocation quota, and you want to trigger RCA only for specific failures.
You want to analyze a past failure that never got an automatic analysis. It could be a flaky check from last week, a historical outage, anything predating when you enabled Rocky.

# Run a fresh analysis on an error group
npx checkly rca run --error-group <group-id> --watch

# Fetch a previously-triggered analysis by its RCA ID
npx checkly rca get <rca-id> --output json

The rca command supports different output formats: detail(default, human-friendly), md (good for PR descriptions), and json (the one agents love).

Not in the terminal? The same data is in the public API

There are three new endpoints to analyze your failing checks and start new Rocky analyses:

GET /v1/root-cause-analyses/{id} to retrieve an existing root cause analysis
POST /v1/root-cause-analyses/error-groups/{errorGroupId} to generate a new root cause analysis for a check error group
POST /v1/root-cause-analyses/test-session-error-groups/{testsessionErrorGroupId} to generate a new root cause analysis for a test session error group

From alert to PR — a fast and optimized agent loop

Here's the workflow end-to-end with an agent driving.

An alert is triggered. Rocky has already analyzed what's going on. It pulled the evidence, classified the failure, and wrote up the root cause. You hand it over to your agent, telling it only that "something's off".

From there, your agent already knows which commands to reach for. The Checkly skill taught it the playbook. It finds the failing check, accesses the analysis, and gets to work — pointed at the right surface, whether the fix lands in the check config, the production code, or infrastructure. The investigation is already done. Less guessing, quicker resolution.

See the full loop in motion. Stefan demos the end-to-end agentic workflow, from generating checks with coding agents to resolving incidents with Rocky AI, on the agentic workflow webinar.

Get started

Install or upgrade to the latest Checkly CLI:

npm install checkly@latest

Install the Checkly skills so your agent knows which commands to use:

npx checkly skills install

Tell your coding agent to investigate a failing check. With automatic analysis on, Rocky AI analysis comes inline:

I've received a Checkly alert. Can you investigate?

And that’s it!

Previously, there were two AI agents, one incident, and a human connecting them.

Now, there's a single prompt. That's the agentic layer in action. Specialized agents work together to resolve production issues as quickly as possible. Isn't this beautiful?

]]>

How to Monitor a Shopify Store with Playwright and Checkly

Vince Graics — Mon, 13 Apr 2026 00:00:00 GMT

This is a guest post by Vince Graics, Staff QA Engineer at World of Books.

If you're running a Shopify storefront and want reliable synthetic monitoring, you'll hit a wall. Shopify's bot detection doesn't care that your headless browser is friendly; it sees datacenter IPs and acts accordingly. Cart API calls get hit with 429 rate limits, Cloudflare challenge pages pop up mid-check, and you're left wondering whether the bug is in your code or in the platform fighting you.

At World of Books, a UK-based secondhand bookseller running on Shopify, we ran into all of this while setting up monitoring with Checkly. The good news: Shopify actually has a fix for the bot-protection problem. The bad news: almost nobody talks about it, and there are a couple of non-obvious traps along the way.

This guide covers how we solved three Shopify-specific problems: bot protection headers, consent pop-up interception, and checkout flow testing. The patterns here work for any Shopify store. Swap out the selectors, plug in your store's URLs, and you have a working setup to monitor your Shopify store with Playwright.

What makes Shopify stores hard to monitor?

Shopify stores introduce platform-level obstacles that standard Playwright tests don't account for.

Bot protection. Shopify evaluates incoming traffic based on IP reputation, headers, and behavioral patterns. Monitoring checks run from cloud infrastructure, and datacenter IPs get treated with suspicion. Sometimes you get a 429 Too Many Requests on a cart API call. Sometimes Cloudflare intercepts the request entirely and serves a challenge page. And sometimes everything looks like it loaded, the page shell renders, the layout is there, but the interactive parts are quietly broken. A search page shows zero product cards. A mini-cart notification never appears.

Privacy and consent interception. Shopify's privacy SDK fires on every page load, showing a cookie consent banner and blocking script execution until the user accepts. In a headless context, that banner just sits there, adding flakiness and wasted seconds to every run.

Checkout DOM isolation. Shopify's checkout runs on checkout.shopify.com with a completely different markup. The selectors you wrote for the storefront don't work on checkout, and the page structure can change between Shopify editions.

Each section below tackles one of these problems and builds on the solution.

How do you bypass Shopify bot protection?

Shopify provides a mechanism for authorized crawlers to identify themselves: HTTP signature headers. You register your crawler in the Shopify admin, and Shopify gives you three headers: Signature, Signature-Input, and Signature-Agent.

When your requests include these headers, Shopify's bot protection lets them through cleanly.

Creating your crawler signature in Shopify

From your Shopify admin:

Go to Online Store > Preferences
In the Crawler access section, click Create signature
Enter a descriptive name (e.g., "Checkly Monitoring")
Select the domain and an expiration period
Copy the Signature-Input and Signature values

The third header, Signature-Agent, is static; its value is always "<https://shopify.com>" (with quotes).

One thing to watch for: signatures have an expiration date. Set a calendar reminder to rotate them before they expire, or your checks will start failing again with zero code changes. When they expire, every check returns a challenge page and it looks like your entire store is down.

Storing headers as Checkly secrets

Add the three values as secrets in your Checkly account:

SHOPIFY_SIGNATURE
SHOPIFY_SIGNATURE_INPUT
SHOPIFY_SIGNATURE_AGENT

This keeps your credentials out of code and makes them available to all your browser checks at runtime.

Injecting the headers in Playwright

The simplest approach, and what you should try first:

await page.setExtraHTTPHeaders({
  'Signature': process.env.SHOPIFY_SIGNATURE!,
  'Signature-Input': process.env.SHOPIFY_SIGNATURE_INPUT!,
  'Signature-Agent': process.env.SHOPIFY_SIGNATURE_AGENT!,
});

setExtraHTTPHeaders adds these headers to every request the page makes. For many Shopify stores, this just works and you're done.

When you need scoped header injection

If your store uses third-party services that don't play nice with extra headers, setExtraHTTPHeaders can cause problems. At World of Books, we hit this with a third-party search provider: the extra headers on cross-origin requests appeared to interfere with its responses, and product listings silently came back empty. No errors, no warnings; just an empty product grid and a check that fails for reasons that take an hour to debug.

The fix is Playwright's page.route(), which scopes header injection to your own domain:

const crawlerHeaders: Record<string, string>
  = (process.env.SHOPIFY_SIGNATURE
    && process.env.SHOPIFY_SIGNATURE_INPUT
    && process.env.SHOPIFY_SIGNATURE_AGENT)
    ? {
        'Signature': process.env.SHOPIFY_SIGNATURE,
        'Signature-Input': process.env.SHOPIFY_SIGNATURE_INPUT,
        'Signature-Agent': process.env.SHOPIFY_SIGNATURE_AGENT,
      }
    : {};

await page.route('**/www.yourstore.com/**', route =>
  route.continue({
    headers: { ...route.request().headers(), ...crawlerHeaders },
  }),
);

Third-party requests pass through untouched. A few things to note:

All three headers are required together. If any are missing, skip the injection entirely; partial headers won't work with Shopify's validation. And spread existing headers first: route.continue() replaces headers entirely, so the spread preserves cookies, content types, and everything else the browser set. After your test, call page.unrouteAll({ behavior: 'ignoreErrors' }) to avoid route handler leaks between checks.

Scaling this with Playwright fixtures

Once you have more than a couple of checks, you don't want to repeat the header setup in every file. Playwright fixtures let you extend the base page object once and use it everywhere:

import { test as base } from '@playwright/test';

export const test = base.extend({
  page: async ({ page }, use) => {
    if (Object.keys(crawlerHeaders).length > 0) {
      await page.route('**/www.yourstore.com/**', route =>
        route.continue({
          headers: { ...route.request().headers(), ...crawlerHeaders },
        }),
      );
    }
    await use(page);
    await page.unrouteAll({ behavior: 'ignoreErrors' });
  },
});

Every check that imports test from this fixture gets crawler header injection automatically. No headers configured? The fixture is a no-op; your checks still work locally without Shopify credentials.

The consent banner problem is frustrating because it's invisible in manual testing (you click it once and forget it) but it breaks every automated check.

Playwright's page.addLocatorHandler() lets you register a handler that fires whenever a matching element appears. Think of it as a popup bouncer for your checks:

export const test = base.extend({
  page: async ({ page }, use) => {
    await page.addLocatorHandler(
      page.locator('[aria-label="Close popup"]'),
      async () => {
        await page.locator('[aria-label="Close popup"]').click();
      },
    );
    await use(page);
  },
});

This works well for stores with a single, consistent popup. If your store has multiple overlays (newsletter signups, region selectors), you can register additional handlers with different locators. The key is keeping the selectors stable; aria-label attributes tend to survive theme updates better than CSS classes.

How do you test Shopify checkout?

Checkout is where monitoring gets tricky, because you're crossing into checkout.shopify.com with its own DOM, its own selectors, and real money involved.

Setting up safe checkout testing with Shopify Flow

You can't just skip checkout monitoring; it's the flow that matters most to your business. But running real transactions in production sounds terrifying. Here's the approach my team at World of Books came up with, using Shopify Flow:

Get a company card
Create a dedicated test user on your production store
Set up a Flow automation: if this specific user purchases something, the workflow immediately refunds the order and cancels the trade
Run your checkout checks against this test user

You can now run regular checkout validations until you run out of the company card budget, and wait for the refunds. It's not elegant, but it works, and it tests the real checkout flow, not a staging approximation.

Deploying to Checkly

Once the checks pass locally, deploy with the Checkly CLI:

# Test against Checkly infrastructure first
checkly test --record

# Deploy
checkly deploy -f -o

Set your environment variables in the Checkly dashboard under the check group. The required ones: SHOPIFY_SIGNATURE, SHOPIFY_SIGNATURE_INPUT, SHOPIFY_SIGNATURE_AGENT. Add LOGIN_EMAIL, LOGIN_PASSWORD, and discount codes if you're monitoring authenticated or promotional flows.

Because everything lives in your repo as code, your Shopify monitoring ships through the same CI/CD pipeline as your application. That's the Monitoring as Code approach: your checks are versioned, reviewed in PRs, and deployed automatically. No clicking through a UI to update a threshold.

Troubleshooting common failures

Challenge page instead of storefront. Your crawler headers are missing or expired. Regenerate them in Shopify Admin and update the env vars in Checkly.

Login fails or loops. Test account credentials are wrong, the account is locked, or MFA got enabled. Use a dedicated test account that the store admin has allowlisted.

Discount code shows "not valid." The code expired, hit its usage limit, or targets the wrong region. Verify it in Shopify Admin > Discounts. Use unlimited test codes.

Checks pass locally, fail on Checkly. Almost always missing environment variables. Run checkly env ls to verify. Second most common cause: viewport mismatch. Set viewport: { width: 1920, height: 1080 } explicitly in your Playwright config.

Wrapping up

At World of Books, customer experience is the reason we monitor our Shopify storefront so closely. If search breaks, checkout stalls, or a pop-up blocks the flow, it's important that we know before our customers do. Shopify can be difficult to test with headless browsers, but once we added crawler-signature headers via Checkly secrets, used a Playwright fixture to centralize header injection and consent handling, and set up Shopify Flow to clean up checkout test orders, the setup became reliable. The checks now see the same storefront our customers see, without the 429s, Cloudflare challenges, or quiet failures that make Shopify monitoring so frustrating.

If your checks keep mysteriously failing, check your headers.

Get started:

]]>

Checkly Playwright Reporter: A Cloud Dashboard for Your Playwright Tests

Pırıl Kavlak,María de Antón — Wed, 01 Apr 2026 00:00:00 GMT

The Checkly Playwright Reporter is an npm package that sends the results of npx playwright test to Checkly as a cloud test session, including traces, screenshots, videos, and full debugging context. Run your Playwright suite in CI or locally, and every result gets a persistent, shareable home in Checkly with AI-powered analysis, richer trace-derived views, and a direct path to production monitoring.

It does not replace Playwright. It makes the output of Playwright much easier to work with.

Today, the reporter is available at no additional cost on all Checkly plans.

Why we built this

Checkly has supported Playwright for years. You could write Playwright tests, validate them with npx checkly test, and deploy them as production monitors with npx checkly deploy. That workflow is powerful for production monitoring.

But most teams run npx playwright test hundreds of times a week in GitHub Actions, GitLab CI, or locally, and those results never reach Checkly. The evidence ends up scattered across CI logs, local trace files, and one-off HTML reports. That's manageable for an individual, but it breaks down for a team.

The Checkly Playwright Reporter helps close that gap by giving teams a shared place to review Playwright runs in Checkly.

From there, teams can run Playwright Check Suites with npx checkly pw-test, and when a test is important enough, npx checkly deploy can take it into production monitoring.

But it is not only about having a place to persist your test results. Once those results are in Checkly, Rocky AI can help analyze failed runs directly in the UI.

When enabled, Rocky AI can include AI-assisted root cause analysis and visibility into recurring failures across runs.

What you get beyond a hosted report

Today, there are several ways to get Playwright test results into the cloud, including Microsoft's own Playwright Workspaces. But most of them stop at hosting: they take the same local Playwright HTML report and put it behind a URL. That solves the sharing problem. It does not solve the understanding problem.

The Checkly Playwright Reporter goes further:

Filterable network view — every request and response captured from the trace, organized so you can focus on the failures instead of scanning a raw list.
Filterable browser console — browser console output pulled into a cleaner view, including an errors-only filter.
Automatic secret scrubbing — secrets can be scrubbed before upload, including support for the CHECKLY_SECRET_ prefix and sensitive header redaction.
System performance graphs — CPU and memory usage tracked throughout the test run, so you can spot whether a flaky or failing test might be tied to resource pressure.
Terminal progress and summary table — a clearer terminal view of what passed, failed, flaked, or got skipped across projects.
Automatic git detection — branch, commit, and author metadata can be attached to sessions automatically in CI and local git environments.
History across runs — instead of treating every run as an isolated local report, you can review patterns over time and see when the same failures keep coming back.

Hosting a report gives you a link. A reporting system gives you answers.

How to set up the Checkly Playwright Reporter

You need:

Node.js 18 or newer
Playwright 1.40 or newer
a Checkly account
your CHECKLY_API_KEY and CHECKLY_ACCOUNT_ID

Install the package:

npm install --save-dev @checkly/playwright-reporter

Add it to your playwright.config.ts:

import { defineConfig } from '@playwright/test'
import { createChecklyReporter } from '@checkly/playwright-reporter'

export default defineConfig({
  reporter: [
    ['list'],
    createChecklyReporter(),
  ],
  use: {
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on',
  },
})

Set credentials:

export CHECKLY_API_KEY=your_api_key
export CHECKLY_ACCOUNT_ID=your_account_id

Then run your suite as usual:

npx playwright test

When the run finishes, you get a Checkly test session URL in the terminal. That is the entire setup. No cloud infrastructure to provision, no storage accounts to link, no role assignments to configure.

Try the Checkly Playwright Reporter

If your team already uses Playwright, this is the fastest way to stop losing useful test context after the run.

Start here:

Frequently Asked Questions

Yes. The reporter is free on all Checkly plans today, including the free tier. As we learn from usage and continue improving the product, pricing may evolve over time. Our goal is to keep it highly competitive relative to other reporting tools and make it easy for teams to get started.

If you are already using the reporter and want to share feedback, get in touch! We'd love to hear from early users helping shape the product.

Yes. The reporter works with local and CI runs, as long as the Checkly credentials are available.

No. The reporter uploads test runs as Checkly test sessions. Production monitoring is the next step in Checkly’s Playwright monitoring workflow, where teams define Playwright Check Suites in a checkly.config.ts file and deploy them as monitors.

]]>

One CLI, Two Audiences: How We Built for Agents and Humans

Stefan Judis — Thu, 26 Mar 2026 00:00:00 GMT

Half of the Checkly CLI users are already coding agents. This is not a prediction — it's what the data shows today.

Since February, more and more agents have been using the CLI to manage and configure their Checkly monitoring setups. Right now, we're at 50% human and 50% agentic CLI users. And we predict that by the end of 2026, it won't be humans using the CLI; the agents will have taken over.

The terminal became the primary interface for AI agents doing real work in the Checkly ecosystem. Configuring monitoring, resolving incidents, and communicating outages; all this can be done automatically.

The CLI became the new agentic layer. So we asked: how do you build for two audiences at once?

The full agentic loop — from alert to resolution

To show what this looks like in practice, here's a real scenario: a Playwright check detects that the storefront's login flow is broken. The old way would be to manually open the Checkly dashboard, click through results, and switch between the IDE, alert information, and Git logs to figure out what's off. We now hand these tasks to the agent with a single prompt.

I just received a monitoring alert in Checkly about a broken login flow. Can you check what's up, investigate a fix, and communicate the issue?

It's a simple but powerful prompt. Here's what happens next:

The agent discovers Checkly capabilities. With the installed Checkly Skill, it pulls in the context it needs to act. Available commands, your account information and plan, investigation playbooks, and best practices; all these things become accessible through checkly skills.
The agent investigates. It calls npx checkly checks to get the current state of all monitors, identify the failing check, and analyze the Playwright error. It reads the check results, correlates them with your codebase, and additional context sources. Eventually, it finds the root cause.
The agent communicates. Before fixing anything, it tries to open an incident so customers know what's happening, but the CLI rejects the command and prompts for confirmation. Agents can't just declare a public incident without approval. Once confirmed, the incident goes live: "Login flow is broken, users cannot log in."
The agent provides a fix and deploys. It applies the fix, commits, and pushes. Your existing CI/CD pipeline takes it from there — the agent waits for the deployment to complete.
The agent verifies. Once deployed, it triggers the Checkly checks to confirm the fix actually works.
The agent resolves. If all the ad hoc tests pass, the incident is resolved with a summary of what happened and what was done. The status page goes green.

This might sound like future music, but you can see this flow in action below.

Coding agents can handle the full incident life cycle today. Detect, understand, communicate, fix, verify, resolve; all this can happen in a single agent session. All actions are enabled through the CLI. There's no need for you to open a monitoring dashboard. There's no context switching. Sit back and let the agent be the incident manager.

But how does this work in detail? How did the agent know what to do?

Checkly Skills: teach your agent how to handle your monitoring

Your agent didn't hallucinate, fetch docs, or browse the internet. It was tasked by the Checkly skill to rely on the built-in documentation accessible via npx checkly skills.

You can install the Checkly skill with a single command:

npx checkly skills install

This command drops a SKILL.md file into your project that any coding agent (Claude Code, Cursor, Codex, Copilot) can pick up. Everybody is raging about skills right now, but most skills have a fundamental flaw. Most skills are static. If you look at all the skills listed on skills.sh, you'll find that the majority of them are on-off documentation that you need to update manually.

The Checkly skill file is different because it's only a thin wrapper around the checkly skills command.

Run `npx checkly skills communicate` for the full protocol details.

### `npx checkly skills initialize`
Learn how to initialize and set up a new Checkly CLI project from scratch.

### `npx checkly skills configure`
Learn how to create and manage monitoring checks using Checkly constructs and the CLI.

#### `npx checkly skills configure api-checks`
Api Check construct (`ApiCheck`), assertions, and authentication setup scripts

Yes, you read that right, we've put optimized agent documentation right into the CLI itself. Required monitoring context ships within the tooling you use to manage your monitoring. Every CLI version update brings updated documentation, new workflows, and current best practices. There's no web search, no stale docs, no external files to maintain. Whenever you update the CLI, your agent gets smarter.

But we didn't stop there. Skills also use progressive disclosure across three levels:

Overview — npx checkly skills lists all available action categories: initialize, configure, investigate, communicate and manage.
Action — npx checkly skills investigate returns the detailed guide for that category — which commands to run, what to look for, and how to triage.
Reference — npx checkly skills configure api-checks returns construct-level documentation for any resource.

The built-in Checkly skills save tokens and tell your agent exactly what to do. No guessing. No hallucinations.

This single command transforms your agent into an up-to-date Checkly expert. But what can your agent handle for you now that it knows every corner of the Checkly ecosystem?

Structured output: the agent's dashboard

If you're facing issues and alerts, as a human, you head into Checkly, open the dashboard, click through the check results, and scan the graphs. And while you could task your agent to open the web app with a browser, this process couldn't be more inefficient. We released new CLI commands to enable the agent to access structured output, replacing the good old Checkly dashboard you know and love.

Read commands support multiple output formats optimized for agentic workflows:

npx checkly checks list --output json
npx checkly checks get 12345 --output md
npx checkly checks stats --output table

It's the same data you'll find in your Checkly web app, but rendered in text. Agents get parseable data they can reason about.

The new read commands cover the full monitoring setup:

checks list : All checks with current status. Filter by name (--search), tag (--tag), or check type (--type).
checks get <id> : Configuration, recent results, error groups, and analytics for a single check. Customize with --stats-range, --group-by, --metrics, and --filter-status.
checks stats : Availability, response times, and key metrics across multiple checks at once.

# Stats for all production API checks over the last 7 days
npx checkly checks stats --range=last7Days --tag=production --type=API

By exposing the monitoring data via the CLI, your agent now understands what an alert means and how to investigate. With access to your codebase, it can get to work and provide a fix.

Incident lifecycle through the CLI

Once the agent (and you) decide that you're facing a real issue, it's time to communicate it. The CLI covers the full incident lifecycle on your status pages:

npx checkly status-pages list          # discover your status pages
npx checkly status-pages get <id>      # inspect a specific page and its services
npx checkly incidents create           # open a new incident
npx checkly incidents update <id>      # post a progress update
npx checkly incidents resolve <id>     # close the incident

The same communication loop a human on-call would run: detect, communicate, resolve. But this time, you'll be faster, don't have to leave your terminal, and can supervise your agent to do the dirty work.

But isn't it dangerous to let your agent handle incidents? We thought so, too!

Guardrails: safe autonomy

Providing agents read-only commands was an easy decision, but what if wrong reasoning could have real consequences? What if the agent accidentally opens an incident, updates your monitors without you reviewing, or destroys your entire Checkly project, thinking it's "absolutely right"?

These actions and commands are deep in the danger zone, and you definitely don't want to wake up your teammates because your agent misconfigured something while you were vibing the time of your life.

We thought about how to handle these situations and decided to implement agent guardrails. If your agent decides to declare a public incident or deploy your monitoring, they need to ask you for confirmation first.

Critical commands implement an agent confirmation protocol. The CLI detects agent usage by checking for environment variables that coding agents set automatically (like CLAUDE_CODE, CURSOR_SESSION, or CODEX). If we detect that an agent is running a critical command, the CLI returns an exit code 2 with a JSON envelope:

{
  "status": "confirmation_required",
  "command": "incidents create",
  "changes": [
    "Will create incident \\"Login flow broken\\" on status page \\"Acme Store\\"",
    "Severity: major"
  ],
  "confirmCommand": "checkly incidents create --title=\\"Login flow broken\\" ... --force"
}

This forces the agent to present the changes to the user, wait for approval, and only then run the provided confirmCommand.

And what if the agent gets it wrong? What if it misdiagnoses the root cause or opens an incident for a false positive? That's exactly why the confirmation protocol exists. The agent proposes, you approve. A misdiagnosis becomes a learning moment, not an incident. The human stays in the loop for every action that has real-world consequences.

CLIs are the new agent interface

CLIs were built for humans first. But with a few design patterns, they become agent-ready. Here's what we learned building for both audiences:

Built-in documentation. Don't make agents search the web or rely on external docs. Ship agent-optimized documentation inside the CLI itself. When the tooling updates, the docs update as well. Minimize stale knowledge and hallucinations.
Structured output. Every read command should support machine-readable formats (--output json, --output md). Agents don’t care about colorful output, and humans shouldn't have to read raw JSON. Support both worlds.
Human/agent detection. Some CLI flows need to behave differently depending on who's running them. Detect the caller through environment variables and adapt — interactive prompts for humans, structured JSON responses, and additional command information for agents.
Agent guardrails. Critical commands need a confirmation protocol. Enforce human approval. Autonomy without oversight is just a liability.

These patterns aren't Checkly-specific. They're how every CLI should think about agent users. One CLI, two audiences — and the design choices made for agents make things better for everyone.

Half of our CLI users are already agents. Keep them coming!

Get started

Upgrade to the latest Checkly CLI:

npm install checkly@latest

Install the Checkly skill:

npx checkly skills install

Or, if you don’t have Checkly set up yet. Tell your agent the following:

Run 'npx checkly skills initialize' and follow the instructions.

It will figure out the rest.

]]>

Checkly and the Agentic Software Layer

Hannes Lenke — Thu, 26 Mar 2026 00:00:00 GMT

November 24th, the Opus 4.5 release turned around the entire tech industry. This was the moment when agents became capable. Capable enough to write solid staff-level code. Capable enough to reason about alerts, investigate root causes much faster than most engineers, and set up the reliability layer faster.

For me, this feels like an iPhone moment on steroids; the adoption of AI is accelerating much faster than any adoption curve I’ve seen over the past few decades. This rapid growth is fueled by early adopters who experienced their eureka moments early on, followed by many startup CEOs who have been actively developing prompting software during the holidays, making significant progress 10x quicker than ever before.

Now, AI is a topic in every customer conversation. While it took years to adopt DevOps, AI is on track to outpace that by a wide margin; in fact, it is already surpassing it.

The rise of AI agents introduced a new user of developer tools, one that doesn't click buttons, doesn't read dashboards, and doesn't care about your beautifully designed settings page. Agents live in the terminal. They reason over text. Over the last few years, we introduced Monitoring as Code (MaC), a workflow that enables developers to own reliability; in text. Accidentally, MaC is exactly what agents need.

You can see where we are going here. The work we did to make Checkly great for developers turns out to be the groundwork for making Checkly great for agents.

But "accidentally ready" isn’t a strategy - it was an opportunity.

Three Layers of Software Interfaces

Every software platform serves its users through interfaces. Historically, there have been two, sometimes three:

UIs are built for humans. They're visual, interactive, optimized for comprehension and exploration. Dashboards, forms, drag-and-drop builders. These are how people interact with software.

APIs are built for machines. They're structured, predictable, and designed for integration between systems. CI/CD pipelines, webhooks, Terraform providers. This is how humans have automated software.

CLIs are built to enable engineers to scale and automate their interactions with APIs. Especially, but not only for developer tools.

Turns out CLIs are great interfaces and became the agentic layer, and this is new. AI agents don't need pixel-perfect UIs or REST endpoints designed for static integrations. They need CLIs that output structured data in Markdown or JSON. They need progressive self-discovery, a way to explore capabilities without reading documentation. They need confirmation protocols for dangerous actions.

Most monitoring platforms were built for layer one. Some invested in layers two and three. Checkly is the first to double down on a new and redefined third layer. An application layer optimized for agents and for the modern software factory.

Why Code & CLI-First Platforms Win

Here's the thing about Monitoring as Code: it was already agent-friendly before agents existed.

When your monitoring lives in .ts files alongside your application code, an AI agent can read it, reason about it, generate new tests and checks, and deploy them the same way it works with any other code in your repository. There's no proprietary DSL to learn, no UI to scrape, no brittle browser automation required.

An agent working with the Checkly CLI can open your __checks__ directory, understand what's being monitored, identify gaps, write new checks using standard Playwright syntax, run npx checkly test to validate against our global infrastructure, and npx checkly deploy to ship the entire monitoring setup. Every step happens in the terminal, in code, using tools the agent already knows.

Legacy monitoring platforms trapped behind proprietary UIs and frameworks can't offer this. They'll bolt on a chatbot and call it AI. That's not the agentic layer: that's an AI-colored coat of paint.

Making the Implicit Explicit

Being code & CLI-first gave Checkly the agentic head start. But a head start doesn’t finish the race. We've shipped a new set of capabilities designed for the speed of this new world and purpose-built for agents.

Teaching your Agent Testing and Monitoring

Your agent understands how to configure Checkly today, because it can read the checkly.config and check.ts files. It also includes Checkly code in its training data, and if nothing else works, it can still fetch the docs to troubleshoot. But we thought, what if we could make it even easier?

Agent Skills are an open standard supported by Claude Code, Cursor, OpenAI Codex, and GitHub Copilot. Skills are providing agents with specialized knowledge on demand. Most skills out there are static files, though. We thought we could do better than providing some Markdown files!

Install the Checkly skill and see the magic happen:

npx checkly skills install

The Checkly SKILL.md file is your agents entry point to learn how to set up, configure and handle Checkly testing and monitoring. However, the static file doesn't include all the Checkly information. The Checkly skill is only a thin wrapper around the new npx checkly skills command. The new command that teaches your agent where to find construct documentation, other CLI commands, and monitoring workflows right in the CLI itself using npx checkly skills.

Once your agent discovers the npx checkly skills it can access agent-optimized docs to initialize, configure and manage the entire testing and monitoring setup. There's no relying on training data, no docs browsing, no context window compacting. Agents access the right docs when they need them.

But what does this mean in practice?

Creating your Monitoring Checks

The big difference between the agentic CLI layer and built-in product AI is that agents already understand and manage your codebase. They read and write files all day. With the right tools, coding agents have all the context they need to automatically set up and test your monitoring.

Your API endpoints, your login flows, your business logic; agents know about all of it. Task them to set up the monitoring and see your agent doing the work! In the previous world, you didn't have the time and energy to manage monitors for every page, endpoint, or application flow; now you do because your agent never gets tired.

Now, your agent can create, test, and deploy your entire monitoring!

More monitors also mean more alerts. Could your agent handle these, too?

Investigating Failures and Monitoring Results

If you received a monitoring alert in the old world, you usually log in to Checkly to understand what check failed and what triggered the alert. And while your agent could log in via a browser, it shouldn't need to do that to understand your monitoring setup.

A new set of read commands lets agents query and analyze your monitoring setup directly from the terminal:

# List all production API checks
npx checkly checks list --tag=production --type=API --output=json 

# Get 7-day stats for a check, grouped by location
npx checkly checks get 12345 --stats-range=last7Days --group-by=location 

# Analytics across all checks tagged "critical" 
npx checkly checks stats --tag=critical --range=last7Days

Every command supports --output table|json|md — table for humans, JSON and Markdown for programmatic agent consumption. Metric defaults adapt to check type: response time percentiles for API checks, Web Vitals for browser checks, latency and packet loss for ICMP.

With these new commands, your agent can access all the monitoring information you usually look at in the Checkly dashboard. Pair this ability with code access and let your agent figure out the issue.

Detecting issues is only one part of the story. Could your agent communicate issues, too?

Acting On Incidents

Agents can manage the full incident lifecycle on your status pages.

npx checkly incidents create --status-page-id <id> --title="API degradation" --severity=minor

npx checkly incidents update <id> --message="Root cause identified"

npx checkly incidents resolve <id>

Write commands implement a confirmation protocol designed for agent safety. When an agent runs a destructive or public-facing command, the CLI returns a JSON envelope describing the changes and provides a confirmation command.

This enforced confirmation is critical. Agents that can take action without guardrails are a liability. Agents with structured confirmation flows are a force multiplier.

If you put all these features together, your agent can handle your entire incident management while you drink your coffee. Don't believe me? Here's a video showing it entirely.

The Agentic Layer is Here

The shift to AI agents isn't coming. It's happening now. 50% percent of our active CLI users are already agents.

Development teams are using Claude Code, Cursor, and Copilot to write, test, and ship code. The tools in their stack that can't participate in that workflow will be replaced by tools that can.

Checkly is built for this moment. Upgrade your CLI, install the skill, and point your agent at your monitoring.

npm install checkly@latest
npx checkly skills install

… or, if you don’t have Checkly set up yet. Tell your agent the following:

Run 'npx checkly skills initialize' and follow the instructions.

… and it will figure out the rest.

]]>

12,479 Applications, Zero Ghosting: A Look at Checkly's 2025 Hiring

Lauren Singh — Thu, 19 Mar 2026 00:00:00 GMT

Hiring is one of the most human things a company does, and one of the least transparent. We want to change that. Here's an honest look at what our hiring actually looked like in 2025: the volume, the speed, and the funnel.

12 job postings. 24 people joined the team. 100% of applicants received a response.

12,479 applications. 12 job postings. A lot of trust placed in us.

In 2025, we received 12,479 applications across 12 job postings, an average of 1,040 applicants per posting. It's worth noting that some postings covered roles we hired for multiple times: for example, we brought on three Enterprise Account Executives from a single posting. Engineering, product, and design postings attracted the most interest, averaging around 1,105 applicants each, while go-to-market postings averaged 909.

Our most competitive posting was Senior Backend Engineer, with a total of 4,578 applicants.

Roughly 1 in 58 applicants progressed to interviews

We try to be thoughtful about who we move forward, and honest about the odds. Of 12,479 applicants, 217 progressed to the Hiring Manager interview stage, that's approximately 1.7%, or roughly 1 in 58. That's a tight funnel, and we don't shy away from saying so. What we will say is this: every single person who applied heard back from us. No ghosting, no black hole. We know that's not the norm, and we think it should be. Job searching is hard enough without silence on the other end. What made the difference for those who progressed? A genuine fit with the role, relevant experience for a developer tools company, and clear, async-friendly communication. We're a remote-first team, how someone writes and communicates matters.

Where our hires came from

Not every hire looks the same, and neither does every pipeline. Here's how our 24 hires broke down by source:

Actively applied: 60%
Referrals from the team: 20%
Sourced by a recruiter: 20%

The majority of our hires came through inbound applications, which means a well-written application genuinely matters. That said, referrals and sourcing play a real role too, and we're proud that team referrals account for a meaningful share of who joins us.

What candidates said about the experience

We ask every candidate who goes through our process to share feedback. In 2025, when asked how likely they were to recommend Checkly as an employer, the average score was 4.25 out of 5.

Here's what a couple of candidates had to say:

"The experience was overwhelmingly positive as I enjoyed speaking with all the stakeholders. Checkly is a place of high integrity that operates with the utmost professionalism."

"I am glad that even though I was not selected to continue the process, I got interviewed anyway. This is a good approach from the company and I hope you continue like that. The application form was unique and my time was respected. Thank you."

We don't share this to pat ourselves on the back, we share it because we think transparency should run both ways. Candidates trust us with their time, and we want to be accountable to that.

From application to offer: 22.5 days (median)

We know job searching is stressful, so we try to move quickly. Our median time from application to offer was 22.5 days, and our fastest? Just 4 days. Here's how the stages broke down:

The challenge stage, which applied to Engineering, Design, Support, and Customer Solutions roles only, naturally adds time, but it's an important part of how we assess real-world problem solving. Candidates are given up to 10 days to complete it, and we aim to make it focused and respectful of their time. If you're applying for a Product or GTM role, your process will typically be faster as there's no take-home challenge involved.

83% offer acceptance rate

In 2025, 24 offers were accepted, an 83% offer acceptance rate. We take this as a signal that the process works. By the time we get to an offer, both sides have had enough real conversation to feel confident. We hope that continues to be true.

What's next

Sharing this data publicly is something we'll keep doing. Hiring is a two-way street, candidates invest real time and energy in applying, and the least we can do is be honest about what that process looks like from our side.

Looking ahead, 2026 is shaping up to be a big year for our go-to-market team. If you're in sales, solutions engineering or customer success and you're excited about the developer tools space, check out our open roles or join our talent community, we'd love to hear from you. And if you've been through our process, whether you joined us or not, we're always open to feedback. Hiring should feel human on both sides, and we're committed to keeping it that way.

]]>

Expanding Uptime Monitoring Down The Stack: ICMP Monitors Are Now Available In Checkly

Susa Tünker,Pırıl Kavlak — Tue, 10 Mar 2026 00:00:00 GMT

When we started building Checkly's uptime monitoring suite, the goal was to give engineering teams complete visibility across every layer of their stack, from application down to network, in one place. URL, TCP, DNS, and Heartbeat monitors covered a lot of that ground. But one fundamental piece was missing: the ability to simply ping a host and know if it's reachable.

We're now closing that gap by adding ICMP monitors to Checkly's uptime monitoring suite, available on all plans today, alongside URL, TCP, DNS, and Heartbeat.

What is ICMP monitoring?

ICMP monitors send scheduled ping requests to any host or IP address to measure reachability, latency, and packet loss. You can set assertions on any of these to define exactly what "healthy" means for each host.

No application logic. No open ports required. ICMP is a low-level monitor that provides a direct network-level signal, independent of what is running on the machine.

What you can monitor with ICMP requests

Infrastructure without an HTTP endpoint

Databases, message queues, internal services, bare-metal servers, routers, and network devices don't serve web traffic, but they still need monitoring. TCP monitors cover many of these cases, but ICMP goes one level lower: no open port required. For hosts where no ports are intentionally exposed, ICMP is the right layer to monitor at. If a host can be pinged, it can be monitored in Checkly without adding extra tools to your stack.

Private networks and internal hosts

ICMP monitors are useful for hosts that aren't exposed to the public internet, making them valuable for teams monitoring internal infrastructure alongside their public-facing checks. This is possible via Checkly Private Locations, with ICMP support coming soon.

Incident triage

When an alert fires, the first question is always: Is the machine down, or is the service down? ICMP gives you that answer immediately. If ICMP fails, it's a network issue. If ICMP passes and your HTTP check fails, the server is up, but the service is broken. The right team gets paged with the right context from the start.

Regional latency comparison

Run ICMP monitors from multiple locations to compare latency across regions. If one region is consistently slower than others, you can catch geographic performance issues before they affect users in that area.

Some failures don't show up as outages. Intermittent packet loss is one of them. One of our beta users ran into exactly this:

"We had a serious issue in which every third packet was dropped, and it went completely undetected until Checkly caught it at the network level. That's the kind of signal that stops a problem before it becomes a customer-facing incident."

You can set separate degraded and failure thresholds on packet loss, and add latency assertions to define when to trigger an alert.

Shipping with full support for Monitoring as Code

ICMP monitors are fully integrated into Checkly's Monitoring as Code workflows, available in the CLI from v7.1.0.

Define your ICMP monitors in code alongside your Playwright and API checks, version them in Git, and deploy through CI/CD:

import { Frequency, IcmpAssertionBuilder, IcmpMonitor } from "checkly/constructs"

new IcmpMonitor('cloudflare-dns-icmp', {
  name: 'Cloudflare DNS ICMP Monitor',
  activated: true,
  frequency: Frequency.EVERY_1M,
  maxPacketLossThreshold: 20,
  degradedPacketLossThreshold: 10,
  request: {
    hostname: '1.1.1.1',
    pingCount: 20,
    assertions: [
      IcmpAssertionBuilder.latency('avg').lessThan(100),
      IcmpAssertionBuilder.latency('max').lessThan(200),
    ]
  }
})

Checkly Tip

Or use an AI agent - copy this prompt to get started:
"Create an ICMP monitor in Checkly for [your-hostname-or-ip]. Ping every minute from us-east-1 and eu-west-1. Set a degraded alert at 10% packet loss and a failure alert at 20% packet loss or average latency over 200ms."

ICMP monitors are also available in Terraform and Pulumi:

Terraform (v1.18.0+): checkly_icmp_monitor

Pulumi (v2.8.0): checkly.IcmpMonitor

Pricing and availability

ICMP monitors are priced as uptime monitors: flat-rate per monitor, available on all plans today. Check out our pricing page for full details.

Get started

Try ICMP monitors in your Checkly dashboard or read the documentation to learn more about configuration, assertions, and alerting.

Questions? Find us in the Checkly community on Slack→

]]>

We Turned Our WireShark Wizard Into a Markdown File

Tim Nolet — Wed, 04 Mar 2026 00:00:00 GMT

Rocky AI — Checkly’s AI agent — is now Generally Available. We developed Rocky AI over the last ~6 to 8 months. This is an aeon in AI-years.

During this period, we learned a ton. About AI, but mostly about how to fit them into an existing SaaS product, not just another chat widget. This is my ramble…

Something with AI

Like almost everyone in SaaS-land, we were pondering doing “something with AI” at least since late 2024. We went with some hunches, looked at competitors, talked to customers, hacked on some prototypes. But it was hard to pinpoint a shippable MVP. We were probably going too broad and promising a bit too much.

Save me time, and no magic please

So we did a hard reset on our AI ambitions. Instead of shipping the whole kitchen sink, what if we focus on one time consuming task in the critical path of our users? One repetitive, but critical taks that came to mind is triaging a failed Playwright Check test. Specifically, could we use LLMs to quickly create an analysis of a Playwright trace file?

We immediately bumped into two challenges:

A Playwright trace file can easily be 100Mb+. We need to parse it, remove unneeded data and turn it into a plain text representation. Counterintuitively, giving MORE data to an LLM does doet always guarantee better results. Or it just blows up the input token limit.
When the data wrangling done, how do we guide the LLM to make sense of the data? Sure, it can do some basic summary by itself, but we want to give it guidelines on what is important to look at, what things are typically related. We essentially want these ingrained triaging steps that (senior) engineers hold in their heads to made explicit.

Over the months we expanded our AI analysis skills to cover all our check types (Playwright, HTTP, TCP, DNS, ICMP etc.). We took the same approach each time:

Look at how we can use artifacts like logs, traces etc. to build up a body of evidence for a specific failure root cause. Data wrangle it into submission so an LLM can use it.
Capture the skill of analyzing and interpreting said artefacts into a semi-structured markdown file.

This is now our Rocky AI Root Cause Analysis Agent. We did bump into some other learnings along the way, so I dumped them into a nice listicle. 👇

#1. Just do what the humans do

Again, this might be blatantly obvious, but we are using AI to just do things humans can do. Example: we literally codified the Wireshark ICMP and PCAP analysis skills of one our engineers into a markdown file that now powers our AI driven ICMP and PCAP analysis. This looks a bit like this:

icmp-rca-skill.md# ICMP Analysis Skill

## Classifications
The root cause can only be classified using the following classifications:

### INFRASTRUCTURE_ERROR
External issues like network errors, packet loss, host unreachable, DNS failures, firewall blocks, or routing issues. Examples: 100% packet loss, DNS resolution failure, ICMP destination unreachable

### CONFIGURATION_ERROR
etc.

## Artifacts

### ICMP_REQUEST
Description: ICMP request properties including hostname, target IP, IP family, and ping count. etc.

## Common Failure Patterns

### PACKET_LOSS
Symptoms: High packet loss percentage, Packets sent but not received, Packet loss threshold exceeded
Causes: Network congestion along the route, Firewall dropping ICMP packets, Target host overloaded or rate limiting ICMP, Intermittent network issues
Analysis: Check packet loss percentage vs configured thresholds, Examine individual ping results for patterns, Compare results across different monitoring locations

### DNS_RESOLUTION_FAILURE
etc.

#2. Data wrangling is still the thing

Even with extended context windows and increased quota’s on input tokens, we still do a ton of data wrangling. As mentioned above, Playwright trace files easily go over 100Mb, a network PCAP file parsed to text can also be very large. These are all data sources want the LLM to take into account.

Currently, we solve this problem by pre-parsing, filtering and summarising these large assets in tool calls. The tool returns the key data from these assets back to the LLM.

#3. Model updates are almost a free lunch

Just the last ~6 months, the upgrade from OpenAI GPT-4.1 to GPT-5.1 (let’s not mention the dreadful GPT-5) was night and day: better answers, more reliable tool calls, faster responses, less processing errors, better schema output. You can see similar patterns for the Opus 4.5 to Opus 4.6 and the Gemini upgrades.

#4. Multi model is like multi cloud. It kinda sucks.

While building our Bring-Your-Own-Model (BYOM) feature, we allowed users to bring Gemini and Anthropic models next to our default OpenAI models. That was a mistake. Turns out that swapping models is quite hard right now if you want to keep some form of quality control and not have dedicated tests, fixtures and code paths.

This is even true with wrapper libraries like the Vercel AI SDK, which we use. In some way it reminds of how Terraform could in theory make your infra “multi cloud”. In practice, it really can’t.

#5. Don’t always default to chat

Obviously, chat is the de-facto UI paradigm for LLMs and generative AI. However, in our RCA case, it would be silly to force a user to go to some chat UI and asked Rocky “please analyse this failing check”. We can ask that question for the user, even when they are sleeping and just ship the initial analysis to their inbox. Chat can be a follow up.

We are totally not done with expanding Rocky AI and adding more tools and features to its RCA skills. Watch this space.

]]>

Introducing Rocky AI to General Availability

Dan Giordano — Wed, 04 Mar 2026 00:00:00 GMT

After months of being available in Beta for our app users, Rocky AI is now generally available to all users and plans. Rocky AI is Checkly’s AI agent that works around the clock, 24/7, to make sure your application’s reliability is optimal. In this first release, Rocky AI ships with the ability to run continual Analysis on test and check failures, giving your teams AI-powered root cause analysis, impact analysis, and more.

Since day 1 of the AI craze, Checkly has made it easy to work with your favorite LLMs and coding agents like Claude Code, Cursor, and OpenAI, allowing you to craft and manage your monitoring right from your IDE. Now, we’re bringing that power right into the Checkly workflow with Rocky AI.

Root Cause Analysis

When a test or monitor fails, Rocky AI analyzes multiple vectors across your application, including logs, traces, and more to understand what exactly caused the failure; and how to best fix it.

For large teams with hundreds of monitors across their apps and services, Rocky AI can help engineers understand the context of the check itself, even if they didn’t create it, reducing cognitive overload in the heat of an outage. For individuals and smaller engineering teams, Rocky AI will cut down on hours of back and forth, working to replicate the problem, and other triage works that delay teams from getting a quick fix back in production.

User Impact

Rocky AI also tries to understand and surface who exactly is impacted during an outage. Are users impacted from logging into the application? Is a core workflow being blocked? Is it a network failure that causes the test to fail, but user impact is actually minimal? Rocky AI can help you get the answers you need, faster than humanly possible, so you can get your application back up and running.

Automation

For teams with heavy alerting systems, Rocky AI can tap in and automatically send its findings inside any alert, right to your inbox or Slack channel. You'll get highly relevant context to the downtime, directly inside your workflow, allowing you to break out, fix it, and get back to work.

Getting Started

Rocky AI Error Analysis is available in Checkly Resolve today. To get started with Rocky AI right now in Checkly, just head over to app.checklyhq.com and click Rocky AI Analysis.

Or read through our documentation to get a better grasp of Rocky AI

What’s Next

With Rocky AI hitting general available with the ability Analyze results, we are only getting started. We will be adding other capabilities and skills to Rocky AI over time, to bring you native AI powered test generation, alert configuring, and more - right inside the Checkly dashboard. If you are a heavy user of LLMs or Agents, then check out Rocky AI or start using Checkly with tools like Claude Code today.

]]>