This is a guest post by Ilter Kavlak, Site Reliability Engineer at Upstash. It's a companion to his post on the Upstash blog, which covers the internal side of the same monitoring stack.
There's a support ticket every SRE dreads: "is something wrong with my database?" The outage is bad enough. Worse is the possibility that the customer knew first.
At Upstash, we treat that scenario as two failures rather than one: the incident itself, and the uptime monitoring gap that let a customer beat us to it. We write a postmortem for the gap, too.
This post is about the layer we built to make sure that never happens: a fleet of Checkly checks that watches our platform from the outside, the way a customer would. One check per replica, running every minute from 18 locations around the world, all of it defined in Terraform. I'll walk through what we check, how a failing check reaches a human, and how we monitor a product where "reachable" and "working" are very different claims.
Why every replica gets its own uptime check
Upstash Redis is multitenant and multizone. Every database is backed by at least two replicas, and with the Prod Pack those replicas live in different availability zones. When a replica dies, the survivors elect the most up-to-date among themselves and the database keeps serving; customers notice nothing.
That resilience is exactly what makes replica-level monitoring easy to skip. If your checks only watch the database endpoint, a dead replica is invisible: the endpoint stays green while you quietly run with less redundancy than you promised your customers. And the next failure is no longer a non-event.
So our rule is blunt. A dead replica is an incident even when no customer can feel it. Every single replica gets its own uptime check, and losing any one of them wakes the oncall.
A couple hundred URL monitors, all in Terraform
Because Upstash Redis speaks HTTP alongside the Redis protocol, health checking stays simple. No Redis client, no connection pooling, no scripted scenario. Each replica exposes a /ping endpoint over HTTPS, and a 200 means that replica is serving its databases the same way it would serve a real client.
We use Checkly URL monitors for these, defined in Terraform. Each replica is one module block:
module "check-eu-central-1-r2" {
source = "../modules/checkly_url_monitor"
name = "eu-central-1-r2"
base_url = "https://<replica-endpoint>.upstash.io/ping"
frequency_minutes = 1
locations = var.default_locations
alert_channel_ids = local.default_alert_channel_ids
}This is Monitoring as Code, and it's the feature that made us commit to Checkly. Our monitoring configuration lives in the same repository as the rest of our infrastructure, and a single apply deploys or updates the whole fleet. By now, the file reads like a census: one block per replica, a couple of hundred in total, spanning AWS and GCP regions from Tokyo to São Paulo. When a new replica joins a cluster, a new block gets written, reviewed, and applied like any other infrastructure change. No monitor is ever created by clicking around a dashboard; if it isn't in git, it doesn't exist in Checkly.
That discipline would be optional at two checks. At two hundred, it's the only way to be sure nobody forgot one.
Redis isn't the only product in that file. Upstash Vector and Search replicas get the same per-replica treatment, except their checks go beyond a ping: each one upserts a record and runs a query against it, because an index that accepts connections but can't answer queries is not "up" in any sense a customer cares about. Keep that idea in mind for the QStash section below.
From 18 locations, every minute
The hardest-working parameter in that module block is locations. Upstash serves clients globally, with replicas placed close to their readers, so a replica answering from one vantage point proves very little. Every monitor runs from all 18 of the Checkly locations we've configured, covering North and South America, Europe, Africa, and Asia-Pacific. A routing problem that only affects clients in Singapore is a full outage for a client in Singapore, so our monitoring treats it as one.
Each monitor runs every minute, and a failed request is retried after a short backoff before anything alerts, since a single dropped packet somewhere on the public internet is no reason to wake a human. And we refuse to conflate slow with down: a response over five seconds marks the monitor degraded, and past ten it counts as failed. Two different signals, routed two different ways.
How a failing check reaches a human
Every monitor is wired to the same set of alert channels, also defined in Terraform:
- Opsgenie pages the oncall on failure. This is the loud one.
- Slack receives both failures and recoveries, giving the rest of the team context without waking anyone.
- Degraded-state notifications land in a Slack channel of their own. Slowness deserves eyes but not a 3 a.m. page, and it shouldn't drown in a busy alerts channel either.
When a replica check fails, the oncall is looking at it within a couple of minutes, whatever the hour. None of that is remarkable. The habit worth writing down is how incidents end: we don't call anything resolved because internal dashboards look healthy. Resolved means the Checkly check is green again, from all 18 locations. Internal metrics can tell you a process is running; only the external check can tell you a client on the other side of the planet is actually being served. The second claim is the one customers experience, so it's the one that closes the incident.
QStash: proving messages actually get delivered
For QStash, our message queue and task scheduler, a health endpoint would prove almost nothing. QStash exists to deliver messages; it could happily return a 200 for every publish, deliver none of them, and a ping check would stay green the whole time.
So instead of a ping, QStash gets a Checkly API check that uses the product the way a customer would. The check publishes a real message through the QStash publish API and points it at an Opsgenie heartbeat endpoint, so every successful delivery resets the heartbeat's timer.
One message tests the whole pipeline, and the two ways it can fail are distinguishable:
- The publish fails. QStash is unreachable or rejecting requests. The Checkly check turns red and pages the oncall.
- The publish succeeds but nothing arrives. QStash accepted the request and then failed to deliver it. The heartbeat falls silent, Opsgenie notices the silence, and the page goes out.
Either way a human is investigating within minutes, and the alert already names which half of the flow broke before anyone opens a dashboard. That's a serious head start.
My favorite property of this check is that it's indistinguishable from a customer. Same publish API, same delivery path, same retries as real traffic. There is no separate test mode that could pass while production fails.
What Checkly doesn't cover, on purpose
External checks answer one question: would a customer be served right now, from where they are? Two other layers answer the questions they can't.
Internally, a federated Prometheus stack watches the clusters themselves and catches pod-level trouble. And in every region, a fleet of benchmark clients (deployed deliberately outside the clusters they measure) pushes a steady mix of reads and writes around the clock, recording every latency it observes. That fleet is our second, independent paging path, and its historical dashboards are where slow-moving problems like release regressions and noisy neighbors show up as trend lines long before they'd trip any single alert. Both layers are managed the same way as the Checkly monitors: Terraform modules in the same repository, one workflow for everything. I've written about that side of the stack in detail on the Upstash blog.
Every layer exists to cover a blind spot in the others. But when internal and external monitoring disagree, the external answer wins the argument, because it's the only one describing what customers actually experience.
The philosophy, in five sentences
Watch every replica, because redundancy you can't verify isn't redundancy. Watch from wherever your users actually are. Treat ‘slow’ as its own signal rather than a milder form of down. Test the thing the product promises to do, not just whether it answers. And don't declare victory until the outside world agrees.
None of this prevents outages; hardware still fails, zones still have bad days, and the internet remains the internet. What it buys is time and clarity: minutes from failure to a human engaged, an alert that names what broke, and a client's-eye definition of "fixed." The ticket asking "is something wrong?" should only ever get one reply: "yes, we know, and we're already on it."
Upstash runs a couple hundred Checkly monitors, every one of them defined in code. If you want the same workflow, start with Monitoring as Code: define your monitors in TypeScript with the Checkly CLI, or use the Terraform provider like Upstash does. Start monitoring for free.
