# Autoscaling Checkly Private Location Agents in Kubernetes > Monitor your Private Location health with Checkly, and autoscale agents with KEDA on live load. Stop guessing at agent capacity. Source: https://www.checklyhq.com/blog/autoscaling-checkly-agents-with-keda-in-kubernetes/ --- [Blog](https://www.checklyhq.com/blog/) / [Product](https://www.checklyhq.com/blog/tag/product/) / [Prometheus](https://www.checklyhq.com/blog/tag/prometheus/) # Autoscaling Checkly Private Location Agents in Kubernetes with KEDA [Edvinas Janusevicius](https://www.checklyhq.com/blog/author/edvinas-janusevicius/) July 2, 2026 · Updated July 3, 2026 Monitoring load is not always steady. A team might add a new batch of checks or run several ad hoc tests during a rollout. When that happens, your Private Location agents need to pick up more work at once. If there aren’t enough agents available during a burst, checks start piling up in the queue, which can delay or disrupt check execution. But solving this by running a high number of agents around the clock has the opposite problem: most of that capacity sits idle until the next busy period. Autoscaling solves both ends of that problem. You scale agents up when checks queue, scale back down when the burst clears, and stop guessing in between. This guide covers two things that landed together: the new Private Location metrics that show you whether your agents are running efficiently, and the KEDA setup that automatically scales your agent deployment. ## How do you know if a Private Location has enough agents? You can now monitor Private Location health with Checkly, so capacity stops being a guessing game. [Open any Private Location in the UI](https://app.checklyhq.com/accounts/private-locations) and you'll find three metrics that tell you exactly how your agents are doing: - **Queue duration**: how long check runs wait before an agent picks them up. This is the number to watch. If it creeps up, your agents may be falling behind on work. - **Agent count**: how many agents are currently reporting for the location. - **Scheduled check runs**: how many runs are queued for the location right now. These metrics are also available via our [Prometheus v2 integration](https://www.checklyhq.com/docs/integrations/observability/prometheus-v2/#private-location-metrics). Together, these signals help you understand whether your agents are healthy and whether your Private Location has enough capacity. For example, if two agents are reporting but queue duration keeps climbing and scheduled runs are stacking up, you may be under-provisioned. If ten agents are reporting and queue duration stays near zero all day, you may be running more capacity than you need. Those are the two states autoscaling exists to fix, and now you can see which one you're in. ## Scaling on the right signal Checkly agents are stateless: they sit in your network, pull scheduled checks from a shared queue, run them, and send the results back. You can add or remove agents freely without losing or duplicating work. That's the property that makes scaling safe. To develop a practical approach to autoscaling, we studied a range of large Private Location setups across customers, including [DVAG](https://www.checklyhq.com/case-study/dvag-migrates-to-checkly/), [LinkedIn](https://www.checklyhq.com/case-study/modernizing-linkedins-monitoring-infrastructure/), Sherwin-Williams, and Auto Trader. Looking at how those fleets handle different checks and workloads helped us identify the signal that best reflects a location’s current demand. For teams running agents on Kubernetes, that signal is `checkly_private_location_check_runs`. This gauge reports the number of check runs routed through a Private Location, broken down by state. Two states matter for capacity: - `queued`: the check run is scheduled, but no agent has picked it up yet. - `inflight`: an agent is running it right now. Add those two and you have the total live load on the location, which is the signal you want driving replica count. To use this metric, make sure [Prometheus V2 metrics](https://www.checklyhq.com/docs/integrations/observability/prometheus-v2/) are enabled for your account. ℹ️ If your existing KEDA config still uses `checkly_private_location_oldest_scheduled_check_run`, consider switching to `checkly_private_location_check_runs` with the `queued` and `inflight` states. The older metric is useful for spotting backlog, but `queued` plus `inflight` gives KEDA a more direct signal for current load. ## Setting up KEDA [KEDA](https://keda.sh/) (Kubernetes Event-Driven Autoscaling) reads a Prometheus metric and scales a Kubernetes deployment up or down based on that value. You'll need KEDA installed in the cluster, your agents deployed via the [Checkly agent Helm chart](https://github.com/checkly/helm-charts/tree/main/charts/agent), and [Prometheus V2](https://www.checklyhq.com/docs/integrations/observability/prometheus-v2/) enabled for your Checkly account. Here's a `ScaledObject` with sensible defaults. Adjust the bounds to your workload: ``` apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: checkly-agent-autoscaler spec: scaleTargetRef: namespace: name: minReplicaCount: 2 maxReplicaCount: 10 advanced: horizontalPodAutoscalerConfig: behavior: scaleUp: policies: - type: Pods value: 1 periodSeconds: 60 scaleDown: selectPolicy: Min policies: - type: Pods value: 1 periodSeconds: 60 triggers: - type: prometheus metadata: serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090 metricName: checkly_private_location_check_runs threshold: "1" query: sum(checkly_private_location_check_runs{state=~"queued|inflight", private_location_slug_name=" "}) ``` The query targets a single Private Location using `private_location_slug_name`. If you run multiple Private Locations, create one `ScaledObject` for each. Because the slug name is immutable, renaming the location in the UI won’t affect your scaling config. If you deploy agents with the Helm chart, template the `ScaledObject` alongside your chart values. This keeps the autoscaler tied to the agent deployment instead of managing it as a separate manifest. For a Prometheus instance outside the cluster, add an `authenticationRef` that points to a `TriggerAuthentication` with the required credentials. ## How many pods will you get? Set the `threshold` to match your agent `JOB_CONCURRENCY`. This defines how many checks one agent pod can process at the same time. With the default `JOB_CONCURRENCY` of `1`, use `threshold: "1"`. That means KEDA will try to keep roughly one queued or in-flight check run per agent pod, within your min and max replica limits. For example, with `minReplicaCount: 2` and `maxReplicaCount: 10`: Queued + in-flight check runs Resulting pods 0 2 (idle floor) 1 2 3 3 7 7 20 10 (capped) Autoscaling in action: a burst of checks spikes the queue, the agent count scales from 1 to 11 to clear the backlog, then settles back to a right-sized floor. A few rules of thumb for the bounds: - Keep `minReplicaCount` at 2 or higher. One agent is a single point of failure, and if it dies your whole location goes dark until a replacement spins up. - Set `maxReplicaCount` above your expected peak load, because check runs that pile up past the cap get dropped after the queue's six-minute TTL. - Leave `threshold` at `"1"` unless you've raised `JOB_CONCURRENCY`; packing more checks per pod can delay long-running checks. - If you want to scale to zero when a location is completely idle, set `minReplicaCount: 0` and tune [`cooldownPeriod`](https://keda.sh/docs/latest/reference/scaledobject-spec/#cooldownperiod), which controls how long KEDA waits after load drops before scaling all the way down. ## Don't kill checks mid-run Scaling down means evicting pods, and an evicted pod might be running a check. Checkly handles this for you: an in-flight check on a terminating pod gets rerun on another agent after a 300-second timeout, so a scale-down never silently loses a result. To give a pod room to finish its current work before Kubernetes sends `SIGKILL`, set `terminationGracePeriodSeconds` above that window on the agent pod spec: ``` spec: template: spec: terminationGracePeriodSeconds: 330 ``` Set it to match your longest check type. Most checks finish fast, but Playwright Check Suites can run up to an hour, so bump the grace period accordingly: Check type Maximum runtime API, TCP, DNS, ICMP 30 seconds Browser 4 minutes Multistep 4 minutes Playwright Check Suite 60 minutes If you run Playwright Check Suites in this location, `terminationGracePeriodSeconds` can go as high as 1800. ## What this gets you Your checks stay reliable under load. When a deploy or a traffic spike floods the queue, agents scale up to clear it instead of letting runs wait. That means faster, more dependable detection from inside your own infrastructure. You also avoid paying for idle capacity. Rather than keeping enough agents running all day to handle occasional peaks, you can maintain a smaller baseline and scale up only when needed. Across multiple Private Locations, that unused capacity can add up quickly. ## Get started Private Locations are available on Checkly's Team and Enterprise plans. If you're already running agents on Kubernetes, follow the [Private Location Autoscaling guide](https://www.checklyhq.com/docs/platform/private-locations/autoscaling/) for the complete setup, including how to verify that KEDA is receiving the metric. Enable [Prometheus V2](https://www.checklyhq.com/docs/integrations/observability/prometheus-v2/), apply in the `ScaledObject`, and run a batch of checks to see how your agent deployment scales with demand. Already using Private Locations? Open a location in the Checkly UI to view its queue duration, agent count, and scheduled check runs. These metrics help you quickly identify whether you have too little or too much capacity. New to Checkly? [See pricing](https://www.checklyhq.com/pricing/) to find the plan that fits, or [start for free](https://app.checklyhq.com/signup) to try the rest of the platform. [Edvinas Janusevicius Backend engineer](https://www.checklyhq.com/blog/author/edvinas-janusevicius/) In This Article - How do you know if a Private Location has enough agents? - Scaling on the right signal - Setting up KEDA - How many pods will you get? - Don't kill checks mid-run - What this gets you - Get started Share on social ## Related Articles [Mastering Prometheus Exporters: Techniques and Best Practices November 3, 2023](https://www.checklyhq.com/blog/mastering-prometheus-exporters-game-changing-techniques/)[The Advent of Monitoring, Day 10: Better Observability Into Your Local Clickhouse Instance With Grafana and Prometheus December 17, 2023](https://www.checklyhq.com/blog/better-observability-into-your-local-clickhouse-instance/)[Running API and Browser Checks Using Terraform, AWS, and Checkly Private Locations February 7, 2023](https://www.checklyhq.com/blog/running-api-and-browser-checks-using-checkly-private-locations-terraform-and-aws/)