Autoscaling Checkly Private Location Agents in Kubernetes with KEDA

Monitoring load is not always steady. A team might add a new batch of checks or run several ad hoc tests during a rollout. When that happens, your Private Location agents need to pick up more work at once.

If there aren’t enough agents available during a burst, checks start piling up in the queue, which can delay or disrupt check execution. But solving this by running a high number of agents around the clock has the opposite problem: most of that capacity sits idle until the next busy period.

Autoscaling solves both ends of that problem. You scale agents up when checks queue, scale back down when the burst clears, and stop guessing in between. This guide covers two things that landed together: the new Private Location metrics that show you whether your agents are running efficiently, and the KEDA setup that automatically scales your agent deployment.

How do you know if a Private Location has enough agents?

You can now monitor Private Location health with Checkly, so capacity stops being a guessing game. Open any Private Location in the UI and you'll find three metrics that tell you exactly how your agents are doing:

• Queue duration: how long check runs wait before an agent picks them up. This is the number to watch. If it creeps up, your agents may be falling behind on work.
• Agent count: how many agents are currently reporting for the location.
• Scheduled check runs: how many runs are queued for the location right now.

These metrics are also available via our Prometheus v2 integration.

Together, these signals help you understand whether your agents are healthy and whether your Private Location has enough capacity. For example, if two agents are reporting but queue duration keeps climbing and scheduled runs are stacking up, you may be under-provisioned. If ten agents are reporting and queue duration stays near zero all day, you may be running more capacity than you need. Those are the two states autoscaling exists to fix, and now you can see which one you're in.

Scaling on the right signal

Checkly agents are stateless: they sit in your network, pull scheduled checks from a shared queue, run them, and send the results back. You can add or remove agents freely without losing or duplicating work. That's the property that makes scaling safe.

To develop a practical approach to autoscaling, we studied a range of large Private Location setups across customers, including DVAG, LinkedIn, Sherwin-Williams, and Auto Trader. Looking at how those fleets handle different checks and workloads helped us identify the signal that best reflects a location’s current demand.

For teams running agents on Kubernetes, that signal is checkly_private_location_check_runs. This gauge reports the number of check runs routed through a Private Location, broken down by state.

Two states matter for capacity:

• queued: the check run is scheduled, but no agent has picked it up yet.
• inflight: an agent is running it right now.

Add those two and you have the total live load on the location, which is the signal you want driving replica count. To use this metric, make sure Prometheus V2 metrics are enabled for your account.

ℹ️ If your existing KEDA config still uses

checkly_private_location_oldest_scheduled_check_run, consider switching to checkly_private_location_check_runs with the queued and inflight states. The older metric is useful for spotting backlog, but queued plus inflight gives KEDA a more direct signal for current load.

Setting up KEDA

KEDA (Kubernetes Event-Driven Autoscaling) reads a Prometheus metric and scales a Kubernetes deployment up or down based on that value. You'll need KEDA installed in the cluster, your agents deployed via the Checkly agent Helm chart, and Prometheus V2 enabled for your Checkly account.

Here's a ScaledObject with sensible defaults. Adjust the bounds to your workload:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: checkly-agent-autoscaler
spec:
  scaleTargetRef:
    namespace: <namespace_for_agent_deployment>
    name: <agent_deployment_name>
  minReplicaCount: 2
  maxReplicaCount: 10
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
        scaleDown:
          selectPolicy: Min
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: checkly_private_location_check_runs
        threshold: "1"
        query: sum(checkly_private_location_check_runs{state=~"queued|inflight", private_location_slug_name="<slug>"})

The query targets a single Private Location using private_location_slug_name. If you run multiple Private Locations, create one ScaledObject for each. Because the slug name is immutable, renaming the location in the UI won’t affect your scaling config.

If you deploy agents with the Helm chart, template the ScaledObject alongside your chart values. This keeps the autoscaler tied to the agent deployment instead of managing it as a separate manifest.

For a Prometheus instance outside the cluster, add an authenticationRef that points to a TriggerAuthentication with the required credentials.

How many pods will you get?

Set the threshold to match your agent JOB_CONCURRENCY. This defines how many checks one agent pod can process at the same time. With the default JOB_CONCURRENCY of 1, use threshold: "1". That means KEDA will try to keep roughly one queued or in-flight check run per agent pod, within your min and max replica limits.

For example, with minReplicaCount: 2 and maxReplicaCount: 10:

A few rules of thumb for the bounds:

• Keep minReplicaCount at 2 or higher. One agent is a single point of failure, and if it dies your whole location goes dark until a replacement spins up.
• Set maxReplicaCount above your expected peak load, because check runs that pile up past the cap get dropped after the queue's six-minute TTL.
• Leave threshold at "1" unless you've raised JOB_CONCURRENCY; packing more checks per pod can delay long-running checks.
• If you want to scale to zero when a location is completely idle, set minReplicaCount: 0 and tune cooldownPeriod, which controls how long KEDA waits after load drops before scaling all the way down.

Don't kill checks mid-run

Scaling down means evicting pods, and an evicted pod might be running a check. Checkly handles this for you: an in-flight check on a terminating pod gets rerun on another agent after a 300-second timeout, so a scale-down never silently loses a result. To give a pod room to finish its current work before Kubernetes sends SIGKILL, set terminationGracePeriodSeconds above that window on the agent pod spec:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 330

Set it to match your longest check type. Most checks finish fast, but Playwright Check Suites can run up to an hour, so bump the grace period accordingly:

If you run Playwright Check Suites in this location, terminationGracePeriodSeconds can go as high as 1800.

What this gets you

Your checks stay reliable under load. When a deploy or a traffic spike floods the queue, agents scale up to clear it instead of letting runs wait. That means faster, more dependable detection from inside your own infrastructure.

You also avoid paying for idle capacity. Rather than keeping enough agents running all day to handle occasional peaks, you can maintain a smaller baseline and scale up only when needed. Across multiple Private Locations, that unused capacity can add up quickly.

Get started

Private Locations are available on Checkly's Team and Enterprise plans. If you're already running agents on Kubernetes, follow the Private Location Autoscaling guide for the complete setup, including how to verify that KEDA is receiving the metric. Enable Prometheus V2, apply in the ScaledObject, and run a batch of checks to see how your agent deployment scales with demand.

Already using Private Locations? Open a location in the Checkly UI to view its queue duration, agent count, and scheduled check runs. These metrics help you quickly identify whether you have too little or too much capacity.

New to Checkly? See pricing to find the plan that fits, or start for free to try the rest of the platform.