How Mistral Monitors Models and Products at Shipping Speed with Checkly

Mistral replaced a fragmented monitoring stack with a Terraform-managed workflow built on Checkly and Rootly, giving its SRE team Playwright-based validation of user workflows, a public status page, standardized synthetic checks across every model, and a foundation for developer self-service through AI coding agents.

Book a demo

98%

Of monitoring changes delegated to AI agents

14

Services on status.mistral.ai, powered by Checkly

2 Min.

Synthetic check interval

The Challenge

Mistral’s product velocity was outpacing the way monitoring was created. The SRE team needed monitoring that adapted to their pace, ensuring consistent coverage across models and products without slowing down delivery. Checkly’s Monitoring as Code became the foundation for that shift.

Fragmented tooling

Before Checkly, synthetic monitors lived in one tool, the status page in another, and neither worked reliably. Synthetic monitors would queue instead of running on schedule, alerts arrived via SMS with no grouping, and every new model meant configuring monitors by hand in a UI, hoping settings matched the others.

No path to infrastructure as code

The team wanted monitoring in Terraform, but the providers available were too unstable. And the previous stack had no Playwright support, so there was no way to validate end-to-end user workflows across Mistral Vibe, Studio, or internal tools.

“Without consistency, we're judging different models by different criteria.

There should be consistency. We had the SRE team try to add as many synthetic checks as possible in Terraform, but we ran into issues with provider stability. If you can't recreate your stack with infrastructure as code, then it's not infrastructure as code. ”

Devon Mizelle

Senior Site Reliability Engineer, Mistral

The Solution

Mistral adopted Checkly alongside Rootly and used Monitoring as Code to make Terraform the source of truth for its monitoring and alerting workflow. Checkly gave the team a way to define synthetic monitoring, Playwright workflows, alerting behavior, tags, and status-page coverage as code. Rootly uses those tags to route alerts to the correct team and escalation policy and to group related alerts into a single incident. No more pager storms.


Tags are a natural side-effect of Monitoring as Code: because checks are defined in Terraform, they're consistently tagged at creation time.

Playwright for Real User Workflows

Mistral runs Playwright check suites that validate real user journeys: logging into Le Chat, storing session state, and running functional tests against the authenticated interface. These checks cover Vibe, Studio, and internal admin tools, and they run against staging before production.

Monitoring as Code for the Live Model Catalog

A Terraform configuration queries Mistral's models endpoint, inspects each model's capabilities, and auto-provisions the matching Checkly checks. Every check sends a known input, validates the response, and applies the same standards. Checks run every two minutes, and paging begins after three consecutive failures.

A Public Status Page

status.mistral.ai is powered by Checkly, which monitors 14 services, including the Chat Completions API, Embeddings API, Agents API, Vibe, and Le Console. Migrating to Checkly's status page allowed Mistral to consolidate another tool from their stack.

Learn more about status pages ->

OpenTelemetry Traces for Faster Investigation

Mistral sends OpenTelemetry traces to Checkly so engineers can move from a failed synthetic check to the related trace context faster. That shortens the path from alert to diagnosis.

OpenTelemetry traces ->

The models automatically showing up as checks and monitors is a really important thing for us because we're constantly changing and experimenting. It's one less thing for developers to remember.

Devon Mizelle

Senior Site Reliability Engineer, Mistral

The Outcome

Checkly did not just help Mistral create more checks. It helped Mistral turn monitoring into a reproducible workflow.

  • Consistency across public models

Mistral now has a repeatable, Terraform-managed way to apply the same monitoring standards across public models, rather than rebuilding that logic by hand every time a new model ships.

  • Cleaner operations

With Checkly sending alerts into Rootly, and Rootly grouping related incidents, the team gets a more scalable alerting flow and a monitoring stack that is easier to reason about.

  • Developer self-service

Because monitoring now lives in code, Mistral built abstractions on top of Terraform so developers do not need to work directly with every Checkly or Rootly resource to make common changes. That makes monitoring easier to extend across teams and easier to review as part of the normal development workflow.

That self-service model is now expanding again through AI coding agents. Internally, Mistral uses its own coding agent, Mistral Vibe, to let developers configure monitoring through conversation rather than code. Developers can ask the agent to add a new team, configure checks, or modify alert routing.

As Devon puts it, "As long as you provide the coding agent proper documentation, it can figure out how to perform certain tasks. 98% of the time I can just pass it off, go get a cup of coffee, and come back and the tests pass."

For Mistral, that changes the SRE team's role. Instead of manually stitching together checks and alert rules, they can focus on building the abstractions, standards, and guardrails that let the rest of the organization move faster.

What's Next

Mistral is already exploring how AI agents can assist with incident investigation by pulling context from tools such as Checkly, Grafana, and Rootly via MCP (Model Context Protocol) servers. The goal is not just faster alerting, but faster understanding, so engineers spend less time assembling context and more time resolving the issue.

The team is also looking at checks that adapt to their code: a system smart enough to correlate a check failure with a recent deployment, roll back the offending change, and notify the team without waking an engineer at 3 a.m.

"The perfect world is an agent getting the page first, looking up all the context, checking Grafana, running queries from metrics, logs, and traces. Things that usually take you 20 minutes, it can do in two."

Devon Mizelle

Senior Site Reliability Engineer, Mistral